Image processing device, endoscope, image processing method, and program

Abstract

This image processing device comprises a processor. The processor: acquires a plurality of medical images in which an observation region is shown and which are in chronological order; and detects changes in the state of the observation region by subjecting the plurality of medical images to image recognition.

Description

Image processing device, endoscope, image processing method, and program

[0001] The technology disclosed herein relates to an image processing device, an endoscope, an image processing method, and a program.

[0002] Japanese Patent Application Laid-Open Publication No. 2017-099509 discloses an endoscopic processing device. Endoscopic images of the inside of a patient's body captured with an endoscope are input as moving images to the endoscopic processing device. A treatment execution detection unit included in the endoscopic processing device detects treatment tools used in endoscopic examinations from frame images included in the endoscopic moving images using image recognition, and detects the treatment being performed based on the detected treatment tools. The treatment execution detection unit also periodically searches for treatment tools within the frame images using image recognition.

[0003] Japanese Patent Application Laid-Open Publication No. 2017-006337 discloses a medical support device including a detection unit that detects an in vivo event based on a time change in temperature at or around the surgical site, and a notification information generation unit that generates notification information to provide a notification in response to the in vivo event detected by the detection unit. The detection unit detects the in vivo event based on a time change in a temperature distribution image of the surgical site or around the surgical site. The detection unit also detects a condition that requires a warning as an in vivo event based on the time change in the temperature distribution image, and the notification information generation unit generates a warning image to provide a warning regarding the condition.

[0004] One embodiment of the technology disclosed herein provides an image processing device, an endoscope, an image processing method, and a program that can detect changes in the state of an observation area with higher accuracy than when detecting changes in the state of an observation area using only a single medical image.

[0005] A first aspect of the technology of the present disclosure is an image processing device that includes a processor, which acquires a plurality of medical images in chronological order, each of which shows an area to be observed, and performs image recognition processing on the plurality of medical images to detect changes in the state of the area to be observed.

[0006] A second aspect of the technology of the present disclosure is the image processing device according to the first aspect, in which the state change includes a change in adhesive color, a change in mucosal state including mucosal structure, and / or a change in mucus adhesion state.

[0007] A third aspect of the technology of the present disclosure is an image processing device according to the first or second aspect, in which, when multiple medical images are generated by an endoscope, the processor performs image recognition processing based on the operation of the endoscope.

[0008] A fourth aspect of the technology of the present disclosure is the image processing device according to any one of the first to third aspects, wherein the processor performs image recognition processing based on given medical instructions.

[0009] A fifth aspect of the technology of the present disclosure is an image processing device according to any one of the first to fourth aspects, in which a processor acquires region of interest information relating to a region of interest included in an observation target region, and performs image recognition processing based on the region of interest information.

[0010] A sixth aspect of the technology of the present disclosure is an image processing device according to any one of the first to fifth aspects, in which a processor acquires part information regarding a part corresponding to an observation target area and performs image recognition processing based on the part information.

[0011] A seventh aspect of the technology of the present disclosure is an image processing device according to any one of the first to sixth aspects, in which the processor starts image recognition processing when a first condition is satisfied.

[0012] An eighth aspect of the technology of the present disclosure is the image processing device of the seventh aspect, in which, when multiple medical images are generated by an endoscope, the first condition includes a condition that the tip of the endoscope has stopped moving or a condition that the movement speed of the tip has decreased.

[0013] A ninth aspect of the technique of the present disclosure is the image processing device according to the seventh or eighth aspect, in which the first condition includes a condition that an instruction to start image recognition processing has been given.

[0014] A tenth aspect of the technology of the present disclosure is an image processing device according to any one of the seventh to ninth aspects, in which the first condition includes a condition that the observation target area includes a region of interest.

[0015] An eleventh aspect of the technology of the present disclosure is an image processing device according to any one of the seventh to tenth aspects, in which the first condition includes a condition that the area corresponding to the observation target area is an area designated as the observation target.

[0016] A twelfth aspect of the technology of the present disclosure is an image processing device according to any one of the first to eleventh aspects, in which the processor terminates the image recognition processing when a second condition is satisfied.

[0017] A thirteenth aspect of the technology of the present disclosure is the image processing device according to the twelfth aspect, in which the processor erases the first information, which is information based on image recognition processing, when the second condition is satisfied.

[0018] A fourteenth aspect of the technology of the present disclosure is the image processing device of the thirteenth aspect, in which the first information is retained from the start to the end of the image recognition processing, and the processor erases the first information when the image recognition processing ends.

[0019] A fifteenth aspect of the technology of the present disclosure is an image processing device according to any one of the twelfth to fourteenth aspects, in which, when multiple medical images are generated by an endoscope, the second condition includes a condition that the tip of the endoscope has started or a condition that the movement speed of the tip has increased.

[0020] A sixteenth aspect of the technology of the present disclosure is an image processing device according to any one of the twelfth to fifteenth aspects, in which the second condition includes a condition that an instruction to terminate the image recognition processing has been given.

[0021] A seventeenth aspect of the technology of the present disclosure is an image processing device according to any one of the twelfth to sixteenth aspects, in which the second condition includes a condition that the observation target area does not include an area of ​​interest.

[0022] An 18th aspect of the technology of the present disclosure is an image processing device according to any one of the 12th to 17th aspects, in which the second condition includes a condition that the area corresponding to the observation target area is a different area from the area specified as the observation target.

[0023] A 19th aspect of the technology of the present disclosure is an image processing device according to any one of the first to eighteenth aspects, in which when multiple medical images are generated by an endoscope, the processor detects a state change based on the operation of the endoscope.

[0024] A twentieth aspect of the technology of the present disclosure is an image processing device according to any one of the first to nineteenth aspects, in which, when multiple medical images are generated by an endoscope and fluid is delivered from the endoscope into a body including an observation target area, a processor acquires fluid delivery information regarding the delivery of the fluid and detects a state change based on the fluid delivery information.

[0025] A twenty-first aspect of the technology of the present disclosure is an image processing device according to any one of the first to twentieth aspects, wherein the processor detects a change in condition based on a given medical instruction.

[0026] A 22nd aspect of the technology of the present disclosure is an image processing device according to any one of the 1st to 21st aspects, in which a processor acquires region of interest information relating to a region of interest included in an observation target region, and detects a state change on the condition that the region of interest information has been acquired.

[0027] A 23rd aspect of the technology of the present disclosure is an image processing device according to any one of the first to 22nd aspects, in which a processor acquires part information regarding a part corresponding to an observation target area and detects a state change on the condition that the part information has been acquired.

[0028] A 24th aspect of the technology of the present disclosure is an image processing device according to any one of the first to 23rd aspects, in which a processor derives lesion information regarding a lesion in an observation target area based on a state change.

[0029] A 25th aspect of the technology of the present disclosure is an image processing device according to the 24th aspect, in which the observation region includes a region of interest, the state change includes a change in the region of interest, and the change in the region of interest is a change from a state in which mucus is attached to the region of interest to a state in which a non-neoplastic polyp appears from the region of interest.

[0030] A 26th aspect of the technology of the present disclosure is an image processing device according to the 24th aspect, in which when an observation region includes a region of interest, multiple medical images are generated by an endoscope, and fluid is delivered from the endoscope into a body including the observation region, the state change includes a change in the region of interest accompanying the delivery of the fluid, and the processor acquires delivery amount information indicating the amount of fluid delivered, and derives lesion information based on the state change and the delivery amount information.

[0031] A 27th aspect of the technology of the present disclosure is the image processing device of the 26th aspect, in which the change in the region of interest is a change from a state in which mucus is attached to the region of interest to a state in which a non-neoplastic polyp is emerging from the region of interest.

[0032] A 28th aspect of the technology of the present disclosure is an image processing device according to any one of the 24th to 27th aspects, in which, when multiple medical images are generated by an endoscope, the processor derives lesion information at a timing determined based on the operation of the endoscope.

[0033] A 29th aspect of the technology of the present disclosure is an image processing device according to any one of the 24th to 28th aspects, in which the processor derives lesion information at a predetermined timing based on given medical instructions.

[0034] A 30th aspect of the technology of the present disclosure is an image processing device according to any one of the 24th to 29th aspects, in which, when the processor derives lesion information, it confirms the lesion information in accordance with a given confirmation instruction.

[0035] A 31st aspect of the technology of the present disclosure is an image processing device according to any one of the 24th to 30th aspects, in which a processor acquires region of interest information relating to a region of interest included in an observation target region, and derives lesion information when the region of interest information is information relating to a specific region of interest.

[0036] A 32nd aspect of the technology of the present disclosure is an image processing device according to any one of the 24th to 31st aspects, in which a processor acquires part information regarding a part corresponding to an observation target area, and derives lesion information when the part information is information regarding a specific part.

[0037] A thirty-third aspect of the technology of the present disclosure is an image processing device according to any one of the first to thirty-second aspects, in which the processor outputs second information, which is information based on image recognition processing.

[0038] A thirty-fourth aspect related to the technology of the present disclosure is the image processing device related to the thirty-third aspect, in which an output destination of the second information is a display device, and the display device displays the second information.

[0039] A thirty-fifth aspect of the technology of the present disclosure is an endoscope comprising an image processing device according to any one of the first to thirty-fourth aspects and an endoscope body that is inserted into a body including an area to be observed.

[0040] A 36th aspect of the technology of the present disclosure is an image processing method that includes acquiring a plurality of medical images in chronological order, each of which shows an area to be observed, and performing image recognition processing on the plurality of medical images to detect a change in the state of the area to be observed.

[0041] A 37th aspect of the technology of the present disclosure is a program for causing a computer to execute processing including acquiring a plurality of medical images in chronological order, each of which shows an area to be observed, and detecting a change in the state of the area to be observed by performing image recognition processing on the plurality of medical images.

[0042] 9B is a conceptual diagram showing an example of an aspect in which an endoscope system is used. FIG. 9C is a conceptual diagram showing an example of the overall configuration of an endoscope system. FIG. 9D is a block diagram showing an example of the hardware configuration of an electrical system of an endoscope system. FIG. 9E is a block diagram showing an example of main functions of a processor of a control device included in an endoscope. FIG. 9F is a conceptual diagram showing an example of the correlation between a camera, an NVM, a reception device, an image acquisition unit, a part detection unit, a region of interest detection unit, and a control unit. FIG. 9G is a conceptual diagram showing an example of the correlation between a camera, an image acquisition unit, and an endoscope detection unit. FIG. 9H ... reception device, an image acquisition unit, a part detection unit, a region of interest detection unit, an endoscope detection unit, a state change detection unit, and a control unit. FIG. 9I is a flowchart showing an example of the correlation between a state change detection unit, a lesion information derivation unit, a control unit, and a display device. FIG. 9I is a continuation of the flowchart shown in FIG. 9A. FIG. 9I is a block diagram showing a modified example of the output timing of a start instruction signal and an end instruction signal. FIG. 9I is a block diagram showing an example of a modified example of a lesion information derivation unit. FIG. 9I is a flowchart showing a first modified example of the flow of medical support processing. FIG. 9I is a flowchart showing a second modified example of the flow of medical support processing.

[0043] Hereinafter, exemplary embodiments of an image processing device, an endoscope, an image processing method, and a program according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0044] First, the terms used in the following description will be explained.

[0045] CPU is an abbreviation for "Central Processing Unit." GPU is an abbreviation for "Graphics Processing Unit." RAM is an abbreviation for "Random Access Memory." NVM is an abbreviation for "Non-volatile memory." EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory." ASIC is an abbreviation for "Application Specific Integrated Circuit." PLD is an abbreviation for "Programmable Logic Device." FPGA is an abbreviation for "Field-Programmable Gate Array." SoC is an abbreviation for "System-on-a-chip." SSD is an abbreviation for "Solid State Drive." USB is an abbreviation for "Universal Serial Bus." HDD is an abbreviation for "Hard Disk Drive." EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". I / F is an abbreviation for "Interface". FIFO is an abbreviation for "First In First Out".

[0046] 1 , an endoscopic system 10 includes an endoscope 12 and a display device 13. The endoscope 12 is used by a doctor 14 during an endoscopic examination. At least one assistant staff member 16 (e.g., a nurse) assists the doctor 14 in the endoscopic examination. Hereinafter, when it is not necessary to distinguish between the doctor 14 and the assistant staff member 16, they will also be referred to as "users" without being assigned reference numerals.

[0047] The endoscope 12 includes an endoscope body 18. The endoscope 12 is a device for performing medical treatment on an observation target region 21 included in the body of a subject 20 (e.g., a patient) (here, as an example, the inside of the large intestine) using the endoscope body 18. The observation target region 21 is a region to be observed by a doctor 14. The endoscope body 18 is inserted into the body of the subject 20. The endoscope 12 causes the endoscope body 18 inserted into the body of the subject 20 to capture images of the observation target region 21 inside the body of the subject 20, and performs various medical procedures on the observation target region 21 as necessary. The endoscope 12 is an example of an "endoscope" according to the technology of the present disclosure. The endoscope body 18 is an example of an "endoscope body" according to the technology of the present disclosure.

[0048] The endoscope 12 captures images of the inside of the body of the subject 20, thereby acquiring and outputting images showing the state of the inside of the body. In the example shown in Fig. 1, a lower extremity endoscope is shown as an example of the endoscope 12. Note that the lower extremity endoscope is merely an example, and the technology of the present disclosure can be applied even if the endoscope 12 is another type of endoscope, such as an upper gastrointestinal endoscope or a bronchial endoscope.

[0049] Furthermore, in this embodiment, the endoscope 12 is an endoscope having an optical imaging function that captures an image of reflected light obtained by irradiating light inside the body and reflecting it off the observation target region 21. However, this is merely one example, and the technology of the present disclosure also applies if the endoscope 12 is an ultrasound endoscope. Furthermore, the technology of the present disclosure also applies if, instead of the endoscope 12, a modality that generates multiple frames for inspection or surgery (for example, moving images obtained by imaging using X-rays or the like, or moving images based on reflected waves of ultrasound emitted from outside the body of the subject 20) is used.

[0050] The endoscope 12 is equipped with a control device 22 and a light source device 24. The control device 22 and the light source device 24 are installed on a wagon 34. The wagon 34 has a plurality of stands arranged in the vertical direction, and the control devices 22 and the light source devices 24 are installed from the lower stand to the upper stand. In addition, a display device 13 is installed on the top stand of the wagon 34.

[0051] The display device 13 displays various information including images. Examples of the display device 13 include a liquid crystal display and an EL display. The display device 13 displays one or more screens arranged side by side. In the example shown in FIG. 1 , screen 36 is shown. Alternatively, a tablet terminal with a display may be used instead of or together with the display device 13.

[0052] An endoscopic image 40 obtained by the endoscope 12 is displayed on the screen 36. The endoscopic image 40 shows the observation target region 21, including a region of interest 21A. The region of interest 21A is a region within the observation target region 21 that is determined to require observation (e.g., a region determined as a region that the physician 14 focuses on when identifying a lesion). The endoscopic image 40 is an image generated by the endoscope 12 capturing an image of the observation target region 21 inside the body of the subject 20. The observation target region 21 may be the inner wall of the large intestine. The inner wall of the large intestine is merely an example, and any region that can be captured by the endoscope 12 may be used. Examples of regions that can be captured by the endoscope 12 include the inner or outer wall of a hollow organ. Examples of hollow organs include the small intestine, duodenum, esophagus, stomach, and bronchi. The observation target region 21 captured by the endoscope 12 is an example of an "observation target region" according to the technology of the present disclosure. The endoscopic image 40 is an example of a "medical image" according to the technology of the present disclosure.

[0053] The endoscopic image 40 displayed on the screen 36 is one frame included in a moving image that includes multiple frames. In other words, the screen 36 displays multiple frames of the endoscopic image 40 at a predetermined frame rate (e.g., several tens of frames per second).

[0054] 2, the endoscope 12 includes an operation section 42 and an insertion section 44. The insertion section 44 is partially curved by operating the operation section 42. The insertion section 44 is inserted while curving in accordance with the shape of the body of the subject 20 (for example, the shape of the large intestine) in accordance with the operation of the operation section 42 by the physician 14.

[0055] The distal end 46 of the insertion section 44 is provided with a camera 48, an illumination device 50, and a treatment opening 52. The camera 48 is a device for capturing images of the interior of the subject's body 20. An example of the camera 48 is a CMOS camera. However, this is merely an example, and other types of cameras, such as a CCD camera, may also be used. The illumination device 50 has illumination windows 50A and 50B. The illumination device 50 emits light through the illumination windows 50A and 50B. Examples of the light emitted from the illumination device 50 include visible light (e.g., white light) and invisible light (e.g., near-infrared light). The illumination device 50 also emits special light through the illumination windows 50A and 50B. Examples of the special light include light for BLI and / or light for LCI. The camera 48 captures images of the interior of the subject's body 20 using an optical method while light is being emitted from the illumination device 50 inside the subject's body 20.

[0056] The treatment opening 52 is used as a treatment tool ejection port for ejecting a treatment tool 54 from the distal end portion 46, a suction port for sucking blood and internal waste, and a delivery port for delivering a fluid 56.

[0057] A treatment tool 54 protrudes from the treatment opening 52 in accordance with the operation of the physician 14. The treatment tool 54 is inserted into the insertion section 44 from a treatment tool insertion port 58. The treatment tool 54 passes through the insertion section 44 and protrudes from the treatment opening 52 into the body of the subject 20. In the example shown in FIG. 2 , forceps protrude from the treatment opening 52 as the treatment tool 54. The forceps are merely one example of the treatment tool 54, and other examples of the treatment tool 54 include a wire, a scalpel, and an ultrasonic probe. The tip portion 46 and the treatment tool 54 are one example of the "tip portion of an endoscope" according to the technology of the present disclosure.

[0058] A suction pump (not shown) is connected to the endoscope body 18, and the treatment opening 52 uses the suction force of the suction pump to suck blood, internal waste, and the like. The suction force of the suction pump is controlled in accordance with instructions given by the physician 14 to the endoscope 12 via the operation unit 42 or the like. A supply pump (not shown) is connected to the endoscope body 18, and the supply pump supplies a fluid 56 (e.g., gas and liquid) into the endoscope body 18. The treatment opening 52 delivers the fluid 56 supplied from the supply pump to the endoscope body 18. From the treatment opening 52, gas (e.g., air) and liquid (e.g., saline) are selectively delivered as the fluid 56 into the body in accordance with instructions given by the physician 14 to the endoscope 12 via the operation unit 42 or the like. The amount of fluid 56 delivered is controlled in accordance with instructions given by the physician 14 to the endoscope 12 via the operation unit 42 or the like.

[0059] The endoscope body 18 is connected to the control device 22 and the light source device 24 via a universal cord 60. The control device 22 is connected to the display device 13 and the reception device 62. The reception device 62 receives instructions from a user and outputs the received instructions as an electrical signal. In the example shown in Fig. 2, a keyboard is given as an example of the reception device 62. However, this is merely an example, and the reception device 62 may also be a mouse, a touch panel, a foot switch, and / or a microphone, etc.

[0060] The control device 22 controls the entire endoscope 12. For example, the control device 22 controls the light source device 24, exchanges various signals with the camera 48, and displays various information on the display device 13. The light source device 24 emits light under the control of the control device 22 and supplies the light to the illumination device 50. The illumination device 50 has a built-in light guide, and the light supplied from the light source device 24 passes through the light guide and is irradiated from illumination windows 50A and 50B. The control device 22 causes the camera 48 to capture an image, acquires an endoscopic image 40 (see FIG. 1 ) from the camera 48, and outputs the image to a predetermined output destination (for example, the display device 13).

[0061] As an example, as shown in Fig. 3, the control device 22 includes a computer 64. The computer 64 is an example of an "image processing device" and a "computer" according to the techniques of the present disclosure. The computer 64 includes a processor 70, a RAM 72, and a NVM 74, which are electrically connected to each other. The processor 70 is an example of a "processor" according to the techniques of the present disclosure.

[0062] The control device 22 includes a computer 64, a bus 66, and an external I / F 68. The computer 64 includes a processor 70, a RAM 72, and an NVM 74. The processor 70, the RAM 72, the NVM 74, and the external I / F 68 are connected to the bus 66.

[0063] For example, the processor 70 has a CPU and a GPU, and controls the entire control device 22. The GPU operates under the control of the CPU, and is responsible for executing various graphic processing and performing calculations using neural networks. The processor 70 may be one or more CPUs that have integrated GPU functionality, or one or more CPUs that do not have integrated GPU functionality.

[0064] The RAM 72 is a memory that temporarily stores information and is used as a work memory by the processor 70. The NVM 74 is a nonvolatile storage device that stores various programs, various parameters, and the like. An example of the NVM 74 is a flash memory (for example, an EEPROM and / or an SSD). Note that the flash memory is merely one example, and the NVM 74 may be another nonvolatile storage device such as an HDD, or may be a combination of two or more types of nonvolatile storage devices.

[0065] The external I / F 68 controls the exchange of various information between a device existing outside the control device 22 (hereinafter also referred to as an "external device") and the processor 70. An example of the external I / F 68 is a USB interface.

[0066] The external I / F 68 is connected to the camera 48 as one of the external devices, and the external I / F 68 controls the exchange of various information between the camera 48 and the processor 70. The processor 70 controls the camera 48 via the external I / F 68. The processor 70 also acquires, via the external I / F 68, an endoscopic image 40 (see FIG. 1 ) obtained by the camera 48 capturing an image of the inside of the body of the subject 20.

[0067] The light source device 24 is connected to the external I / F 68 as one of the external devices, and the external I / F 68 controls the exchange of various information between the light source device 24 and the processor 70. The light source device 24 supplies light to the illumination device 50 under the control of the processor 70. The illumination device 50 irradiates the light supplied from the light source device 24.

[0068] The external I / F 68 is connected to the display device 13 as one of the external devices, and the processor 70 controls the display device 13 via the external I / F 68 to cause the display device 13 to display various information.

[0069] The external I / F 68 is connected to a reception device 62 as one of the external devices, and the processor 70 acquires instructions accepted by the reception device 62 via the external I / F 68 and executes processing according to the acquired instructions.

[0070] Recently, a technique has become popular in which a modality such as an endoscope 12 performs AI-based image recognition processing to detect the presence or absence of a lesion or identify the type of lesion. In this technique, the AI-based image recognition processing is performed on only a single frame, and the presence or absence of a lesion is detected or the type of lesion is identified from the image recognition results obtained. Since this method results in blurring between frames, post-processing, such as averaging the image recognition results obtained over time, is performed to reduce this blurring. However, in practice, the physician 14 differentiates between lesions based on a comprehensive assessment of the area being observed, changes in the state of mucus due to the fluid 56, and / or changes in the state of the mucosa due to the fluid 56, etc.

[0071] For example, serrated lesions, a type of colorectal neoplastic polyp, are difficult to distinguish from hyperplastic polyps visually and are known to be prone to cancer. Therefore, it is necessary to carefully distinguish between serrated lesions and hyperplastic polyps by comprehensively assessing various information obtained over time from the endoscope 12. For example, serrated lesions are characterized by the presence of mucus on the polyp and their tendency to occur in the right-sided large intestine (e.g., the ascending colon and transverse colon). Therefore, it is important for the physician 14 to not overlook these characteristics and to capture them as information for making a decision when differentiating between lesions.

[0072] Similarly to serrated lesions, Barrett's esophageal cancer is also known to be one of the lesions that are difficult to differentiate. Barrett's esophageal cancer often occurs at the junction between the stomach and esophagus, making it difficult to observe, and there are few changes (i.e., protrusions, color changes, etc.) from the surrounding mucosa. The doctor 14 carefully observes the Barrett's mucosa and then comprehensively considers the time-series changes in the state of the mucosa caused by changing the state of air supply from the endoscope 12 (for example, a state change in which the mucosa is flat with air supply but protrudes without air supply) to differentiate whether or not it is Barrett's esophageal cancer.

[0073] In view of the above circumstances, in this embodiment, medical support processing is performed by the processor 70 of the control device 22 to support the doctor 14 in distinguishing between lesions (see FIGS. 9A and 9B ). The medical support processing includes processing for acquiring a plurality of endoscopic images 40 in time series showing the observation target region 21, and performing image recognition processing on the plurality of endoscopic images 40 to detect changes in the state of the observation target region 21.

[0074] 4 , a medical support processing program 76 is stored in the NVM 74. The medical support processing program 76 is an example of a "program" according to the technology of the present disclosure. The processor 70 reads the medical support processing program 76 from the NVM 74 and executes the read medical support processing program 76 on the RAM 72. The medical support processing is realized by the processor 70 operating as an image acquisition unit 70A, a site detection unit 70B, a region of interest detection unit 70C, an endoscope detection unit 70D, a state change detection unit 70E, a control unit 70F, and a lesion information derivation unit 70G in accordance with the medical support processing program 76 executed on the RAM 72.

[0075] The NVM 74 stores a first trained model 78, a plurality of second trained models 80, a third trained model 82, a fourth trained model 84, and a fifth trained model 86. The first trained model 78, the plurality of second trained models 80, the third trained model 82, the fourth trained model 84, and the fifth trained model 86 are all optimized by performing machine learning on the neural network in advance.

[0076] The first trained model 78 is used by the region detection unit 70B. The multiple second trained models 80 are selectively used by the region of interest detection unit 70C. The third trained model 82 is used by the endoscope detection unit 70D. The multiple fourth trained models 84 are selectively used by the state change detection unit 70E. The fifth trained model 86 is used by the lesion information derivation unit 70G.

[0077] For ease of explanation, the first trained model 78, the multiple second trained models 80, the third trained model 82, the fourth trained model 84, and the fifth trained model 86 will be referred to as "trained models" when there is no need to distinguish between them. For ease of explanation, the processing using the trained models will be described below as processing actively performed by the trained models. In other words, for ease of explanation, the trained models will be described as a function that processes input information and outputs the processing results.

[0078] As an example, as shown in FIG. 5, the image acquisition unit 70A acquires the endoscopic image 40 generated by the camera 48 capturing images at an imaging frame rate (e.g., several tens of frames per second) from the camera 48, one frame at a time.

[0079] The image acquisition unit 70A holds a time-series image group 89. The time-series image group 89 is a plurality of endoscopic images 40 in time series that capture the observation target region 21. The time-series image group 89 includes, for example, a certain number of frames (for example, a predetermined number of frames within a range of several tens to several hundreds of frames) of endoscopic images 40. The image acquisition unit 70A updates the time-series image group 89 in a FIFO manner every time it acquires an endoscopic image 40 from the camera 48.

[0080] Although an example in which the time-series image group 89 is stored and updated by the image acquisition unit 70A is given here, this is merely one example. For example, the time-series image group 89 may be stored and updated in a memory connected to the processor 70, such as the RAM 72.

[0081] When the first condition is satisfied, the control unit 70F outputs a start instruction signal 91 to the part detection unit 70B, the region of interest detection unit 70C, the endoscope detection unit 70D, and the state change detection unit 70E. When the start instruction signal 91 is input from the control unit 70F, the part detection unit 70B, the region of interest detection unit 70C, the endoscope detection unit 70D, and the state change detection unit 70E start AI-based image recognition processing. Here, AI-based image recognition processing refers to, for example, image recognition processing by the part detection unit 70B using the first trained model 78, image recognition processing by the region of interest detection unit 70C using the second trained model 80, image recognition processing by the endoscope detection unit 70D using the third trained model 82, and image recognition processing by the state change detection unit 70E using the fourth trained model 84.

[0082] The first condition may include, for example, a condition that a start instruction has been given to the endoscope 12 (in the example shown in FIG. 7 , a condition that the start instruction has been accepted by the acceptance device 62). The start instruction refers to an instruction to start the AI-based image recognition processing.

[0083] The part detection unit 70B detects parts of the subject 20 by performing image recognition processing using a first trained model 78 on the time-series image group 89 (i.e., the multiple time-series endoscopic images 40 stored by the image acquisition unit 70A). The first trained model 78 is a trained model for object detection using an AI system and is optimized by performing machine learning on a neural network using first training data. Examples of the first training data include training data in which multiple images (e.g., multiple images corresponding to the multiple time-series endoscopic images 40) obtained in time series by capturing images of parts that may be the subject of endoscopic examination are used as example data, and part information 90 related to the parts is used as correct answer data. Note that, although an example in which only one first trained model 78 is used by the part detection unit 70B is given here, this is merely an example. For example, a first trained model 78 selected from multiple first trained models 78 may be used by the part detection unit 70B. In this case, each first trained model 78 is created by performing machine learning specialized for each type of endoscopic examination, and the first trained model 78 corresponding to the type of endoscopic examination currently being performed is selected and used by the part detection unit 70B.

[0084] An example of the site information 90 is information indicating the name of a site. Here, examples of sites that may be the subject of endoscopic examination include a single site within the large intestine (e.g., the ascending colon) and multiple adjacent sites (e.g., the striated colon and the descending colon). While sites within the large intestine are listed here, this is merely an example, and sites within hollow organs such as the stomach, esophagus, gastroesophageal junction, duodenum, or bronchi may also be used.

[0085] The part detection unit 70B acquires a time-series image group 89 and inputs the acquired time-series image group 89 to the first trained model 78. As a result, the first trained model 78 outputs part information 90 corresponding to the input multiple endoscopic images 40. The part detection unit 70B acquires the part information 90 output from the first trained model 78. The part information 90 acquired by the part detection unit 70B is information regarding a part corresponding to the observation target region 21 shown in the endoscopic image 40. The part information 90 is an example of "part information" and "first information that is information based on image recognition processing" according to the technology of the present disclosure.

[0086] The region of interest detection unit 70C performs image recognition processing on the time-series image group 89 using the second trained model 80, thereby detecting a region of interest 21A included in the observation target region 21. Examples of the region of interest 21A include a mucus region (i.e., a region where mucus is attached), Barrett's mucosa, and / or Barrett's adenocarcinoma.

[0087] The second trained model 80 used by the region of interest detection unit 70C is selected from the multiple second trained models 80 stored in the NVM 74. The multiple second trained models 80 stored in the NVM 74 correspond to different body parts. The region of interest detection unit 70C selects and uses the second trained model 80 corresponding to the body part information 90 acquired by the body part detection unit 70B from the multiple second trained models 80. For example, if the body part information 90 is information about the ascending colon, the region of interest detection unit 70C selects the second trained model 80 created for the ascending colon from the multiple second trained models 80. Furthermore, for example, if the body part information 90 is information about the sigmoid rectum and the descending colon, the region of interest detection unit 70C selects the second trained model 80 created for the sigmoid rectum and the descending colon from the multiple second trained models 80. Note that, although an example is given here in which the second trained model 80 used by the region of interest detection unit 70C is selected from multiple second trained models 80 in accordance with the site information 90, this is merely one example. For example, a single second trained model 80 corresponding to all sites that may be the target of endoscopic examination may be predefined as the second trained model 80 used by the region of interest detection unit 70C.

[0088] The second trained model 80 is a trained model for object detection using an AI system, and is optimized by performing machine learning on the neural network using second training data. Examples of the second training data include training data in which multiple images (e.g., multiple images corresponding to multiple endoscopic images 40 in time series) obtained by capturing an area of ​​interest 21A that may be the target of an endoscopic examination (e.g., an area of ​​interest 21A occurring in an area identified from the area information 90) are used as example data, and region of interest information 92 related to the area of ​​interest 21A is used as correct answer data. Examples of the region of interest information 92 include information indicating the presence or absence of the area of ​​interest 21A and information indicating the name of the area of ​​interest 21A.

[0089] The region of interest detection unit 70C acquires a time-series image group 89 and inputs the acquired time-series image group 89 to the second trained model 80. As a result, the second trained model 80 outputs region of interest information 92 corresponding to the input time-series image group 89. The region of interest detection unit 70C acquires the region of interest information 92 output from the second trained model 80. The region of interest information 92 acquired by the region of interest detection unit 70C is information about the region of interest 21A included in the observation target region 21 shown in the endoscopic image 40. The region of interest information 92 is an example of "region of interest information" and "first information that is information based on image recognition processing" related to the disclosed technology.

[0090] An example of the region of interest information 92 is information indicating the name of the region of interest 21 A. The region of interest 21 A that may be the subject of an endoscopic examination may be, for example, a single region of interest 21 A (e.g., a specific mucus region) or multiple regions of interest 21 A that transition over time (e.g., a specific mucus region and a specific mucous membrane region).

[0091] As an example, as shown in FIG. 6 , the endoscope detection unit 70D detects the movement of the endoscope 12 by performing image recognition processing on a time-series image group 89 using a third trained model 82. The third trained model 82 is a trained model for object detection using an AI system and is optimized by performing machine learning on a neural network using third training data. Examples of the third training data include training data in which multiple images obtained in time series by capturing images of the inside of the body with the camera 48 are used as example data and endoscopic information 94 regarding the movement of the endoscope 12 is used as correct answer data. Note that, while an example in which only one third trained model 82 is used by the endoscope detection unit 70D is given here, this is merely an example. For example, the endoscope detection unit 70D may use a third trained model 82 selected from multiple third trained models 82. In this case, each third trained model 82 is created by performing machine learning specialized for each type of endoscopic examination, and the third trained model 82 corresponding to the type of endoscopic examination currently being performed (here, as an example, the type of endoscope 12) is selected and used by the endoscope detection unit 70D.

[0092] The endoscope detection unit 70D acquires a time-series image group 89 and inputs the acquired time-series image group 89 to the third trained model 82. As a result, the third trained model 82 outputs endoscope information 94 corresponding to the input multiple endoscope images 40. The endoscope detection unit 70D acquires the endoscope information 94 output from the third trained model 82. The endoscope information 94 acquired by the endoscope detection unit 70D is information regarding the operation of the endoscope 12 currently being used. The endoscope information 94 is an example of "first information that is information based on image recognition processing" according to the technology of the present disclosure.

[0093] Examples of the endoscope information 94 include treatment tool information 94A, operating speed information 94B, and fluid delivery information 94C. The treatment tool information 94A is information related to the treatment tool 54 (see FIG. 2). Examples of the information related to the treatment tool 54 include information indicating whether the treatment tool 54 is in use or not, and information indicating the type of treatment tool 54 being used. The operating speed information 94B is information related to the operating speed of the tip portion 46 (see FIG. 2) of the endoscope 12 (for example, information related to the speed expressed in units of "millimeters per second").

[0094] The fluid delivery information 94C is information related to the delivery of the fluid 56 (see FIG. 2). The information related to the delivery of the fluid 56 refers, for example, to information related to the amount of fluid 56 delivered per unit time (for example, information related to the amount of delivery expressed in units of "milliliters / second"). The fluid delivery information 94C includes air delivery amount information 94C1 and water delivery amount information 94C2. The air delivery amount information 94C1 is information related to the amount of gas delivered (for example, information related to the amount of gas delivered per unit time). The water delivery amount information 94C2 is information related to the amount of liquid delivered (for example, information related to the amount of liquid delivered per unit time). The fluid delivery information 94C is an example of "fluid delivery information" related to the technology disclosed herein.

[0095] The endoscope information 94 may include operation information of the endoscope 12 (e.g., information based on the results of measuring the operation time of the camera 48, etc.) and / or information obtained from various sensors (e.g., pressure sensors, etc.) mounted on the tip 46.

[0096] 7 , the state change detection unit 70E detects state changes in the observation target region 21 by performing image recognition processing on a time-series image group 89 using the fourth trained model 84. Examples of state changes in the observation target region 21 include changes in adhesive color, changes in mucosal state including mucosal structure, and / or changes in mucus adhesion state.

[0097] The fourth trained model 84 is a trained model for object detection using an AI system and is optimized by performing machine learning on the neural network using fourth training data. Examples of the fourth training data include training data in which multiple images obtained in time series by capturing images of the inside of the body with the camera 48, region of interest information 92, and endoscope information 94 are used as example data, and state change information 96 regarding state changes in the observation target region 21 is used as correct answer data. While an example in which only one fourth trained model 84 is used by the state change detection unit 70E is given here, this is merely an example. For example, a fourth trained model 84 selected from multiple fourth trained models 84 may be used by the state change detection unit 70E. In this case, each fourth trained model 84 is created by performing machine learning specialized for each type of endoscopic examination, and the fourth trained model 84 corresponding to the type of endoscopic examination currently being performed is selected and used by the state change detection unit 70E.

[0098] The state change of the observation region 21 also includes a change in the region of interest 21A (e.g., a change in the region of interest 21A due to the delivery of the fluid 56). A change in the region of interest 21A can be, for example, a change from a state in which mucus is attached to the region of interest 21A to a state in which a non-neoplastic polyp is emerging from the region of interest 21A. The state change information 96 includes mucus information 96A, mucosa information 96B, and polyp information 96C. The mucus information 96A is information related to changes in mucus. Examples of the mucus information 96A include information indicating changes in the state of mucus attachment and / or information indicating changes in the color of the mucus. The mucosa information 96B is information related to changes in the state of the mucosa. Examples of the mucosa information 96B include information indicating changes in the mucosal structure and / or information indicating changes in the color of the mucosa. The polyp information 96C is information related to polyps. The polyp information 96C is, for example, information indicating a change from a state in which mucus is attached to the region of interest 21A to a state in which a polyp (for example, a non-neoplastic polyp or a neoplastic polyp) appears from the region of interest 21A.

[0099] The state change detection unit 70E acquires a time-series image group 89 and inputs the acquired time-series image group 89 to the fourth trained model 84. The state change detection unit 70E also inputs region of interest information 92 acquired by the region of interest detection unit 70C to the fourth trained model 84. The state change detection unit 70E also inputs endoscope information 94 acquired by the endoscope detection unit 70D to the fourth trained model 84. In this manner, when the plurality of time-series endoscopic images 40, region of interest information 92, and endoscope information 94 are input to the fourth trained model 84, the fourth trained model 84 outputs state change information 96 corresponding to the input plurality of time-series endoscopic images 40, region of interest information 92, and endoscope information 94. The state change detection unit 70E acquires the state change information 96 output from the fourth trained model 84. The state change information 96 acquired by the state change detection unit 70E is information relating to a state change in the observation area 21 currently being observed using the endoscope 12.

[0100] That is, the state change detection unit 70E detects a state change by performing AI-based image recognition processing based on the region of interest information 92 and the endoscope information 94. The region of interest information 92 is information obtained by image recognition processing using the second trained model 80 selected based on the body part information 90 (see FIG. 5 ). Therefore, it can be said that the detection of a state change by the state change detection unit 70E is indirectly achieved by performing AI-based image recognition processing based on the body part information 90.

[0101] When the second condition is satisfied, the control unit 70F outputs a termination instruction signal 98 to the state change detection unit 70E. The second condition may include, for example, a condition that a termination instruction has been given to the endoscope 12 (in the example shown in FIG. 7, a condition that a termination instruction has been accepted by the accepting device 62). The termination instruction refers to an instruction to terminate the AI-based image recognition processing.

[0102] When the state change detection unit 70E receives an end instruction signal 98 from the control unit 70F, the state change detection unit 70E ends the AI-based image recognition process (in the example shown in FIG. 7 , the image recognition process using the fourth trained model 84) and erases the state change information 96. The state change information 96 is an example of the “first information that is information based on the image recognition process” according to the technology of the present disclosure.

[0103] When the second condition is satisfied, the control unit 70F also outputs an end instruction signal 98 to the region of interest detection unit 70C and the endoscope detection unit 70D. When the end instruction signal 98 is input from the control unit 70F, the part detection unit 70B ends the AI-based image recognition process (in the example shown in FIG. 5, the image recognition process using the first trained model 78) and erases the part information 90 (see FIG. 5). When the end instruction signal 98 is input from the control unit 70F, the region of interest detection unit 70C ends the AI-based image recognition process (in the example shown in FIG. 5, the image recognition process using the second trained model 80) and erases the region of interest information 92 (see FIG. 5). When the end instruction signal 98 is input from the control unit 70F, the endoscope detection unit 70D ends the image recognition process (in the example shown in FIG. 5, the image recognition process using the third trained model 82) and erases the endoscope information 94 (see FIG. 6).

[0104] In this way, from the time when the processor 70 starts the AI-based image recognition processing until the time when it finishes it, the information based on the AI-based image recognition processing (i.e., the part information 90, the region of interest information 92, the endoscope information 94, and the status change information 96) that is held by the processor 70 is erased when the AI-based image recognition processing ends (here, as an example, when the second condition is satisfied).

[0105] When the second condition is satisfied, the control unit 70F also outputs an end instruction signal 98 to the image acquisition unit 70A. When the end instruction signal 98 is input from the control unit 70F, the image acquisition unit 70A erases the currently held time-series image group 89. In other words, the information used in the AI-based image recognition processing (i.e., the time-series image group 89) is also erased when the AI-based image recognition processing is terminated (here, as an example, when the second condition is satisfied).

[0106] The image recognition processing performed using the first trained model 78 (see Figure 5), the image recognition processing performed using the second trained model 80 (see Figure 5), the image recognition processing performed using the third trained model 82 (see Figure 6), and the image recognition processing performed using the fourth trained model 84 (see Figure 7) are examples of "image recognition processing" related to the technology disclosed herein.

[0107] As an example, as shown in FIG. 8 , the lesion information derivation unit 70G derives lesion information 102 regarding a lesion in the observation region 21 based on the state change detected by the state change detection unit 70E. Examples of the lesion information 102 include information indicating the presence or absence of a lesion and / or information indicating the type of lesion. In this case, for example, the lesion information derivation unit 70G acquires state change information 96 from the state change detection unit 70E and derives the lesion information 102 by performing an AI-based derivation process using the acquired state change information 96. That is, the lesion information derivation unit 70G derives the lesion information 102 by performing an AI-based derivation process using the fifth trained model 86 on the state change information 96.

[0108] The fifth trained model 86 is optimized by performing machine learning on the neural network using fifth training data, such as training data in which the condition change information 96 is used as example data and the lesion information 102 is used as correct answer data.

[0109] The lesion information derivation unit 70G acquires state change information 96 from the state change detection unit 70E and inputs the acquired state change information 96 to the fifth trained model 86. As a result, the fifth trained model 86 outputs lesion information 102 corresponding to the input state change information 96. The lesion information derivation unit 70G acquires the lesion information 102 output from the fifth trained model 86. The lesion information 102 acquired by the lesion information derivation unit 70G is information regarding a lesion in the observation target region 21 currently being observed using the endoscope 12 (e.g., a lesion relative to the region of interest 21A included in the observation target region 21).

[0110] The control unit 70F outputs information based on the lesion information 102 derived by the lesion information derivation unit 70G. The output destination is the display device 13. That is, the control unit 70F displays the information based on the lesion information 102 derived by the lesion information derivation unit 70G on the screen 36 of the display device 13. A first example of the information based on the lesion information 102 is information indicating the presence or absence of a lesion and information indicating the type of lesion. A second example of the information based on the lesion information 102 is information derived from the lesion information 102 (e.g., information indicating the reliability of the lesion information 102). In the example shown in FIG. 8 , the information based on the lesion information 102 is displayed adjacent to the endoscopic image 40 on the screen 36. In the example shown in FIG. 8 , a detection frame 103A surrounding an image region 103 indicating the region of interest 21A in the endoscopic image 40 is also displayed on the screen 36. For example, the detection frame 103A is a frame formed based on a bounding box used in AI-based image recognition processing by the region of interest detection unit 70C. The display device 13 is an example of an "output destination" and a "display device" according to the technology of the present disclosure, and the information based on the lesion information 102 is an example of "information based on image recognition processing" and "second information" according to the technology of the present disclosure.

[0111] Here, the display device 13 is exemplified as an output destination of information based on the lesion information 102, but the technology of the present disclosure is not limited to this, and the output destination of information based on the lesion information 102 may be, for example, an information processing device such as a server to which the endoscope system 10 is communicably connected. Furthermore, the information based on the lesion information 102 may be stored in a storage medium (for example, the NVM 74 and / or a memory of a device provided outside the endoscope 12). Furthermore, the information based on the lesion information 102 may be registered in an electronic medical record.

[0112] Next, the operation of the portion of the endoscope system 10 related to the technique of the present disclosure will be described with reference to FIGS. 9A and 9B.

[0113] 9A and 9B show an example of the flow of medical support processing performed by the processor 70. The flow of medical support processing shown in Fig. 9A and 9B is an example of an "image processing method" according to the technique of the present disclosure.

[0114] 9A , first, in step ST10, the image acquisition unit 70A determines whether or not one frame of image has been captured by the camera 48. If one frame of image has not been captured by the camera 48 in step ST10, the determination is negative, and the determination in step ST10 is made again. If one frame of image has been captured by the camera 48 in step ST10, the determination is positive, and the medical support process proceeds to step ST12.

[0115] In step ST12, the image acquisition unit 70A acquires one frame of the endoscopic image 40 from the camera 48. After the processing of step ST12 is executed, the medical support processing proceeds to step ST14.

[0116] In step ST14, the image acquisition unit 70A determines whether or not it holds a certain number of frames of endoscopic images 40. If it does not hold the certain number of frames of endoscopic images 40 in step ST14, the determination is negative, and the medical support processing proceeds to step ST10. If it does hold the certain number of frames of endoscopic images 40 in step ST14, the determination is positive, and the medical support processing proceeds to step ST16.

[0117] In step ST16, the image acquisition unit 70A adds the endoscopic images 40 acquired in step ST12 to the time-series image group 89 in a FIFO manner, thereby updating the time-series image group 89. After the processing of step ST16 is executed, the medical support processing proceeds to step ST18.

[0118] In step ST18, the control unit 70F determines whether the first condition is satisfied. If the first condition is not satisfied in step ST18, the determination is negative, and the medical support process proceeds to step ST10. If the first condition is satisfied in step ST18, the determination is positive, and the medical support process proceeds to step ST20.

[0119] In step ST20, the body part detection unit 70B acquires body part information 90 by performing image recognition processing using the first trained model 78 on the time-series image group 89 held by the image acquisition unit 70A. After the processing of step ST20 is executed, the medical support processing proceeds to step ST22.

[0120] In step ST22, the region of interest detection unit 70C selects the second trained model 80 corresponding to the body part information 90 acquired in step ST20 from the multiple second trained models 80. After the processing of step ST22 is executed, the medical support processing proceeds to step ST24.

[0121] In step ST24, the region-of-interest detection unit 70C performs image recognition processing using the second trained model selected in step ST22 on the time-series image group 89 held by the image acquisition unit 70A, thereby acquiring region-of-interest information 92. After the processing of step ST24 is executed, the medical support processing proceeds to step ST26 shown in FIG.

[0122] In step ST26, the endoscope detection unit 70D acquires endoscope information 94 by performing image recognition processing using the third trained model 82 on the time-series image group 89 held by the image acquisition unit 70A. After the processing of step ST26 is executed, the medical support processing proceeds to step ST28.

[0123] In step ST28, the state change detection unit 70E inputs the time-series image group 89 held by the image acquisition unit 70A, the region of interest information 92 acquired in step ST24, and the endoscope information 94 acquired in step ST26 into the fourth trained model 84, thereby acquiring state change information 96 from the fourth trained model 84. After the processing of step ST28 is executed, the medical support processing proceeds to step ST30.

[0124] In step ST30, the control unit 70F determines whether the second condition is satisfied. If the second condition is not satisfied in step ST30, the determination is negative, and the medical support process proceeds to step ST34. If the second condition is satisfied in step ST30, the determination is positive, and the medical support process proceeds to step ST32.

[0125] In step ST32, the processor 70 terminates the AI-based image recognition process and erases information based on the image recognition process. Specifically, the part detection unit 70B terminates the AI-based image recognition process (in the example shown in FIG. 5 , image recognition process using the first trained model 78) and erases the part information 90 (see FIG. 5 ). The region of interest detection unit 70C terminates the AI-based image recognition process (in the example shown in FIG. 5 , image recognition process using the second trained model 80) and erases the region of interest information 92 (see FIG. 5 ). The endoscope detection unit 70D terminates the image recognition process using the third trained model 82 and erases the endoscope information 94 (see FIG. 6 ). The image acquisition unit 70A erases the time-series image group 89 currently held. After the process of step ST32 is executed, the medical support process proceeds to step ST34.

[0126] In step ST34, the lesion information derivation unit 70G inputs the condition change information 96 acquired in step ST28 into the fifth trained model 86, thereby deriving lesion information 102 corresponding to the condition change information 96 from the fifth trained model 86. After the processing of step ST34 is executed, the medical support processing proceeds to step ST36.

[0127] In step ST36, the control unit 70F displays information based on the lesion information 102 derived in step ST34 on the screen 36 of the display device 13. After the processing of step ST36 is executed, the medical support processing proceeds to step ST38.

[0128] In step ST38, the control unit 70F determines whether a condition for terminating the medical support process is satisfied. One example of the condition for terminating the medical support process is that an instruction to terminate the medical support process has been given to the endoscope system 10 (for example, that an instruction to terminate the medical support process has been accepted by the acceptance device 62).

[0129] In step ST38, if the condition for terminating the medical support process is not satisfied, the determination is negative, and the medical support process proceeds to step ST10 shown in Fig. 9A. In step ST38, if the condition for terminating the medical support process is satisfied, the determination is positive, and the medical support process ends.

[0130] As described above, in the endoscopic system 10, a change in the state of the observation region 21 is detected by performing AI image recognition processing on the time-series image group 89 (e.g., a plurality of time-series endoscopic images 40 that can identify the state of the observation region 21 before and after the delivery of the fluid 56) generated by the camera 48. Therefore, the change in the state of the observation region 21 can be detected with higher accuracy than when the change in the state of the observation region 21 is detected using only a single endoscopic image 40 (e.g., only a single endoscopic image 40 obtained at a time when the fluid 56 is not being delivered). In this way, if the change in the state of the observation region 21 can be detected with high accuracy, the doctor 14 can, for example, accurately distinguish between serrated lesions and Barrett's esophageal cancer, which are generally known to be difficult to distinguish.

[0131] Furthermore, in the endoscope system 10, the state change detection unit 70E detects a change in adhesive color, a change in the mucosal state including the mucosal structure, and / or a change in the mucus adhesion state as a state change in the observation region 21. Therefore, compared to detecting a change in adhesive color, a change in the mucosal state including the mucosal structure, and / or a change in the mucus adhesion state using only a single endoscopic image 40, a change in adhesive color, a change in the mucosal state including the mucosal structure, and / or a change in the mucus adhesion state can be detected with high accuracy as a state change in the observation region 21.

[0132] Furthermore, in the endoscope system 10, the endoscope detection unit 70D performs AI image recognition processing on the time-series image group 89 to acquire endoscope information 94. Then, the state change detection unit 70E detects a state change based on the endoscope information 94. Therefore, the state change can be detected with higher accuracy than when the state change is detected without any consideration of the endoscope information 94. Furthermore, the endoscope information 94 includes fluid delivery information 94C. Therefore, the state change can be detected with higher accuracy than when the state change is detected without any consideration of the fluid delivery information 94C.

[0133] Furthermore, in the endoscope system 10, the AI-based image recognition processing is started when the first condition is satisfied. Therefore, compared to when the AI-based image recognition processing is started unconditionally, the AI-based image recognition processing can be started at a more suitable timing. Furthermore, in this embodiment, the first condition is a condition that a start instruction has been given to the endoscope 12 (in the example shown in FIG. 7, a condition that a start instruction has been accepted by the acceptance device 62). Therefore, the AI-based image recognition processing can be started at a timing intended by the doctor 14.

[0134] Furthermore, in the endoscope system 10, the AI-based image recognition processing ends when the second condition is satisfied. Therefore, compared to when the AI-based image recognition processing ends unconditionally, the AI-based image recognition processing can be ended at a more appropriate timing. Furthermore, in this embodiment, a condition including a condition that an end instruction has been given to the endoscope 12 (in the example shown in FIG. 7 , a condition that an end instruction has been accepted by the acceptance device 62) is applied as the second condition. Therefore, the AI-based image recognition processing can be ended at a timing intended by the doctor 14.

[0135] Furthermore, in the endoscope system 10, when the second condition is satisfied, information based on the AI ​​image recognition processing is deleted. Therefore, compared to when information based on the AI ​​image recognition processing is deleted unconditionally, it is possible to prevent information based on the AI ​​image recognition processing from being deleted at a time unintended by the doctor 14.

[0136] Furthermore, in the endoscope system 10, information based on the AI-based image recognition processing is retained from the start to the end of the AI-based image recognition processing, and the information based on the AI-based image recognition processing is deleted when the AI-based image recognition processing is completed. Therefore, processing using the information based on the AI-based image recognition processing (e.g., processing by the lesion information derivation unit 70G) can be performed until the AI-based image recognition processing is completed, and the information based on the AI-based image recognition processing can be deleted when the AI-based image recognition processing is no longer necessary.

[0137] Furthermore, in the endoscope system 10, the state change detection unit 70E detects a state change in the observation target region 21 on the condition that the region of interest information 92 has been acquired by the region of interest detection unit 70C. Therefore, compared to detecting a state change in the observation target region 21 before the region of interest information 92 is detected, it is possible to detect a state change in the observation target region 21 at a more suitable timing (for example, the timing when the region of interest 21A is included in the observation target region 21). In other words, it is possible to prevent a state change in the observation target region 21 from being detected when the region of interest 21A is not included in the observation target region 21, for example.

[0138] Furthermore, in the endoscopic system 10, the state change detection unit 70E detects a state change in the observation target area 21 on the condition that the area information 90 has been acquired by the area detection unit 70B. Therefore, compared to detecting a state change in the observation target area 21 before the area information 90 is detected, it is possible to detect a state change in the observation target area 21 at a more suitable timing (for example, at the timing when a specified area appears in the endoscopic image 40 as an area corresponding to the observation target area 21). In other words, for example, if the area appearing in the endoscopic image 40 is a different area from the specified area, it is possible to prevent a state change in the observation target area 21 from being detected.

[0139] Furthermore, in the endoscope system 10, the lesion information deriving unit 70G derives the lesion information 102 based on the state change detected by the state change detecting unit 70E. Therefore, compared to a case where a state change in the observation region 21 is predicted using only a single endoscopic image 40 and the lesion information 102 is derived based on the predicted state change, it is possible to derive more reliable lesion information 102.

[0140] Furthermore, in the endoscope system 10, a change in the region of interest 21A is detected by the state change detection unit 70E. For example, the state change detection unit 70E detects a change in the region of interest 21A from a state in which mucus is attached to the region of interest 21A to a state in which a non-neoplastic polyp is emerging from the region of interest 21A. Therefore, compared to a case in which the change in the region of interest 21A from a state in which mucus is attached to the region of interest 21A to a state in which a non-neoplastic polyp is emerging from the region of interest 21A is predicted using only a single endoscopic image 40 and the lesion information 102 is derived based on the predicted change, the lesion information 102 can be derived with higher reliability.

[0141] Furthermore, in the endoscope system 10, the control unit 70F outputs information based on the lesion information 102. Examples of the output destination include the display device 13 and / or an information processing device (a server, a personal computer, or a tablet terminal). Therefore, processing using the information based on the lesion information 102 can be performed at the output destination of the information based on the lesion information 102.

[0142] Furthermore, in the endoscope system 10, the control unit 70F displays information based on the lesion information 102 on the screen 36 of the display device 13. Therefore, the information based on the lesion information 102 can be grasped by the user.

[0143] In the above embodiment, an example has been described in which the time-series image group 89, the region of interest information 92, and the endoscope information 94 are input to the fourth trained model 84, and the corresponding state change information 96 is output from the fourth trained model 84. However, the technology of the present disclosure is not limited to this. For example, the fourth trained model 84 may further receive site information 90, and the fourth trained model 84 may output state change information 96 corresponding to the site information 90, the time-series image group 89, the region of interest information 92, and the endoscope information 94. In this case, the fourth training data used to create the fourth trained model 84 may be training data in which, for example, multiple images obtained in time series by capturing images of the inside of the body with the camera 48, the site information 90, the region of interest information 92, and the endoscope information 94 are used as example data, and state change information 96 related to state changes in the observation target region 21 is used as correct answer data.

[0144] Furthermore, in addition to the time-series image group 89, site information 90, region of interest information 92, and / or endoscope information 94 may be input to the fourth trained model 84, and state change information 96 corresponding to the input may be output from the fourth trained model 84. In this case, too, as the fourth training data used to create the fourth trained model 84, training data in which the information input to the fourth trained model 84 is used as example data may be used in the same manner as above. Furthermore, the fourth training data used to create the fourth trained model 84 may be training data in which a plurality of images obtained in time series by capturing images of the inside of the body with the camera 48 are used as example data, and state change information 96 regarding state changes in the observation target region 21 is used as correct answer data. In this case, when the time-series image group 89 is input to the fourth trained model 84, state change information 96 corresponding to the input time-series image group 89 is output from the fourth trained model 84. That is, the state change detection unit 70E performs AI-based image recognition processing (i.e., image recognition processing using the fourth trained model 84) on the time-series image group 89, thereby detecting state changes in the observation target area 21.

[0145] In the above embodiment, an example was given in which AI-based image recognition processing was performed based on a start command given to the endoscope 12, but the technology of the present disclosure is not limited to this. For example, AI-based image recognition processing may be performed based on the operation of the endoscope 12. In this case, for example, the operation of the endoscope 12 is identified from the endoscope information 94.

[0146] Here, as an example of performing AI-based image recognition processing based on the operation of the endoscope 12, for example, as shown in FIG. 10 , the control unit 70F outputs a start instruction signal 91 to the part detection unit 70B, the region of interest detection unit 70C, the endoscope detection unit 70D, and the state change detection unit 70E when a first condition using the endoscope information 94 is satisfied. A first example of the first condition using the endoscope information 94 is a condition that the tip 46 of the endoscope 12 has stopped. Whether or not the tip 46 of the endoscope 12 has stopped is determined based on the motion speed information 94B. A second example of the first condition using the endoscope information 94 is a condition that the movement speed of the tip 46 of the endoscope 12 has decreased. Whether or not the movement speed of the tip 46 of the endoscope 12 has decreased is determined based on the motion speed information 94B. Here, a decrease in the movement speed refers to, for example, a decrease in the movement speed below a predetermined speed (e.g., a decrease from several tens of millimeters per second to several millimeters per second).

[0147] In this way, since AI-based image recognition processing is performed based on the operation of the endoscope 12, AI-based image recognition processing can be performed at more suitable timing than when AI-based image recognition processing is performed regardless of the operation of the endoscope 12. Furthermore, AI-based image recognition processing is started when the condition that the tip 46 of the endoscope 12 has stopped or the condition that the movement speed of the tip 46 of the endoscope 12 has decreased is satisfied. Therefore, AI-based image recognition processing can be started at more suitable timing than when AI-based image recognition processing is started regardless of the movement speed of the tip 46 of the endoscope 12.

[0148] Furthermore, the AI-based image recognition processing may be performed based on region of interest information 92. In this case, for example, as shown in FIG. 10 , when a first condition using the region of interest information 92 is satisfied, the control unit 70F outputs a start instruction signal 91 to the part detection unit 70B, the region of interest detection unit 70C, the endoscope detection unit 70D, and the state change detection unit 70E. A first example of the first condition using the region of interest information 92 is a condition that the region of interest 21A is included in the observation target region 21. Whether the region of interest 21A is included in the observation target region 21 is determined based on the region of interest information 92. A second example of the first condition using the region of interest information 92 is a condition that the observation target region 21 includes a specific region of interest 21A (e.g., a region with mucus attached). Whether the specific region of interest 21A is included in the observation target region 21 is determined based on the region of interest information 92.

[0149] As described above, since the AI-based image recognition processing is performed based on the region of interest information 92, the AI-based image recognition processing can be performed at a more suitable timing than when the AI-based image recognition processing is performed regardless of the region of interest information 92. Furthermore, the AI-based image recognition processing is started when the condition that the region of interest 21A is included in the observation target region 21 is satisfied. Therefore, the AI-based image recognition processing can be started at a more suitable timing than when the AI-based image recognition processing is started regardless of whether the region of interest 21A is included in the observation target region 21. In other words, for example, it is possible to prevent the image recognition processing from being started when the region of interest 21A is not included in the observation target region 21.

[0150] Furthermore, the AI-based image recognition processing may be performed based on region information 90. In this case, for example, as shown in FIG. 10 , when a first condition using the region information 90 is satisfied, the control unit 70F outputs a start instruction signal 91 to the region detection unit 70B, the region of interest detection unit 70C, the endoscope detection unit 70D, and the state change detection unit 70E. An example of the first condition using the region information 90 is that the region corresponding to the observation target region 21 is a region designated as an observation target (e.g., the ascending colon). The region may be designated, for example, via the reception device 62 or via a communication device capable of communicating with the endoscope system 10, and any method for designating the region may be used. Whether the region corresponding to the observation target region 21 is a region designated as an observation target is determined based on the region information 90.

[0151] As described above, since the AI-based image recognition processing is performed based on the region information 90, the AI-based image recognition processing can be performed at a more suitable timing than when the AI-based image recognition processing is performed regardless of the region information 90. Furthermore, the AI-based image recognition processing is started when the condition that the region corresponding to the observation region 21 is the region designated as the observation target is satisfied. Therefore, the AI-based image recognition processing can be started at a more suitable timing than when the AI-based image recognition processing is started regardless of the region corresponding to the observation region 21. In other words, for example, it is possible to prevent the image recognition processing from being started when the region corresponding to the observation region 21 is a region (e.g., the descending colon) different from the region designated as the observation target (e.g., the ascending colon).

[0152] Furthermore, the AI-based image recognition processing and the detection of a state change by the state change detection unit 70E may be performed based on a first medical instruction given to the endoscope 12. In this case, for example, as shown in FIG. 10 , when the control unit 70F satisfies a first condition using the first medical instruction, it outputs a start instruction signal 91 to the part detection unit 70B, the region of interest detection unit 70C, the endoscope detection unit 70D, and the state change detection unit 70E. The first medical instruction is an example of a "medical instruction" according to the technology of the present disclosure. The first medical instruction refers to, for example, an instruction given to the endoscope 12 via the operation unit 42 and / or the reception device 62. An example of the first medical instruction is an instruction to deliver fluid 56 from the treatment opening 52. In this way, since the AI-based image recognition processing is performed based on the first medical instruction given to the endoscope 12, the AI-based image recognition processing can be performed at a more appropriate timing than when the AI-based image recognition processing is performed regardless of whether or not the first medical instruction is given to the endoscope 12. Furthermore, compared to when the state change detection unit 70E detects a state change regardless of whether or not a first medical instruction is given to the endoscope 12, the state change can be detected at a more suitable timing.

[0153] In addition, the control unit 70F may be configured to output a start instruction signal 91 when multiple pre-specified first conditions are satisfied, including the first condition using the start instruction described in the above embodiment, the first condition using the endoscope information 94, the first condition using the region of interest information 92, the first condition using the part information 90, and the first condition using the first medical instruction.

[0154] In the above embodiment, an example was given in which the AI-based image recognition processing is terminated based on a termination instruction given to the endoscope 12, but the technology of the present disclosure is not limited to this. For example, the AI-based image recognition processing may be terminated based on the operation of the endoscope 12. In this case, for example, the operation of the endoscope 12 is identified from the endoscope information 94.

[0155] Here, an example of terminating the AI-based image recognition processing based on the operation of the endoscope 12 is, for example, as shown in FIG. 10 , an embodiment in which the control unit 70F outputs a termination instruction signal 98 to the part detection unit 70B, the region of interest detection unit 70C, the endoscope detection unit 70D, and the state change detection unit 70E when a second condition using the endoscope information 94 is satisfied. A first example of the second condition using the endoscope information 94 is a condition that the tip 46 of the endoscope 12 has started. Whether or not the tip 46 of the endoscope 12 has started is determined based on the operation speed information 94B. A second example of the first condition using the endoscope information 94 is a condition that the movement speed of the tip 46 of the endoscope 12 has increased. Whether or not the movement speed of the tip 46 of the endoscope 12 has increased is determined based on the operation speed information 94B. Here, an increase in the movement speed refers to, for example, an increase in the movement speed above a predetermined speed (e.g., an increase from several millimeters per second to several tens of millimeters per second).

[0156] In this way, since the AI-based image recognition processing is terminated based on the operation of the endoscope 12, it is possible to terminate the AI-based image recognition processing at a more suitable timing than when the AI-based image recognition processing is terminated regardless of the operation of the endoscope 12. Furthermore, the AI-based image recognition processing is terminated when the condition that the tip 46 of the endoscope 12 has started or the condition that the movement speed of the tip 46 of the endoscope 12 has increased is satisfied. Therefore, it is possible to terminate the AI-based image recognition processing at a more suitable timing than when the AI-based image recognition processing is terminated regardless of the movement speed of the tip 46 of the endoscope 12.

[0157] Alternatively, the AI-based image recognition processing may be terminated based on the region of interest information 92. In this case, for example, as shown in FIG. 10 , when a second condition using the region of interest information 92 is satisfied, the control unit 70F outputs a termination instruction signal 98 to the part detection unit 70B, the region of interest detection unit 70C, the endoscope detection unit 70D, and the state change detection unit 70E. A first example of the second condition using the region of interest information 92 is a condition that the region of interest 21A is not included in the observation region 21. Whether the region of interest 21A is not included in the observation region 21 is determined based on the region of interest information 92. A second example of the second condition using the region of interest information 92 is a condition that the observation region 21 does not include a specific region of interest 21A (e.g., a region with mucus attached). Whether the observation region 21 does not include a specific region of interest 21A is determined based on the region of interest information 92.

[0158] In this way, since the AI-based image recognition processing is terminated based on the region of interest information 92, it is possible to terminate the AI-based image recognition processing at a more appropriate timing than when the AI-based image recognition processing is terminated regardless of the region of interest information 92. Furthermore, the AI-based image recognition processing is terminated when the condition that the region of interest 21A is not included in the observation target region 21 is satisfied. Therefore, it is possible to terminate the AI-based image recognition processing at a more appropriate timing than when the AI-based image recognition processing is terminated regardless of whether the region of interest 21A is included in the observation target region 21. In other words, it is possible to prevent the image recognition processing from being terminated when the region of interest 21A is included in the observation target region 21, for example.

[0159] Alternatively, the AI-based image recognition processing may be terminated based on the region information 90. In this case, for example, as shown in FIG. 10 , when a second condition using the region information 90 is satisfied, the control unit 70F outputs a termination instruction signal 98 to the region detection unit 70B, the region of interest detection unit 70C, the endoscope detection unit 70D, and the state change detection unit 70E. An example of the second condition using the region information 90 is that the region corresponding to the observation region 21 is a region (e.g., the descending colon) different from the region designated as the observation target (e.g., the ascending colon). Whether the region corresponding to the observation region 21 is a region different from the region designated as the observation target is determined based on the region information 90.

[0160] In this way, since the AI-based image recognition processing is terminated based on the region information 90, it is possible to terminate the AI-based image recognition processing at a more appropriate timing than when the AI-based image recognition processing is terminated regardless of the region information 90. Furthermore, the AI-based image recognition processing is terminated when the condition that the region corresponding to the observation target region 21 is a region different from the region designated as the observation target is satisfied. Therefore, it is possible to terminate the AI-based image recognition processing at a more appropriate timing than when the AI-based image recognition processing is terminated regardless of the region corresponding to the observation target region 21. In other words, it is possible to prevent the image recognition processing from being terminated when, for example, the region corresponding to the observation target region 21 is a region designated as the observation target (e.g., the ascending colon).

[0161] Furthermore, the AI-based image recognition processing may be performed based on a second medical instruction given to the endoscope 12. In this case, for example, as shown in FIG. 10 , when a second condition using the second medical instruction is satisfied, the control unit 70F outputs a termination instruction signal 98 to the part detection unit 70B, the region of interest detection unit 70C, the endoscope detection unit 70D, and the state change detection unit 70E. The second medical instruction refers to, for example, an instruction given to the endoscope 12 via the operation unit 42 and / or the reception device 62. An example of the second medical instruction is an instruction to stop the delivery of the fluid 56 from the treatment opening 52. In this way, since the AI-based image recognition processing is performed based on the second medical instruction given to the endoscope 12, it is possible to terminate the AI-based image recognition processing at a more appropriate timing than when the AI-based image recognition processing is terminated regardless of whether or not a second medical instruction is given to the endoscope 12.

[0162] In addition, the control unit 70F may be configured to output a termination instruction signal 98 when multiple pre-specified second conditions are satisfied, including the second condition using the termination instruction described in the above embodiment, the second condition using the endoscope information 94, the second condition using the region of interest information 92, the second condition using the part information 90, and the second condition using the second medical instruction.

[0163] 10, the control unit 70F may output a termination instruction signal 98 to the image acquisition unit 70A when one or more of the second conditions described above are satisfied. In this case, as in the above embodiment, the image acquisition unit 70A also deletes the time-series image group 89 currently being held.

[0164] In the above embodiment, an example has been described in which the lesion information derivation unit 70G derives the lesion information 102 by performing processing using the fifth trained model 86, but the technology of the present disclosure is not limited to this. For example, as shown in FIG. 11 , the lesion information derivation unit 70G may derive the lesion information 102 by performing processing using the sixth trained model 104.

[0165] The sixth trained model 104 is optimized by performing machine learning on the neural network using sixth training data. Examples of the sixth training data include training data in which the state change information 96 and the delivery amount information 106 are used as example data and the lesion information 102 is used as correct answer data. The delivery amount information 106 is an example of "delivery amount information" according to the technology of the present disclosure. The delivery amount information 106 is information indicating the delivery amount of the fluid 56 (see FIG. 2 ). For example, the delivery amount information 106 is included in the endoscope information 94. Examples of the delivery amount information 106 include air delivery amount information 94C1 and / or water delivery amount information 94C2. Note that the delivery amount information 106 may be information obtained by measuring the delivery amount using a sensor or the like, or may be information obtained from a value accepted by the accepting device 62.

[0166] The lesion information derivation unit 70G acquires the state change information 96 and the emission amount information 106, and inputs the acquired state change information 96 and emission amount information 106 to the sixth trained model 104. As a result, the sixth trained model 86 outputs lesion information 102 corresponding to the input state change information 96 and emission amount information 106. The lesion information derivation unit 70G acquires the lesion information 102 output from the sixth trained model 86. The lesion information 102 acquired by the lesion information derivation unit 70G is information relating to a lesion in the observation target region 21 currently being observed using the endoscope 12 (e.g., a lesion relative to the region of interest 21A included in the observation target region 21).

[0167] 11, the lesion information 102 is derived based on the condition change information 96 and the output amount information 106. Therefore, compared to the case where the lesion information 102 is derived without using the output amount information 106, the lesion information 102 can be derived with higher reliability.

[0168] In the above embodiment, an example has been described in which the lesion information derivation unit 70G derives the lesion information 102 on the condition that a change in the state of the observation region 21 has been detected by the state change detection unit 70E, but the condition for deriving the lesion information 102 is not limited to this. For example, as shown in Figure 12, the process of step ST50, which is a process for determining the condition for deriving the lesion information 102, may be inserted between the process of step ST32 and the process of step ST34 of the medical support process.

[0169] In step ST50 of the medical support process shown in FIG. 12, the lesion information derivation unit 70G determines whether or not the conditions for deriving the lesion information 102 (hereinafter referred to as "derivation conditions") are satisfied.

[0170] A first example of the derivation condition is a condition that the timing determined based on the operation of the endoscope 12 is the first timing. The first timing refers to, for example, the timing when the distal end portion 46 of the endoscope 12 starts moving or the timing when the moving speed of the distal end portion 46 of the endoscope 12 increases. For example, whether the timing determined based on the operation of the endoscope 12 is the first timing is determined based on the operation speed information 94B (see FIG. 6 ).

[0171] A second example of the derivation condition is a condition that the above-mentioned second medical instruction has been given to the endoscope 12. Here, the second medical instruction is an example of a "medical instruction" according to the technology of the present disclosure.

[0172] A third example of the derivation condition is that the region of interest information 92 is information about a specific region of interest 21A (e.g., a region where mucus is attached). Here, the specific region of interest 21A is an example of a “specific region of interest” according to the technology of the present disclosure.

[0173] A fourth example of the derivation condition is that the region information 90 is information about a specific region (for example, the ascending colon). Here, the specific region is an example of a “specific region” according to the technology of the present disclosure.

[0174] If the derivation conditions are not satisfied in step ST50, the determination is negative, and the medical support process shown in Fig. 12 proceeds to step ST38. If the derivation conditions are satisfied in step ST50, the determination is positive, and the medical support process shown in Fig. 12 proceeds to step ST34. Step ST34 is executed in the manner described in the above embodiment, and lesion information 102 is derived.

[0175] 12 , the lesion information 102 is derived when the derivation condition is satisfied, which is that the timing determined based on the operation of the endoscope 12 is the first timing. Therefore, compared to when the lesion information 102 is derived without taking into account the operation of the endoscope 12, the lesion information 102 for the observation region 21 intended by the user can be derived with high accuracy.

[0176] 12 , the lesion information 102 is derived when the derivation condition is satisfied, that is, the second medical instruction has been given to the endoscope 12. Therefore, compared to when the lesion information 102 is derived without taking into account the second medical instruction given to the endoscope 12, the lesion information 102 for the observation region 21 intended by the user can be derived with high accuracy.

[0177] 12 , the lesion information 102 is derived when the derivation condition that the region of interest information 92 is information relating to the specific region of interest 21A is satisfied, and the derivation condition that the second medical instruction described above is given to the endoscope 12 is satisfied. Therefore, compared to when the lesion information 102 is derived without taking the region of interest information 92 into consideration, the lesion information 102 for the specific region of interest 21A can be derived with high accuracy.

[0178] 12, lesion information 102 is derived when the derivation condition that the site information 90 is information about a specific site is satisfied. Therefore, compared to when the lesion information 102 is derived without taking the site information 90 into consideration, the lesion information 102 about a specific site can be derived with high accuracy.

[0179] The lesion information 102 derived by the lesion information derivation unit 70G may be determined by the control unit 70F in accordance with a given instruction (e.g., an instruction from the doctor 14). In this case, for example, as shown in FIG. 13 , the processing of step ST60 and the processing of step ST62 are inserted between the processing of step ST36 and the processing of step ST38 of the medical support processing.

[0180] In step ST60 shown in Fig. 13, the control unit 70F determines whether an instruction to confirm the lesion information 102 derived by the lesion information derivation unit 70G (hereinafter referred to as a "confirmation instruction") has been given to the endoscope 12 (for example, whether the confirmation instruction has been received by the reception device 62). If a confirmation instruction has not been given to the endoscope 12 in step ST60, the determination is negative, and the medical support processing shown in Fig. 13 proceeds to step ST38. If a confirmation instruction has been given to the endoscope 12 in step ST60, the determination is positive, and the medical support processing shown in Fig. 13 proceeds to step ST62.

[0181] In step ST62, the control unit 70F executes a confirmation process. A first example of the confirmation process is a process of displaying confirmation information on the screen 36 of the display device 13. An example of the confirmation information is a mark or character that allows the user to visually understand that the lesion information 102 has been confirmed. A second example of the confirmation process is a process of changing the display mode of information based on the lesion information 102 displayed on the screen 36. In this case, the display mode before and after the change allows the user to visually understand that the lesion information 102 has been confirmed. A third example of the confirmation process is a process of causing an audio playback device to play audio indicating that the lesion information 102 has been confirmed. A fourth example of the confirmation process is a process of registering the lesion information 102 in an electronic medical record or the like in association with the current time (e.g., the time when the confirmation instruction was received). After the process of step ST62 is executed, the medical support process shown in FIG. 13 proceeds to step ST38.

[0182] 13, the lesion information 102 derived by the lesion information derivation unit 70G is confirmed by the control unit 70F in accordance with the confirmation instruction, allowing the doctor 14 to confirm on the screen 36 that the lesion information 102 derived by the lesion information derivation unit 70G is correct before finalizing the lesion information 102.

[0183] In the above embodiment, an AI-based image recognition process is exemplified, but the technology of the present disclosure is not limited to this, and a non-AI-based image recognition process (e.g., an image recognition process using template matching) may be used instead of the AI-based image recognition process. Furthermore, the AI-based image recognition process and the non-AI-based image recognition process may be used together.

[0184] In the above embodiment, an example in which the lesion information 102 is derived by AI processing has been described, but the technology of the present disclosure is not limited to this, and the lesion information 102 may be derived by non-AI processing. In this case, for example, the lesion information 102 may be derived from a table in which the condition change information 96 and the lesion information 102 are associated with each other, or from an arithmetic expression in which a value indicating the condition change information 96 is an independent variable and a value indicating the lesion information 102 is a dependent variable.

[0185] In the above embodiment, an example was described in which medical support processing was performed by the processor 70 of the computer 64 included in the endoscope 12, but the technology of the present disclosure is not limited to this, and the device that performs medical support processing may be provided outside the endoscope 12. Examples of devices that may be provided outside the endoscope 12 include at least one server and / or at least one personal computer that are communicably connected to the endoscope 12. Furthermore, medical support processing may be performed in a distributed manner by multiple devices.

[0186] In the above embodiment, an example has been described in which the medical support processing program 76 is stored in the NVM 74, but the technology of the present disclosure is not limited to this. For example, the medical support processing program 76 may be stored in a portable non-transitory storage medium such as an SSD or USB memory. The medical support processing program 76 stored in the non-transitory storage medium is installed in the computer 64 of the endoscope 12. The processor 70 executes medical support processing in accordance with the medical support processing program 76.

[0187] Alternatively, the medical support processing program 76 may be stored in a storage device such as another computer or server connected to the endoscope 12 via a network, and the medical support processing program 76 may be downloaded and installed on the computer 64 in response to a request from the endoscope 12.

[0188] It is not necessary to store the entire medical support processing program 76 in the storage device of another computer or server device connected to the endoscope 12, or in the NVM 74; only a portion of the medical support processing program 76 may be stored therein.

[0189] The hardware resources that execute the medical support processing can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes medical support processing by executing software, i.e., a program. Examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with a circuit configuration specifically designed to execute specific processing. Each processor has built-in or connected memory, and executes medical support processing by using the memory.

[0190] The hardware resource for executing the medical support processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the medical support processing may be a single processor.

[0191] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute medical support processing, on a single IC chip, as typified by SoC. In this way, medical support processing is realized using one or more of the various processors described above as hardware resources.

[0192] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0193] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0194] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0195] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A processor is provided. The processor, Obtain multiple medical images of the target area in a time series, A change in state of the observation target region is detected by performing image recognition processing on the plurality of medical images. Image processing device.

2. The change in state includes a change in adhesive color, a change in mucosal state including mucosal structure, and / or a change in mucus adhesion state. The image processing device according to claim 1 .

3. The processor starts the image recognition process when a first condition is satisfied. The image processing device according to claim 1 .

4. The first condition includes a condition that an instruction to start the image recognition process is given. The image processing device according to claim 3 .

5. The first condition includes a condition that a region of interest is included in the observation target region. The image processing device according to claim 3 .

6. The first condition includes a condition that a portion corresponding to the observation target region is a portion designated as an observation target. The image processing device according to claim 3 .

7. The processor terminates the image recognition process when a second condition is satisfied. The image processing device according to claim 1 .

8. The processor erases the first information, which is information based on the image recognition processing, when the second condition is satisfied. The image processing device according to claim 7.

9. The first information is held from the start to the end of the image recognition process, The processor erases the first information when the image recognition process is completed. The image processing device according to claim 8.

10. When the plurality of medical images are generated by an endoscope, The second condition includes a condition that the distal end of the endoscope starts moving or a condition that the moving speed of the distal end increases. The image processing device according to claim 7.

11. The second condition includes a condition that an instruction to end the image recognition process is given. The image processing device according to claim 7.

12. The second condition includes a condition that the observation target area does not include a region of interest. The image processing device according to claim 7.

13. When the plurality of medical images are generated by an endoscope, The processor detects the state change based on a movement of the endoscope. The image processing device according to claim 1 .

14. The processor, acquiring region of interest information relating to a region of interest included in the observation target region; The state change is detected on the condition that the region of interest information is acquired. The image processing device according to claim 1 .

15. The processor, acquiring site information relating to a site corresponding to the observation target region; The state change is detected on the condition that the part information is acquired. The image processing device according to claim 1 .

16. The processor derives lesion information regarding a lesion in the observation region based on the state change. The image processing device according to claim 1 .

17. The observation region includes a region of interest; the state change includes a change in the region of interest; The change in the region of interest is a change from a state in which mucus is attached to the region of interest to a state in which a non-neoplastic polyp is emerging from the region of interest. The image processing device according to claim 16.

18. The observation region includes a region of interest; When the plurality of medical images are generated by an endoscope and a fluid is delivered from the endoscope into a body including the observation target region, the change in state includes a change in the region of interest associated with the delivery of the fluid; The processor, obtaining delivery rate information indicative of a delivery rate of the fluid; Deriving the pathology information based on the condition change and the delivery amount information. The image processing device according to claim 16.

19. The change in the region of interest is a change from a state in which mucus is attached to the region of interest to a state in which a non-neoplastic polyp is emerging from the region of interest. The image processing device according to claim 18.

20. An image processing device according to any one of claims 1 to 19, an endoscope body that is inserted into a body including the observation target area; Endoscope.