Medical assistance device, endoscope, medical assistance method, and program

Abstract

This medical assistance device comprises a processor. The processor recognizes a plurality of sites within a subject being observed on the basis of a plurality of medical images containing the subject being observed, and when the plurality of sites include an unrecognized site within the subject being observed, outputs unrecognized site information that can identify the presence of the unrecognized site.

Description

Medical support device, endoscope, medical support method, and program

[0001] The technology of the present disclosure relates to a medical support device, an endoscope, a medical support method, and a program.

[0002] International Publication No. 2021 / 176664 discloses an examination support system that includes an acquisition unit that acquires images captured by an imaging unit of an endoscope inside a patient's tubular organ and spatial arrangement information of the tip of the insertion unit of the endoscope, a presence rate calculation unit that calculates the presence rate of polyps in unobserved areas within the tubular organ identified based on at least the images and the spatial arrangement information, and an examination plan creation unit that creates an examination plan including a schedule for the next examination of the tubular organ based on at least the presence rate of polyps.

[0003] Japanese Patent Application Laid-Open Publication No. 2015-198928 discloses a medical image processing device that displays at least one medical image of a subject, and that includes a position detection unit that detects the position of a characteristic local structure of the human body from the medical image, a confirmation information determination unit that determines confirmation information that indicates the local structure to be confirmed, an interpretation judgment unit that determines whether the local structure to be confirmed that is indicated in the confirmation information has been interpreted based on the position of the local structure detected from the medical image, and a display unit that displays the judgment result of the interpretation judgment unit.

[0004] Japanese Patent Application Laid-Open Publication No. 2015-217120 discloses an image diagnosis support device that includes a display means for displaying a tomographic image obtained from a three-dimensional medical image on a display screen, a detection means for detecting the user's gaze position on the display screen, a determination means for determining an observed area in the tomographic image based on the gaze position detected by the detection means, and an identification means for identifying an observed area in the three-dimensional medical image based on the observed area in the tomographic image determined by the determination means.

[0005] One embodiment of the technique of the present disclosure provides a medical support device, an endoscope, a medical support method, and a program that can contribute to reducing oversight of recognition of a site within an observation object.

[0006] A first aspect of the technology of the present disclosure is a medical support device that includes a processor, which recognizes multiple parts within an object to be observed based on multiple medical images that show the object to be observed, and if an unrecognized part exists within the multiple parts within the object to be observed, outputs unrecognized information that can identify the existence of the unrecognized part.

[0007] A second aspect of the technology disclosed herein is a medical support device according to the first aspect, in which the multiple parts include a subsequent part that is scheduled to be recognized by the processor after the unrecognized part, and the processor outputs unrecognized information on the condition that it has recognized the subsequent part.

[0008] A third aspect of the technology of the present disclosure is a medical support device according to the first or second aspect, in which the processor outputs unrecognized information based on a first order, which is the order in which multiple parts are recognized by the processor, and a second order, which is the order in which multiple planned parts, including unrecognized parts, that are scheduled to be recognized by the processor, are recognized by the processor.

[0009] A fourth aspect of the technology of the present disclosure is a medical support device according to any one of the first to third aspects, in which importance is assigned to multiple parts and the unrecognized information includes importance information that allows the importance to be identified.

[0010] A fifth aspect of the technology of the present disclosure is the medical support device according to the fourth aspect, in which the importance is determined according to an instruction given from outside.

[0011] A sixth aspect of the technique of the present disclosure is the medical support device according to the fourth or fifth aspect, in which the importance is determined according to past examination data performed on a plurality of body parts.

[0012] A seventh aspect of the technology of the present disclosure is a medical support device according to any one of the fourth to sixth aspects, in which the importance is determined according to the position of the unrecognized area within the object of observation.

[0013] The eighth aspect of the technology of the present disclosure is a medical support device according to any one of the fourth to seventh aspects, in which the importance corresponding to a part of the multiple parts that is scheduled to be recognized by the processor before a specified part is higher than the importance corresponding to a part of the multiple parts that is scheduled to be recognized after the specified part.

[0014] A ninth aspect of the technology of the present disclosure is a medical support device according to any one of the fourth to eighth aspects, in which the importance of a part of the multiple parts that is determined to be a part that is typically prone to being overlooked is higher than the importance of a part of the multiple parts that is determined to be a part that is typically unlikely to be overlooked.

[0015] A tenth aspect of the technology of the present disclosure is a medical support device according to any one of the fourth to ninth aspects, in which a plurality of body parts are classified into major categories and minor categories within the major categories, and the importance of a body part classified into a minor category among the plurality of body parts is higher than the importance of a body part classified into a major category among the plurality of body parts.

[0016] An eleventh aspect of the technology of the present disclosure is a medical support device according to any one of the first to tenth aspects, in which a plurality of parts are classified into major categories and minor categories within the major categories, and an unrecognized part is a part classified into a minor category among the plurality of parts.

[0017] A twelfth aspect of the technology of the present disclosure is a medical support device according to an eleventh aspect, in which the major categories are divided into a first major category and a second major category, the parts classified in the second major category are scheduled to be recognized by the processor later than the parts classified in the first major category, the unrecognized parts are parts that belong to a minor category included in the first major category among the multiple parts, and the processor outputs unrecognized information on the condition that it has recognized a part classified in the second major category among the multiple parts.

[0018] A thirteenth aspect of the technology of the present disclosure is a medical support device according to the eleventh or twelfth aspect, in which the plurality of parts include a plurality of small classification parts classified into small classifications, the plurality of small classification parts include a first small classification part and a second small classification part that is scheduled to be recognized by the processor after the first small classification part, the unrecognized part is the first small classification part, and the processor outputs unrecognized information on the condition that it has recognized the second small classification part.

[0019] A fourteenth aspect of the technology of the present disclosure is a medical support device according to the eleventh or twelfth aspect, in which the plurality of parts include a plurality of sub-categorization parts belonging to a sub-category, the plurality of sub-categorization parts include a first sub-categorization part and a plurality of second sub-categorization parts that are scheduled to be recognized by the processor after the first sub-categorization part, the unrecognized part is the first sub-categorization part, and the processor outputs unrecognized information on the condition that it has recognized the plurality of second sub-categorization parts.

[0020] A fifteenth aspect of the technology of the present disclosure is the medical support device according to any one of the first to fourteenth aspects, in which an output destination of the unrecognized information includes a display device.

[0021] A sixteenth aspect of the technology of the present disclosure is a medical support device according to the fifteenth aspect, in which the unrecognized information includes a first image capable of identifying an unrecognized part and a second image capable of identifying other parts of the multiple parts other than the unrecognized part, and the first image and the second image are displayed on the display device in a manner that allows them to be distinguished.

[0022] A seventeenth aspect of the technology of the present disclosure is a medical support device according to the sixteenth aspect, in which the display device displays a schematic diagram in which the object to be observed is divided into multiple regions corresponding to multiple parts, and the first image and the second image are displayed in a distinguishable manner within the schematic diagram.

[0023] An 18th aspect of the technology of the present disclosure is a medical support device according to the 17th aspect, in which the object to be observed is a hollow organ, and the schematic diagram is a first schematic diagram showing a schematic aspect of at least one path for observing the hollow organ, a second schematic diagram showing a schematic aspect of the hollow organ viewed through, and / or a third schematic diagram showing an aspect of the hollow organ when it is expanded.

[0024] A 19th aspect of the technology of the present disclosure is a medical support device according to any one of the first to eighteenth aspects, in which the first image is displayed on the display device in a state where it is more emphasized than the second image.

[0025] A 20th aspect of the technology of the present disclosure is a medical support device according to any of the 16th to 19th aspects, in which importance is assigned to multiple parts and the display manner of the first image differs depending on the importance.

[0026] A twenty-first aspect of the technology of the present disclosure is a medical support device according to any one of the sixteenth to twentieth aspects, in which the display mode of the first image differs depending on the type of unrecognized part.

[0027] A 22nd aspect of the technology of the present disclosure is a medical support device according to any one of the first to 21st aspects, in which the medical images are images obtained from an endoscope inserted into the body, and when the processor recognizes the first portion upstream in the insertion direction of the endoscope inserted into the body in order from a first portion downstream in the insertion direction to a second portion downstream in the insertion direction, the processor outputs unrecognized information according to a first path defined from the upstream side to the downstream side in the insertion direction, and when the processor recognizes the third portion downstream in the insertion direction to a fourth portion upstream in the insertion direction in order, the processor outputs unrecognized information according to a second path defined from the downstream side to the upstream side in the insertion direction.

[0028] A 23rd aspect of the technology of the present disclosure is an endoscope comprising a medical support device according to any one of the first to 22nd aspects and an image acquisition device that acquires endoscopic images as medical images.

[0029] A 24th aspect of the technology of the present disclosure is a medical support method that includes recognizing multiple parts within an object to be observed based on multiple medical images that show the object to be observed, and, if an unrecognized part exists within the object to be observed in the multiple parts, outputting unrecognized information that can identify the existence of the unrecognized part.

[0030] A 25th aspect of the technology of the present disclosure is a program for causing a computer to execute processing including recognizing multiple parts within an object to be observed based on multiple medical images showing the object to be observed, and, if an unrecognized part exists within the object to be observed in the multiple parts, outputting unrecognized information that can identify the existence of the unrecognized part.

[0031] 1 is a conceptual diagram showing an example of an aspect in which an endoscope system is used. FIG. 1 is a conceptual diagram showing an example of the overall configuration of an endoscope system. FIG. 2 is a block diagram showing an example of the hardware configuration of the electrical system of an endoscope system. FIG. 3 is a block diagram showing an example of the main functions of a processor included in an endoscope. FIG. 4 is a conceptual diagram showing an example of the correlation between a camera, an NVM, an image acquisition unit, and a recognition unit. FIG. 5 is a conceptual diagram showing an example of the configuration of a recognition portion confirmation table. FIG. 6 is a conceptual diagram showing an example of the configuration of an importance table. FIG. 7 is a conceptual diagram showing an example of the correlation between a control unit and a display device. FIG. 8 is a conceptual diagram showing an example of a medical support image displayed on the screen of a display device. FIG. 9 is a flowchart showing an example of the flow of medical support processing. FIG. 10 is a conceptual diagram showing a first modified example of a medical support image displayed on the screen of a display device. FIG. 11 is a conceptual diagram showing a second modified example of a medical support image displayed on the screen of a display device.

[0032] Hereinafter, exemplary embodiments of a medical support device, an endoscope, a medical support method, and a program according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0033] First, the terms used in the following description will be explained.

[0034] CPU is an abbreviation for "Central Processing Unit." GPU is an abbreviation for "Graphics Processing Unit." RAM is an abbreviation for "Random Access Memory." NVM is an abbreviation for "Non-volatile memory." EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory." ASIC is an abbreviation for "Application Specific Integrated Circuit." PLD is an abbreviation for "Programmable Logic Device." FPGA is an abbreviation for "Field-Programmable Gate Array." SoC is an abbreviation for "System-on-a-chip." SSD is an abbreviation for "Solid State Drive." USB is an abbreviation for "Universal Serial Bus." HDD is an abbreviation for "Hard Disk Drive." EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". I / F is an abbreviation for "Interface". FIFO is an abbreviation for "First In First Out".

[0035] 1, an endoscopic system 10 includes an endoscope 12 and a display device 13. The endoscope 12 is used by a doctor 14 in an endoscopic examination. The endoscope 12 is communicably connected to a communication device (not shown), and information obtained by the endoscope 12 is transmitted to the communication device. The communication device receives the information transmitted from the endoscope 12 and executes processing using the received information (for example, processing to record the information in an electronic medical record, etc.).

[0036] The endoscope 12 includes an endoscope body 18. The endoscope 12 is a device for performing medical treatment on an observation target 21 (e.g., the upper digestive tract) contained within the body of a subject 20 (e.g., a patient) using the endoscope body 18. The observation target 21 is an object to be observed by a doctor 14. The endoscope body 18 is inserted into the body of the subject 20. The endoscope 12 causes the endoscope body 18 inserted into the body of the subject 20 to capture images of the observation target 21 inside the body of the subject 20, and performs various medical procedures on the observation target 21 as necessary. The endoscope 12 is an example of an "endoscope" according to the technology of the present disclosure.

[0037] The endoscope 12 captures images of the inside of the body of the subject 20, thereby acquiring and outputting images showing the state of the inside of the body. In the example shown in Fig. 1, an upper endoscope is shown as an example of the endoscope 12. Note that the upper endoscope is merely an example, and the technology of the present disclosure can be applied even if the endoscope 12 is another type of endoscope, such as a lower gastrointestinal endoscope or a bronchial endoscope.

[0038] Furthermore, in this embodiment, the endoscope 12 is an endoscope having an optical imaging function that captures reflected light obtained by irradiating light inside the body and reflecting it off the object of observation 21. However, this is merely one example, and the technology of the present disclosure also applies even if the endoscope 12 is an ultrasound endoscope. Furthermore, the technology of the present disclosure also applies even if a modality that generates frames for examination or surgery (e.g., radiographic images obtained by imaging using radiation, or ultrasound images based on reflected waves of ultrasound emitted from outside the body of the subject 20) is used instead of the endoscope 12. Note that frames obtained for examination or surgery are an example of a "medical image" according to the technology of the present disclosure.

[0039] The endoscope 12 is equipped with a control device 22 and a light source device 24. The control device 22 and the light source device 24 are installed on a wagon 34. The wagon 34 has a plurality of stands arranged in the vertical direction, and the control devices 22 and the light source devices 24 are installed from the lower stand to the upper stand. In addition, a display device 13 is installed on the top stand of the wagon 34.

[0040] The display device 13 displays various information including images. Examples of the display device 13 include a liquid crystal display and an EL display. Alternatively, a tablet terminal with a display may be used instead of the display device 13 or together with the display device 13.

[0041] The display device 13 displays multiple screens side by side. In the example shown in FIG. 1 , screens 36 and 37 are shown. An endoscopic image 40 obtained by the endoscope 12 is displayed on the screen 36. The endoscopic image 40 shows an observation target 21. The endoscopic image 40 is an image generated by the endoscope 12 capturing an image of the observation target 21 inside the body of the subject 20. The observation target 21 can be an upper digestive organ. For ease of explanation, the following description will use the stomach as an example of the upper digestive organ. The stomach is an example of a "hollow organ" according to the technology of the present disclosure. Note that the stomach is merely an example, and any region that can be imaged by the endoscope 12 can be used. Examples of regions that can be imaged by the endoscope 12 include hollow organs such as the large intestine, small intestine, duodenum, esophagus, and bronchi. The endoscopic image 40 is an example of a "medical image" according to the technology of the present disclosure.

[0042] A moving image including a plurality of frames of the endoscopic image 40 is displayed on the screen 36. That is, the plurality of frames of the endoscopic image 40 are displayed on the screen 36 at a predetermined frame rate (for example, several tens of frames per second).

[0043] A medical support image 41 is displayed on the screen 37. The medical support image 41 is an image that the doctor 14 refers to during the endoscopic examination. The doctor 14 refers to the medical support image 41 to check whether any of the multiple regions that are scheduled to be observed during the endoscopic examination have been missed.

[0044] 2, the endoscope 12 includes an operation section 42 and an insertion section 44. The insertion section 44 is partially curved by operating the operation section 42. The insertion section 44 is inserted while curving in accordance with the shape of the observation target 21 (for example, the shape of the stomach) in accordance with the operation of the operation section 42 by the physician 14.

[0045] A camera 48, an illumination device 50, and a treatment opening 52 are provided at the distal end 46 of the insertion section 44. The camera 48 is a device that captures an image of the inside of the body of the subject 20 to obtain an endoscopic image 40 as a medical image. The camera 48 is an example of an "image acquisition device" according to the technology of the present disclosure. An example of the camera 48 is a CMOS camera. However, this is merely an example, and other types of cameras such as a CCD camera may also be used.

[0046] The illumination device 50 has illumination windows 50A and 50B. The illumination device 50 emits light through the illumination windows 50A and 50B. Examples of the type of light emitted from the illumination device 50 include visible light (e.g., white light) and invisible light (e.g., near-infrared light). The illumination device 50 also emits special light through the illumination windows 50A and 50B. Examples of the special light include light for BLI and / or light for LCI. The camera 48 captures images of the inside of the subject 20 by an optical method while light is being emitted from the illumination device 50 inside the subject 20.

[0047] The treatment opening 52 is used as a treatment tool ejection port for ejecting a treatment tool 54 from the distal end portion 46, a suction port for sucking blood and internal waste, and a delivery port for delivering fluid.

[0048] A treatment tool 54 protrudes from the treatment opening 52 in accordance with the operation of the physician 14. The treatment tool 54 is inserted into the insertion section 44 from a treatment tool insertion port 58. The treatment tool 54 passes through the insertion section 44 via the treatment tool insertion port 58 and protrudes from the treatment opening 52 into the body of the subject 20. In the example shown in Fig. 2, forceps protrude from the treatment opening 52 as the treatment tool 54. Forceps are merely one example of the treatment tool 54, and other examples of the treatment tool 54 include a wire, a scalpel, and an ultrasonic probe.

[0049] A suction pump (not shown) is connected to the endoscope body 18, and the treatment opening 52 uses the suction force of the suction pump to suck blood, internal waste, etc. from the observation subject 21. The suction force of the suction pump is controlled in accordance with instructions given by the doctor 14 to the endoscope 12 via the operation unit 42 or the like.

[0050] A supply pump (not shown) is connected to the endoscope body 18, and the supply pump supplies fluid (e.g., gas and / or liquid) into the endoscope body 18. The treatment opening 52 delivers the fluid supplied from the supply pump to the endoscope body 18. From the treatment opening 52, gas (e.g., air) and liquid (e.g., saline) are selectively delivered as fluid into the body in accordance with instructions given by the physician 14 to the endoscope 12 via the operation unit 42, etc. The amount of fluid delivered is controlled in accordance with instructions given by the physician 14 to the endoscope 12 via the operation unit 42, etc.

[0051] Here, an example is given in which the treatment opening 52 is used as a treatment tool ejection opening, a suction opening, and a delivery opening, but this is merely one example, and the treatment tool ejection opening, the suction opening, and the delivery opening may be provided separately at the tip portion 46, or the tip portion 46 may be provided with a treatment tool ejection opening and an opening that serves as both a suction opening and a delivery opening.

[0052] The endoscope body 18 is connected to the control device 22 and the light source device 24 via a universal cord 60. The control device 22 is connected to the display device 13 and the reception device 62. The reception device 62 receives instructions from a user and outputs the received instructions as an electrical signal. In the example shown in Fig. 2, a keyboard is given as an example of the reception device 62. However, this is merely an example, and the reception device 62 may also be a mouse, a touch panel, a foot switch, and / or a microphone, etc.

[0053] The control device 22 controls the entire endoscope 12. For example, the control device 22 controls the light source device 24, exchanges various signals with the camera 48, and displays various information on the display device 13. The light source device 24 emits light under the control of the control device 22 and supplies the light to the illumination device 50. The illumination device 50 has a built-in light guide, and the light supplied from the light source device 24 passes through the light guide and is irradiated from illumination windows 50A and 50B. The control device 22 causes the camera 48 to capture an image, acquires an endoscopic image 40 (see FIG. 1 ) from the camera 48, and outputs the image to a predetermined output destination (for example, the display device 13).

[0054] As an example, as shown in Fig. 3, the control device 22 includes a computer 64. The computer 64 is an example of a "medical support device" and a "computer" according to the techniques of the present disclosure. The computer 64 includes a processor 70, a RAM 72, and a NVM 74, which are electrically connected to each other. The processor 70 is an example of a "processor" according to the techniques of the present disclosure.

[0055] The control device 22 includes a computer 64, a bus 66, and an external I / F 68. The computer 64 includes a processor 70, a RAM 72, and an NVM 74. The processor 70, the RAM 72, the NVM 74, and the external I / F 68 are connected to the bus 66.

[0056] For example, the processor 70 has a CPU and a GPU, and controls the entire control device 22. The GPU operates under the control of the CPU, and is responsible for executing various graphic processing and performing calculations using neural networks. The processor 70 may be one or more CPUs that have integrated GPU functionality, or one or more CPUs that do not have integrated GPU functionality.

[0057] The RAM 72 is a memory that temporarily stores information and is used as a work memory by the processor 70. The NVM 74 is a nonvolatile storage device that stores various programs, various parameters, and the like. An example of the NVM 74 is a flash memory (for example, an EEPROM and / or an SSD). Note that the flash memory is merely one example, and the NVM 74 may be another nonvolatile storage device such as an HDD, or may be a combination of two or more types of nonvolatile storage devices.

[0058] The external I / F 68 controls the exchange of various information between devices that exist outside the control device 22 (hereinafter also referred to as "external devices") and the processor 70. An example of the external I / F 68 is a USB interface.

[0059] The external I / F 68 is connected to the camera 48 as one of the external devices, and the external I / F 68 controls the exchange of various information between the camera 48 and the processor 70. The processor 70 controls the camera 48 via the external I / F 68. The processor 70 also acquires, via the external I / F 68, an endoscopic image 40 (see FIG. 1 ) obtained by the camera 48 capturing an image of the inside of the body of the subject 20.

[0060] The light source device 24 is connected to the external I / F 68 as one of the external devices, and the external I / F 68 controls the exchange of various information between the light source device 24 and the processor 70. The light source device 24 supplies light to the illumination device 50 under the control of the processor 70. The illumination device 50 irradiates the light supplied from the light source device 24.

[0061] The external I / F 68 is connected to the display device 13 as one of the external devices, and the processor 70 controls the display device 13 via the external I / F 68 to cause the display device 13 to display various information.

[0062] The external I / F 68 is connected to a reception device 62 as one of the external devices, and the processor 70 acquires instructions accepted by the reception device 62 via the external I / F 68 and executes processing according to the acquired instructions.

[0063] In general, in endoscopic examinations, lesions are detected using image recognition processing (e.g., AI-based image recognition processing), and in some cases, procedures such as excision of the lesion are performed. Furthermore, in endoscopic examinations, the physician 14 simultaneously operates the insertion portion 44 of the endoscope 12 and differentiates between lesions, placing a heavy burden on the physician 14 and raising concerns about overlooking lesions. To prevent overlooking lesions, it is important that the image recognition processing recognizes all of the multiple regions that are planned in advance within the observation target 21. However, it is extremely difficult for the physician 14 to proceed with the work while checking whether the image recognition processing has recognized all of the multiple regions that are planned in advance.

[0064] In view of the above circumstances, in order to prevent missed recognition during image recognition processing for multiple regions that are planned in advance, in this embodiment, medical support processing is performed by the processor 70 of the control device 22 (see FIGS. 4 and 10). Note that in this embodiment, missed recognition is synonymous with missed observation, as described above.

[0065] The medical support processing includes processing for recognizing a plurality of parts within the observation target 21 based on a plurality of endoscopic images 40 showing the observation target 21, and outputting unrecognized information that can identify the presence of an unrecognized part if an unrecognized part (i.e., a part not recognized by the processor 70) exists within the plurality of parts within the observation target 21. The medical support processing will be described in more detail below.

[0066] 4 , a medical support processing program 76 is stored in the NVM 74. The medical support processing program 76 is an example of a “program” according to the technology of the present disclosure. The processor 70 reads the medical support processing program 76 from the NVM 74 and executes the read medical support processing program 76 on the RAM 72. The medical support processing is realized by the processor 70 operating as an image acquisition unit 70A, a recognition unit 70B, and a control unit 70C in accordance with the medical support processing program 76 executed on the RAM 72.

[0067] A trained model 78 is stored in the NVM 74. In this embodiment, the recognition unit 70B performs AI-based image recognition processing as image recognition processing for object detection. The AI-based image recognition processing by the recognition unit 70B refers to image recognition processing using the trained model 78. The trained model 78 is a mathematical model for object detection, and is obtained by optimizing a neural network by performing machine learning on the neural network in advance. Hereinafter, the image recognition processing using the trained model 78 will be described as processing actively performed by the trained model 78. In other words, for convenience of explanation, the trained model 78 will be described below as a function that processes input information and outputs the processing results.

[0068] The NVM 74 stores a recognition portion confirmation table 80 and an importance table 82. Both the recognition portion confirmation table 80 and the importance table 82 are used by the control unit 70C.

[0069] As an example, as shown in FIG. 5, the image acquisition unit 70A acquires the endoscopic image 40 generated by the camera 48 capturing images at an imaging frame rate (e.g., several tens of frames per second) from the camera 48, one frame at a time.

[0070] The image acquisition unit 70A holds a time-series image group 89. The time-series image group 89 is a plurality of endoscopic images 40 in time series that capture the object of observation 21. The time-series image group 89 includes, for example, a certain number of frames (e.g., a predetermined number of frames within a range of several tens to several hundreds of frames) of endoscopic images 40. The image acquisition unit 70A updates the time-series image group 89 in a FIFO manner every time it acquires an endoscopic image 40 from the camera 48.

[0071] Here, an example is given in which the time-series image group 89 is stored and updated by the image acquisition unit 70A, but this is merely one example. For example, the time-series image group 89 may be stored and updated in a memory connected to the processor 70, such as the RAM 72.

[0072] The recognition unit 70B recognizes a region of the observation target 21 by performing image recognition processing using the trained model 78 on the time-series image group 89 (i.e., the plurality of time-series endoscopic images 40 held by the image acquisition unit 70A). In other words, region recognition can also be considered region detection. In this embodiment, region recognition refers to the process of identifying the name of the region and storing the endoscopic image 40 showing the recognized region and the name of the region shown in the endoscopic image 40 in a state of association in memory (for example, the NVM 74 and / or an external storage device, etc.).

[0073] The trained model 78 is obtained by optimizing the neural network through machine learning using first training data. The first training data may be training data in which, as example data, multiple images (e.g., multiple images corresponding to multiple endoscopic images 40 in time series) obtained by capturing images of potential endoscopically examined regions (e.g., regions within the observation target 21) are used as example data, and region information 90 regarding the potential endoscopically examined regions is used as correct answer data. There are multiple regions, such as the cardia, the fundus, the anterior wall of the greater curvature of the upper gastric body, the posterior wall of the greater curvature of the upper gastric body, the anterior wall of the greater curvature of the middle gastric body, the posterior wall of the greater curvature of the middle gastric body, the anterior wall of the greater curvature of the lower gastric body, and the posterior wall of the greater curvature of the lower gastric body. Machine learning is performed on the neural network using first training data created for each region. The region information 90 includes information indicating the name of the region and coordinates by which the position of the region within the observation target 21 can be identified.

[0074] Note that, although an example in which only one trained model 78 is used by the recognition unit 70B is given here, this is merely one example. For example, a trained model 78 selected from a plurality of trained models 78 may be used by the recognition unit 70B. In this case, each trained model 78 is created by performing machine learning specialized for each type of endoscopic examination, and the trained model 78 corresponding to the type of endoscopic examination currently being performed is selected and used by the recognition unit 70B.

[0075] In this embodiment, as an example of the trained model 78 used by the recognition unit 70B, a trained model created by performing machine learning specialized for endoscopic examination of the stomach is applied.

[0076] Note that, although an example in which a trained model is created by performing machine learning on a neural network that is specialized for endoscopic examination of the stomach has been described above, this is merely one example. When an endoscopic examination of a hollow organ other than the stomach is performed, a trained model created by performing machine learning on a neural network that is specialized for the type of hollow organ that will be examined endoscopically can be used. Examples of hollow organs other than the stomach include the large intestine, small intestine, esophagus, duodenum, and bronchi. Furthermore, trained models created by performing machine learning on a neural network that assumes endoscopic examination of multiple hollow organs, such as the stomach, large intestine, small intestine, esophagus, duodenum, and bronchi, may be used as trained model 78.

[0077] The recognition unit 70B recognizes multiple regions (hereinafter simply referred to as "multiple regions") contained in the stomach by performing image recognition processing using the trained model 78 on the time-series image group 89 acquired by the image acquisition unit 70A. The multiple regions are classified into major categories and minor categories contained in the major categories. The "major categories" referred to here are an example of a "major category" according to the technology of the present disclosure. Furthermore, the "minor categories" referred to here are an example of a "minor category" according to the technology of the present disclosure.

[0078] The multiple regions are broadly classified into the cardia, fundus, greater curvature of the upper body, greater curvature of the middle body, greater curvature of the lower body, greater curvature of the angle of the stomach, greater curvature of the antrum, bulb, pylorus ring, lesser curvature of the antrum, lesser curvature of the angle of the stomach, lesser curvature of the lower body, lesser curvature of the middle body, and lesser curvature of the upper body.

[0079] The greater curvature of the upper body of the stomach is subdivided into the anterior wall of the greater curvature of the upper body and the posterior wall of the greater curvature of the upper body. The greater curvature of the middle body of the stomach is subdivided into the anterior wall of the greater curvature of the middle body and the posterior wall of the greater curvature of the middle body. The greater curvature of the lower body of the stomach is subdivided into the anterior wall of the greater curvature of the lower body and the posterior wall of the greater curvature of the lower body. The greater curvature of the gastric angle is subdivided into the anterior wall of the greater curvature of the gastric angle and the posterior wall of the greater curvature of the gastric angle. The greater curvature of the antrum is subdivided into the anterior wall of the greater curvature of the antrum and the posterior wall of the greater curvature of the antrum. The lesser curvature of the antrum is subdivided into the anterior wall of the lesser curvature of the antrum and the posterior wall of the lesser curvature of the antrum. The lesser curvature of the gastric angle is subdivided into the anterior wall of the gastric angle on the lesser curvature side and the posterior wall of the gastric angle on the lesser curvature side. The lesser curvature of the lower body of the stomach is subdivided into the anterior wall of the lower body on the lesser curvature side and the posterior wall of the lower body on the lesser curvature side. The lesser curvature of the middle body of the stomach is subdivided into the anterior wall of the middle body on the lesser curvature side and the posterior wall of the middle body on the lesser curvature side. The lesser curvature of the upper body of the stomach is subdivided into the anterior wall of the upper body on the lesser curvature side and the posterior wall of the upper body on the lesser curvature side.

[0080] The recognition unit 70B acquires a time-series image group 89 from the image acquisition unit 70A and inputs the acquired time-series image group 89 to the trained model 78. As a result, the trained model 78 outputs body part information 90 corresponding to the input time-series image group 89. The recognition unit 70B acquires the body part information 90 output from the trained model 78.

[0081] The recognition part confirmation table 80 is a table used to confirm whether or not a part that is scheduled to be recognized by the recognition unit 70B has been recognized. The recognition part confirmation table 80 associates the above-mentioned multiple parts with information indicating whether or not each part has been recognized by the recognition unit 70B. Since the part name is identified from the part information 90, the recognition unit 70B updates the recognition part confirmation table 80 in accordance with the part information 90 acquired from the trained model 78. In other words, the recognition unit 70B updates the information corresponding to each part in the recognition part confirmation table 80 (i.e., information indicating whether or not it has been recognized by the recognition unit 70B).

[0082] The control unit 70C displays the endoscopic image 40 acquired by the image acquisition unit 70A on the screen 36. The control unit 70C generates a detection frame 23 based on the body part information 90 and superimposes the generated detection frame 23 on the endoscopic image 40. The detection frame 23 is a frame that can identify the position of a body part identified from the body part information 90. For example, the detection frame 23 is generated based on a bounding box used in AI-based image recognition processing. The detection frame 23 may be a rectangular frame made of continuous lines, or a frame of a shape other than a rectangular frame. Furthermore, instead of a rectangular frame made of continuous lines, a frame made of discontinuous lines (i.e., intermittent lines) may be used. Furthermore, for example, multiple marks may be displayed to identify portions corresponding to the four corners of the detection frame 23. Furthermore, the body part identified from the body part information 90 may be filled in with a predetermined color (e.g., a translucent color).

[0083] Note that, although an example embodiment in which AI-based processing (e.g., processing by the recognition unit 70B) is performed by the control device 22 has been described above, the technology of the present disclosure is not limited to this. For example, AI-based processing may be performed by a device separate from the control device 22. In this case, for example, the device separate from the control device 22 acquires the endoscopic image 40 and various parameters used for observing the observation target 21 with the endoscope 12, and outputs an image in which the detection frame 23 and / or various maps (e.g., medical support image 41, etc.) are superimposed on the endoscopic image 40 to the display device 13, etc.

[0084] As an example, as shown in FIG. 6 , the recognition part confirmation table 80 is a table in which part names 92 are associated with part flags 94 and major classification flags 96. The part names 92 are the names of the parts. In the recognition part confirmation table 80, the multiple part names 92 are arranged in a planned recognition order 97. The planned recognition order 97 refers to the order in which the parts are planned to be recognized by the recognition unit 70B. The parts planned to be recognized by the recognition unit 70B are an example of a "planned part" according to the technology of the present disclosure, and the planned recognition order 97 is an example of a "second order" according to the technology of the present disclosure.

[0085] The part flag 94 is a flag that indicates whether the part corresponding to the part name 92 has been recognized by the recognition unit 70B. The part flag 94 can be switched on (for example, 1) or off (for example, 0). The part flag 94 is off by default. When the recognition unit 70B recognizes the part corresponding to the part name 92, it turns on the part flag 94 that corresponds to the part name 92 indicating the recognized part.

[0086] The major category flag 96 is a flag that indicates whether a part corresponding to a major category has been recognized by the recognition unit 70B. The major category flag 96 is switched on (e.g., 1) or off (e.g., 0). The major category flag 96 is off by default. When the recognition unit 70B recognizes any part classified into a major category (e.g., any part classified into a minor category of parts classified into a major category), i.e., a part corresponding to the part name 92, it turns on the major category flag 96 that corresponds to the major category into which the recognized part is classified. In other words, when any part flag 94 corresponding to the major category flag 96 is turned on, the major category flag 96 is turned on.

[0087] 7 , the importance table 82 is a table in which importance 98 is associated with part names 92. That is, importance 98 is assigned to multiple parts. The importance 98 is an example of "importance" according to the technology of the present disclosure.

[0088] In the importance table 82, a plurality of part names 92 are arranged in the order of parts that are scheduled to be recognized by the recognition unit 70B. That is, in the importance table 82, the plurality of part names 92 are arranged in a scheduled recognition order 97. The importance 98 is the importance of a part identified from the part name 92. The importance 98 is defined as one of three levels: "high", "medium", and "low". Parts classified into small categories are assigned the importance 98 of "high" or "medium", and parts classified into large categories are assigned the importance 98 of "low".

[0089] In the example shown in FIG. 7 , an importance level of “high” is assigned to the posterior wall of the greater curvature of the upper gastric body, the anterior wall of the greater curvature of the middle gastric body, the anterior wall of the greater curvature of the lower gastric body, the anterior wall of the lesser curvature of the lower gastric body, the posterior wall of the lesser curvature of the lower gastric body, the anterior wall of the lesser curvature of the middle gastric body, the posterior wall of the lesser curvature of the middle gastric body, and the posterior wall of the lesser curvature of the upper gastric body.

[0090] Each part classified into the subcategory other than the posterior wall of the greater curvature of the upper body of the stomach, the anterior wall of the greater curvature of the middle body of the stomach, the anterior wall of the greater curvature of the lower body of the stomach, the anterior wall of the lesser curvature of the lower body of the stomach, the posterior wall of the lesser curvature of the lower body of the stomach, the anterior wall of the lesser curvature of the middle body of the stomach, the posterior wall of the lesser curvature of the middle body of the stomach, and the posterior wall of the lesser curvature of the upper body of the stomach is given an importance level of 98, which is "medium." That is, an importance level of "medium" of 98 is assigned to the anterior wall of the greater curvature of the upper body of the stomach, the posterior wall of the greater curvature of the middle body of the stomach, the posterior wall of the greater curvature of the lower body of the stomach, the anterior wall of the greater curvature of the gastric angle, the posterior wall of the greater curvature of the gastric angle, the anterior wall of the greater curvature of the vestibule, the posterior wall of the greater curvature of the vestibule, the anterior wall of the lesser curvature of the vestibule, the posterior wall of the lesser curvature of the vestibule, the anterior wall of the lesser curvature of the gastric angle, the posterior wall of the lesser curvature of the gastric angle, and the anterior wall of the lesser curvature of the upper body of the stomach.

[0091] The parts classified into major categories, such as the cardia, fundus, greater curvature of the upper body, greater curvature of the middle body, greater curvature of the lower body, greater curvature of the angle of the stomach, greater curvature of the antrum, bulb, pylorus ring, lesser curvature of the antrum, lesser curvature of the angle of the stomach, lesser curvature of the lower body, lesser curvature of the middle body, and lesser curvature of the upper body, have a lower importance level of 98 than the parts classified into minor categories. 7, the cardia, fundus, greater curvature of the upper gastric body, greater curvature of the middle gastric body, greater curvature of the lower gastric body, greater curvature of the gastric angle, greater curvature of the antrum, bulb, pylorus ring, lesser curvature of the antrum, lesser curvature of the gastric angle, lesser curvature of the lower gastric body, lesser curvature of the middle gastric body, and lesser curvature of the upper gastric body are assigned a "low" importance 98. In other words, a higher importance 98 is assigned to the parts classified into the minor categories than to the parts classified into the major categories.

[0092] The importance levels 98 of "high," "medium," and "low" are determined according to instructions given from outside to the endoscope 12. The reception device 62 is an example of a first means for giving the instruction of the importance level 98 to the endoscope 12. A communication device (e.g., a tablet terminal, a personal computer, and / or a server) connected to the endoscope 12 so as to be able to communicate with the endoscope 12 is an example of a second means for giving the instruction of the importance level 98 to the endoscope 12.

[0093] Furthermore, the importance 98 associated with the multiple part names 92 is determined according to past test data performed on the multiple parts (e.g., statistical data based on past test data obtained from multiple subjects 20).

[0094] For example, the importance 98 corresponding to a part of the multiple parts that is determined to be typically prone to oversight is set higher than the importance 98 corresponding to a part of the multiple parts that is determined to be typically unlikely to be oversight. Whether or not a part is typically prone to oversight is determined by a statistical method or the like from past test data performed on the multiple parts. In this embodiment, the importance 98 "high" indicates a high level of likelihood that oversight will typically occur. Furthermore, the importance 98 "medium" indicates a medium level of likelihood that oversight will typically occur. Furthermore, the importance 98 "low" indicates a low level of likelihood that oversight will typically occur.

[0095] 8, the control unit 70C outputs unrecognized information 100 when an unrecognized portion exists in a plurality of portions of the observation target 21 in accordance with the recognition portion confirmation table 80 and the importance table 82. The unrecognized information 100 is information that can identify the presence of an unrecognized portion. The unrecognized information 100 includes importance information 102. The importance information 102 is information that can identify the importance 98 obtained from the importance table 82.

[0096] The output destination of the unrecognized information 100 is the display device 13. However, this is merely an example, and the output destination of the unrecognized information 100 may be a tablet terminal, a personal computer, and / or a server that are communicatively connected to the endoscope 12.

[0097] The unrecognized information 100 is displayed on the screen 37 as a medical support image 41 by the control unit 70C. The medical support image 41 is an example of a "schematic diagram" and a "first schematic diagram" according to the technology of the present disclosure. The importance information 102 included in the unrecognized information 100 is displayed as an importance mark 104 within the medical support image 41 by the control unit 70C.

[0098] The display mode of the importance mark 104 differs depending on the importance information 102. The importance mark 104 is categorized into a first importance mark 104A, a second importance mark 104B, and a third importance mark 104C. The first importance mark 104A is a mark that represents a "high" level of importance 98. The second importance mark 104B is a mark that represents a "medium" level of importance 98. The third importance mark 104C is a mark that represents a "low" level of importance 98. In other words, the first importance mark 104A, the second importance mark 104B, and the third importance mark 104C are marks that represent "high," "medium," and "low" levels of importance in a distinguishable manner. The second importance mark 104B is displayed in a more emphasized state than the third importance mark 104C, and the first importance mark 104A is displayed in a more emphasized state than the second importance mark 104B.

[0099] In the example shown in FIG. 8 , the first importance mark 104A includes multiple exclamation marks (two, for example), while the second importance mark 104B and the third importance mark 104C each include a single exclamation mark. The size of the exclamation mark included in the third importance mark 104C is smaller than the size of the exclamation mark included in the first importance mark 104A and the second importance mark 104B. The second importance mark 104B is colored to be more noticeable than the third importance mark 104C, and the first importance mark 104A is colored to be more noticeable than the second importance mark 104B. The brightness of the second importance mark 104B is higher than the brightness of the third importance mark 104C, and the brightness of the first importance mark 104A is higher than the brightness of the second importance mark 104B. In this way, the relationship of conspicuousness is "first importance mark 104A>second importance mark 104B>third importance mark 104C".

[0100] As an example, as shown in Fig. 9 , the medical support image 41 includes a path 106. The path 106 is a path that schematically represents the order in which the stomach is observed using the endoscope 12 (here, as an example, the planned recognition order 97 (see Figs. 6 and 7 )), and is a schematic diagram in which the observation object 21 is divided into a plurality of regions corresponding to a plurality of parts. In the example shown in Fig. 9 , as examples of "plurality of regions," the cardia, fundus, upper body, middle body, lower body, angle, antrum, pylorus ring, and bulb are displayed in text within the medical support image 41, and the path 106 is divided into the cardia, fundus, upper body, middle body, lower body, angle, antrum, pylorus ring, and bulb.

[0101] The path 106 branches into a greater curvature path 106A and a lesser curvature path 106B midway from the most upstream side to the downstream side of the stomach, and then merges again. On the path 106, large circular marks 108A are assigned to areas classified into major categories, and small circular marks 108B are assigned to areas classified into minor categories. For ease of explanation, the circular marks 108A and 108B will be referred to as "circular marks 108" when there is no need to distinguish between them.

[0102] In the path 106, from the most upstream side of the stomach to just before the branching point between the greater curvature path 106A and the lesser curvature path 106B, circular marks 108A corresponding to the cardia and circular marks 108A corresponding to the dome are arranged from the most upstream side of the stomach to the downstream side of the stomach.

[0103] The greater curvature-side path 106A is provided with a circular mark 108A corresponding to the greater curvature, a circular mark 108B corresponding to the anterior wall, and a circular mark 108B corresponding to the posterior wall, arranged in units of regions classified into major categories. The circular mark 108A corresponding to the greater curvature is located in the center of the greater curvature-side path 106A, and the circular mark 108B corresponding to the anterior wall and the circular mark 108B corresponding to the posterior wall are located on the left and right of the circular mark 108A corresponding to the greater curvature.

[0104] The lesser curvature-side path 106B is provided with a circular mark 108A corresponding to the lesser curvature, a circular mark 108B corresponding to the anterior wall, and a circular mark 108B corresponding to the posterior wall, arranged in units of major sections. The circular mark 108A corresponding to the lesser curvature is located in the center of the lesser curvature-side path 106B, and the circular mark 108B corresponding to the anterior wall and the circular mark 108B corresponding to the posterior wall are located on the left and right of the circular mark 108A corresponding to the lesser curvature.

[0105] In the path 106, from the confluence of the greater curvature path 106A and the lesser curvature path 106B to the most downstream part of the stomach, circular marks 108A corresponding to the pylorus ring and circular marks 108A corresponding to the bulb are arranged.

[0106] The circular mark 108 is blank by default. When the recognition unit 70B recognizes a region corresponding to the circular mark 108, the region corresponding to the region recognized by the recognition unit 70B is filled in with a specific color (e.g., a predetermined color selected from the three primary colors of light and the three primary colors of color). In contrast, when the recognition unit 70B does not recognize a region corresponding to the circular mark 108, the region corresponding to the region not recognized by the recognition unit 70B is not filled in. However, an importance mark 104 corresponding to the region not recognized by the recognition unit 70B is displayed in the circular mark 108 corresponding to the region not recognized by the recognition unit 70B. In this way, the display device 13 displays the circular marks 108 corresponding to the region recognized by the recognition unit 70B and the circular marks 108 corresponding to the region not recognized by the recognition unit 70B in the medical support image 41 in a manner that allows them to be distinguished from each other.

[0107] The image obtained by filling the circular mark 108 with a specific color is an example of a "second image capable of identifying a part other than the unrecognized part among the multiple parts" according to the technology of the present disclosure. The image obtained by displaying the importance mark 104 according to the part importance 98 within the circular mark 108 is an example of a "first image capable of identifying the unrecognized part" according to the technology of the present disclosure.

[0108] When the major classification flag 96 in the recognition part confirmation table 80 is turned on, the control unit 70C updates the content of the medical support image 41. The update of the content of the medical support image 41 is realized by the output of unrecognized information 100 by the control unit 70C.

[0109] When a major classification flag 96 in the recognition region confirmation table 80 is turned on, the control unit 70C fills in a specific color the circular mark 108A of the region corresponding to the turned-on major classification flag 96. Furthermore, when a region flag 94 is turned on, the control unit 70C fills in a specific color the circular mark 108B of the region corresponding to the turned-on region flag 94.

[0110] In addition, when a major classification includes multiple minor classifications, when a part flag 94 corresponding to a part classified into one minor classification is turned on, a major classification flag 96 corresponding to a part classified into the minor classification whose part flag 94 is turned on is also turned on.

[0111] On the other hand, when a site is not recognized by the recognition unit 70B, the control unit 70C displays an importance mark 104 in the circular mark 108 corresponding to the site not recognized by the recognition unit 70B, provided that the recognition unit 70B recognizes a subsequent site that is scheduled to be recognized by the recognition unit 70B after the site not recognized by the recognition unit 70B. That is, when it is determined that the order in which the sites were recognized by the recognition unit 70B deviates from the expected recognition order 97 ( FIGS. 6 and 7 ), the control unit 70C displays the importance mark 104 in the circular mark 108 corresponding to the site not recognized by the recognition unit 70B. The reason for this is to ensure that a notification of a site not recognized by the recognition unit 70B is made when it is determined that the site was not recognized by the recognition unit 70B (e.g., when it becomes highly likely that a site was not observed while the doctor 14 was operating the endoscope 12). Note that the order in which the site was recognized by the recognition unit 70B is an example of the “first order” according to the technology of the present disclosure.

[0112] Here, an example of a subsequent part that is scheduled to be recognized after a part not recognized by the recognition unit 70B is a part that is classified into a major category that is scheduled to be recognized one category after the major category into which the part not recognized by the recognition unit 70B is classified. Here, the major category into which the part not recognized by the recognition unit 70B is classified is an example of a "first major category" according to the technology of the present disclosure. Also, the major category that is scheduled to be recognized one category after the major category into which the part not recognized by the recognition unit 70B is classified is an example of a "second major category" according to the technology of the present disclosure.

[0113] 9 , if the posterior wall of the greater curvature of the upper stomach body is not recognized by the recognition unit 70B, a second importance mark 104B is superimposed on the circular mark 108B corresponding to the posterior wall of the greater curvature of the upper stomach body, provided that the recognition unit 70B recognizes a region classified into a major category that is scheduled to be recognized one category after the major category to which the posterior wall of the greater curvature of the upper stomach body is classified. Here, the major category to which the posterior wall of the greater curvature of the upper stomach body is classified refers to the greater curvature of the upper stomach body. Furthermore, the major category that is scheduled to be recognized one category after the major category to which the posterior wall of the greater curvature of the upper stomach body is classified refers to the greater curvature of the middle stomach body.

[0114] 9 , if the anterior wall of the greater curvature side of the mid-body of the stomach is not recognized by the recognition unit 70B, a second importance mark 104B is superimposed on the circular mark 108B corresponding to the anterior wall of the greater curvature side of the mid-body of the stomach, provided that the recognition unit 70B recognizes a region classified into a major category that is scheduled to be recognized one category after the major category into which the anterior wall of the greater curvature side of the mid-body of the stomach is classified. Here, the major category into which the anterior wall of the greater curvature side of the mid-body of the stomach is classified refers to the greater curvature of the mid-body of the stomach. Furthermore, the major category that is scheduled to be recognized one category after the major category into which the anterior wall of the greater curvature side of the mid-body of the stomach is classified refers to the greater curvature of the lower part of the stomach.

[0115] 9 , if the anterior wall of the greater curvature of the lower stomach body is not recognized by the recognition unit 70B, a first importance mark 104A is superimposed on the circular mark 108B corresponding to the anterior wall of the greater curvature of the lower stomach body, provided that the recognition unit 70B recognizes a region classified into a major category that is scheduled to be recognized one category later than the major category to which the anterior wall of the greater curvature of the lower stomach body is classified. Here, the major category to which the anterior wall of the greater curvature of the lower stomach body is classified refers to the greater curvature of the lower stomach body. Also, the major category that is scheduled to be recognized one category later than the major category to which the anterior wall of the greater curvature of the lower stomach body is classified refers to the greater curvature of the angle of the stomach.

[0116] In this embodiment, to facilitate identification of a portion not recognized by the recognition unit 70B, the image obtained by superimposing the importance mark 104 on the circular mark 108 is displayed in a more emphasized state than the image obtained by filling the circular mark 108 with a specific color. In the example shown in FIG. 9 , the contour of the image obtained by superimposing the importance mark 104 on the circular mark 108 is displayed in a more emphasized state than the contour of the image obtained by filling the circular mark 108 with a specific color. The contour emphasis is achieved, for example, by adjusting the brightness of the contour. Furthermore, the image obtained by filling the circular mark 108 with a specific color does not include an exclamation mark, whereas the image obtained by superimposing the importance mark 104 on the circular mark 108 includes an exclamation mark. Therefore, the presence or absence of an exclamation mark visually identifies the portion not recognized by the recognition unit 70B and the portion recognized by the recognition unit 70B.

[0117] Next, the operation of the portion of the endoscope system 10 related to the technology of the present disclosure will be described with reference to FIG.

[0118] Fig. 10 shows an example of the flow of medical support processing performed by the processor 70. The flow of medical support processing shown in Fig. 10 is an example of a "medical support method" according to the technology of the present disclosure.

[0119] 10 , first, in step ST10, the image acquisition unit 70A determines whether one frame of image data has been captured by the camera 48. If one frame of image data has not been captured by the camera 48 in step ST10, the determination is negative, and the determination in step ST10 is made again. If one frame of image data has been captured by the camera 48 in step ST10, the determination is positive, and the medical support process proceeds to step ST12.

[0120] In step ST12, the image acquisition unit 70A acquires one frame of the endoscopic image 40 from the camera 48. After the processing of step ST12 is executed, the medical support processing proceeds to step ST14.

[0121] In step ST14, the image acquisition unit 70A determines whether or not it holds a certain number of frames of endoscopic images 40. If it does not hold the certain number of frames of endoscopic images 40 in step ST14, the determination is negative, and the medical support processing proceeds to step ST10. If it does hold the certain number of frames of endoscopic images 40 in step ST14, the determination is positive, and the medical support processing proceeds to step ST16.

[0122] In step ST16, the image acquisition unit 70A adds the endoscopic images 40 acquired in step ST12 to the time-series image group 89 in a FIFO manner, thereby updating the time-series image group 89. After the processing of step ST16 is executed, the medical support processing proceeds to step ST18.

[0123] In step ST18, the recognition unit 70B starts executing the AI-based image recognition process (i.e., the image recognition process using the trained model 78) on the time-series image group 89 updated in step ST16. After the process of step ST18 is executed, the medical support process proceeds to step ST20.

[0124] In step ST20, the recognition unit 70B determines whether or not it has recognized any of the multiple parts in the observation target 21. If the recognition unit 70B has not recognized any of the multiple parts in the observation target 21 in step ST20, the determination is negative, and the medical support processing proceeds to step ST30. If the recognition unit 70B has recognized any of the multiple parts in the observation target 21 in step ST20, the determination is positive, and the medical support processing proceeds to step ST22.

[0125] In step ST22, the recognition unit 70B updates the recognized body part confirmation table 80. That is, the recognition unit 70B updates the recognized body part confirmation table 80 by turning on the body part flag 94 and the major classification flag 96 corresponding to the recognized body part. After the processing of step ST22 is executed, the medical support processing proceeds to step ST24.

[0126] In step ST24, the control unit 70C determines whether or not there is a recognition omission among the parts that are previously scheduled to be recognized by the recognition unit 70B. The determination of whether or not there is a recognition omission is realized, for example, by determining whether or not the order of parts recognized by the recognition unit 70B deviates from the scheduled recognition order 97. In step ST24, if there is a recognition omission among the parts that are previously scheduled to be recognized by the recognition unit 70B, the determination is affirmative, and the medical support processing proceeds to step ST26. In step ST24, if there is no recognition omission among the parts that are previously scheduled to be recognized by the recognition unit 70B, the determination is negative, and the medical support processing proceeds to step ST30.

[0127] In step ST24, if the determination is negative when the medical support image 41 is not displayed on the screen 37, the control unit 70C displays the medical support image 41 on the screen 37 and fills in the circular marks 108 corresponding to the areas recognized by the recognition unit 70B with a specific color. Furthermore, in step ST24, if the determination is negative when the medical support image 41 is displayed on the screen 37, the control unit 70C updates the content of the medical support image 41. That is, the control unit 70C fills in the circular marks 108 corresponding to the areas recognized by the recognition unit 70B with a specific color. This allows the doctor 14 to visually determine which areas have been recognized by the recognition unit 70B from the medical support image 41 displayed on the screen 37.

[0128] In step ST26, the control unit 70C determines whether a subsequent part of the part not recognized by the recognition unit 70B has been recognized by the recognition unit 70B. The subsequent part of the part not recognized by the recognition unit 70B refers to, for example, a part classified into a major category that is scheduled to be recognized by the recognition unit 70B one category after the major category into which the part not recognized by the recognition unit 70B is classified. In step ST26, if the subsequent part of the part not recognized by the recognition unit 70B has not been recognized by the recognition unit 70B, the determination is negative, and the medical support processing proceeds to step ST30. In step ST26, if the subsequent part of the part not recognized by the recognition unit 70B has been recognized by the recognition unit 70B, the determination is positive, and the medical support processing proceeds to step ST28.

[0129] In step ST28, the control unit 70C refers to the importance table 82 and displays the unrecognized image in the medical support image 41 in a display mode corresponding to the importance 98 of the unrecognized region. That is, the control unit 70C superimposes an importance mark 104 corresponding to the importance 98 of the unrecognized region on the circular mark 108. A first importance mark 104A, a second importance mark 104B, and a second importance mark 104C are selectively superimposed on the circular mark 108 according to the importance 98 corresponding to the unrecognized region. This allows the doctor 14 to visually grasp which regions have not been recognized by the recognition unit 70B and visually distinguish the importance 98 assigned to each region. After the processing of step ST28 is executed, the medical support processing proceeds to step ST30.

[0130] In step ST30, the recognition unit 70B ends the execution of the AI-based image recognition process on the time-series image group 89. After the process of step ST30 is executed, the medical support process proceeds to step ST32.

[0131] In step ST32, the control unit 70C determines whether a condition for terminating the medical support process is satisfied. One example of the condition for terminating the medical support process is that an instruction to terminate the medical support process has been given to the endoscope system 10 (for example, that an instruction to terminate the medical support process has been accepted by the acceptance device 62).

[0132] In step ST32, if the condition for terminating the medical support process is not satisfied, the determination is negative, and the medical support process proceeds to step ST10 shown in Fig. 10. In step ST32, if the condition for terminating the medical support process is satisfied, the determination is positive, and the medical support process ends.

[0133] As described above, in the endoscopic system 10, the processes of steps ST10 to ST32 of the medical support process are repeatedly executed, thereby allowing the recognition unit 70B to recognize multiple regions. If an unrecognized region (i.e., a region not recognized by the recognition unit 70B) exists within the observation target 21 (here, the stomach, as an example), the control unit 70C outputs unrecognized information 100 to the display device 13. The unrecognized information 100 is displayed on the screen 37 as a medical support image 41. The unrecognized region is displayed as an importance mark 104 in the medical support image 41. This allows the doctor 14 to visually identify the unrecognized region. Therefore, the doctor 14 can retry imaging the unrecognized region using the camera 48 while referring to the medical support image 41. If the recognition unit 70B again performs AI-based image recognition processing on the endoscopic image 40 obtained by retrying imaging the unrecognized region, it becomes possible to recognize the region that was previously unrecognized. In this way, the endoscope system 10 can contribute to reducing oversight of recognition of parts within the observation object 21 .

[0134] Furthermore, in the endoscope system 10, if a site is not recognized by the recognition unit 70B, the control unit 70C outputs unrecognized information 100 to the display device 13 on the condition that a subsequent site that is scheduled to be recognized by the recognition unit 70B after the unrecognized site has been recognized. For example, if a site is not recognized by the recognition unit 70B, the control unit 70C outputs unrecognized information 100 to the display device 13 on the condition that a site classified into a major category that is scheduled to be recognized by the recognition unit 70B after the unrecognized site has been recognized. Therefore, the endoscope system 10 can make the doctor 14 aware that a site within the observation target 21 has been overlooked in a situation where there is a high possibility that a site within the observation target 21 has been overlooked.

[0135] Furthermore, in the endoscope system 10, unrecognized information 100 is output to the display device 13 based on the order in which the multiple parts are recognized by the recognition unit 70B and the planned recognition order 97. That is, if the order in which the multiple parts are recognized by the recognition unit 70B deviates from the planned recognition order 97, the unrecognized information 100 is output to the display device 13. Therefore, it is possible to easily identify whether or not a part in the observation target 21 is an unrecognized part.

[0136] Furthermore, in the endoscope system 10, the unrecognized information 100 output from the control unit 70C includes importance information 102, and the importance information 102 is displayed as an importance mark 104 in the medical support image 41. Therefore, the doctor 14 can visually grasp the importance 98 of the unrecognized part.

[0137] Furthermore, in the endoscope system 10, the importance 98 assigned to each region is determined in accordance with an externally provided instruction, so that it is possible to prevent overlooking of a region among a plurality of regions that has a high importance 98 determined in accordance with an externally provided instruction.

[0138] Furthermore, in the endoscope system 10, the importance 98 assigned to a region is determined based on past examination data obtained from a plurality of regions, thereby preventing regions with high importance 98 determined based on past examination data from being overlooked.

[0139] Furthermore, in the endoscope system 10, the importance 98 corresponding to a part of the plurality of parts that is determined as a part that is typically prone to oversight is set higher than the importance 98 corresponding to a part of the plurality of parts that is determined as a part that is typically unlikely to oversight. Therefore, oversight of a part of the plurality of parts that is determined as a part that is typically prone to oversight can be suppressed.

[0140] Furthermore, in the endoscope system 10, a higher importance 98 is assigned to regions classified into minor categories than to regions classified into major categories. Therefore, compared to a case where the same level of importance 98 is assigned to regions classified into major categories and regions classified into minor categories, it is possible to prevent regions classified into minor categories from being overlooked.

[0141] Furthermore, in the endoscope system 10, a medical support image 41 is displayed on the screen 37. An image obtained by filling the circular mark 108 with a specific color and an image obtained by superimposing the importance mark 104 on the circular mark 108 are displayed within the medical support image 41. The image obtained by filling the circular mark 108 with a specific color is an image corresponding to a region recognized by the recognition unit 70B, and the image obtained by superimposing the importance mark 104 on the circular mark 108 is an image corresponding to a region not recognized by the recognition unit 70B. Therefore, the doctor 14 can visually distinguish between unrecognized regions and regions other than the unrecognized regions (i.e., regions recognized by the recognition unit 70B) from the medical support image 41 displayed on the screen 37.

[0142] Furthermore, in the endoscope system 10, a medical support image 41 is displayed on the screen 37. The medical support image 41 is a schematic diagram and includes a path 106. The path 106 is a path that represents the planned recognition order 97, and is also a schematic diagram in which the observation object 21 is divided into multiple regions corresponding to multiple parts. Therefore, it is possible to make it easier for the doctor 14 to grasp the positional relationship between an unrecognized part and other parts within the observation object 21.

[0143] Furthermore, in the endoscope system 10, a medical support image 41 is displayed on the screen 37. An image obtained by filling the circular mark 108 with a specific color and an image obtained by superimposing the importance mark 104 on the circular mark 108 are displayed within the medical support image 41. The image obtained by superimposing the importance mark 104 on the circular mark 108 is displayed in a more emphasized state than the image obtained by filling the circular mark 108 with a specific color. This makes it easier for the doctor 14 to recognize an overlooked part.

[0144] Furthermore, in the endoscope system 10, the display mode of the importance mark 104 superimposed on the circular mark 108 varies depending on the importance 98 assigned to a plurality of regions. Therefore, the doctor 14 can vary the level of attention to the unrecognized region depending on the importance 98 assigned to the unrecognized region.

[0145] In the above embodiment, an example in which the screens 36 and 37 are displayed on the display device 13 in a comparable state has been described, but this is merely one example, and the screens 36 and 37 may be selectively displayed. The size ratio between the screens 36 and 37 may be changed in accordance with instructions received by the reception device 62 and / or the current state of the endoscope 12 (for example, the operating status of the endoscope 12), etc.

[0146] In the above embodiment, an example in which the recognition unit 70B performs AI-based image recognition processing has been described, but the technology of the present disclosure is not limited to this. For example, the recognition unit 70B may perform non-AI-based image recognition processing (e.g., template matching) to recognize a region. Furthermore, the recognition unit 70B may use both AI-based image recognition processing and non-AI-based image recognition processing to recognize a region.

[0147] In the above embodiment, an example was given in which the recognition unit 70B recognizes a part by performing image recognition processing on a time-series image group 89, but this is merely one example, and a part may also be recognized by performing image recognition processing on a single frame of endoscopic image 40.

[0148] In the above embodiment, the recognition unit 70B performs the image recognition process on the condition that the time-series image group 89 has been updated, but the technology of the present disclosure is not limited to this. For example, the recognition unit 70B may perform the image recognition process on the condition that a specific instruction (e.g., an instruction to start the image recognition process to the recognition unit 70B) is given to the endoscope 12 from the doctor 14 via the reception device 62 or a communication device communicably connected to the endoscope 12.

[0149] In the above embodiment, the display modes of the first importance mark 104A, the second importance mark 104B, and the third importance mark 104C differ depending on the importance level 98, but the technology of the present disclosure is not limited to this. For example, the display modes of the first importance mark 104A, the second importance mark 104B, and the third importance mark 104C may differ depending on the type of unrecognized region. For example, the display mode of the importance mark 104 superimposed on the circular mark 108B corresponding to the posterior wall of the greater curvature side of the upper gastric body may be distinguishably different from the display mode of the importance mark 104 superimposed on the circular mark 108B corresponding to the anterior wall of the greater curvature side of the middle gastric body. This allows the physician 14 to visually understand the type of unrecognized region.

[0150] Furthermore, even when the display mode of the importance mark 104 is changed depending on the type of unrecognized portion, the display mode of the importance mark 104 according to the importance 98 may be maintained, as in the above embodiment. Furthermore, the importance 98 may be changed depending on the type of unrecognized portion, and the first importance mark 104A, the second importance mark 104B, and the third importance mark 104C may be selectively displayed depending on the changed importance 98.

[0151] In the above embodiment, an example was given in which the importance 98 is defined as one of three levels: "high," "medium," and "low." However, this is merely an example, and the importance 98 may be one or two of "high," "medium," and "low." In this case, the importance marks 104 may also be defined so as to be distinguishable for each level of importance 98. For example, if the importance 98 is only "high" and "medium," the first importance mark 104A and the second importance mark 104B may be selectively displayed in the medical support image 41 according to the importance 98, and the third importance mark 104C may not be displayed in the medical support image 41.

[0152] The importance 98 may be divided into four or more levels, and in this case, the importance marks 104 may be set so as to be distinguishable for each level of the importance 98.

[0153] In the above embodiment, an example is given in which the medical support image 41 is displayed on the screen 37, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 11 , a medical support image 110 may be displayed on the screen 37 instead of the medical support image 41.

[0154] The unrecognized information 100 is displayed on the screen 37 as a medical support image 110 by the control unit 70C. The medical support image 110 is an example of a "schematic diagram" and a "second schematic diagram" according to the technology of the present disclosure. The importance information 102 is displayed in the medical support image 110 by the control unit 70C as an importance mark 112 instead of the importance mark 104 described in the above embodiment. The medical support image 110 is a schematic diagram showing a schematic perspective view of the stomach. The importance mark 104 is a curved mark and is attached to each of the multiple parts described in the above embodiment. In the example shown in FIG. 11 , the importance mark 104 is attached to a location along the inner wall of the stomach shown in the medical support image 110.

[0155] In the example shown in Fig. 11, a first importance mark 112A is shown instead of the first importance mark 104A described in the above embodiment. Also, in the example shown in Fig. 11, a second importance mark 112B is shown instead of the second importance mark 104B described in the above embodiment. Furthermore, in the example shown in Fig. 11, a third importance mark 112C is shown instead of the third importance mark 104C described in the above embodiment.

[0156] The second importance mark 112B is displayed in a more emphasized state than the third importance mark 112C. Furthermore, the first importance mark 112A is displayed in a more emphasized state than the second importance mark 112B. In the example shown in Fig. 11 , the line thickness of the second importance mark 112B is thicker than the line thickness of the third importance mark 112C, and the line thickness of the first importance mark 112A is thicker than the line thickness of the second importance mark 112B.

[0157] In the above embodiment, an example was given in which the circular mark 108 corresponding to the part recognized by the recognition unit 70B was filled in with a specific color, but in the example shown in Fig. 11, the importance mark 112 corresponding to the part recognized by the recognition unit 70B is erased. As a result, the part in the medical support image 110 to which the importance mark 104 is attached is displayed in a more emphasized state than the part in the medical support image 110 from which the importance mark 112 has been erased.

[0158] This allows the doctor 14 to easily visually understand that the areas in the medical support image 110 where the importance marks 112 remain correspond to areas not recognized by the recognition unit 70B, and the areas where the importance marks 112 have been erased correspond to areas recognized by the recognition unit 70B. Note that the importance marks 112 in the medical support image 110 are an example of a "first image" according to the technology of the present disclosure, and the areas in the medical support image 110 where the importance marks 112 have been erased are an example of a "second image" according to the technology of the present disclosure.

[0159] 11 , the doctor 14 can visually determine which parts of the stomach are not recognized by the recognition unit 70B from the positions of the importance marks 104 in the medical support image 110. Furthermore, by determining whether the first importance mark 112A, the second importance mark 112B, or the third importance mark 112C remains in the medical support image 110, the doctor 14 can visually determine whether the part is likely to be overlooked by the recognition unit 70B. Thus, even when the medical support image 110 is displayed on the screen 37, the same effects as those of the above embodiment can be expected.

[0160] As an example, as shown in FIG. 12 , a medical support image 114 may be displayed on the screen 37 instead of the medical support image 41 described in the above embodiment. In this case, the unrecognized information 100 is displayed on the screen 37 as the medical support image 114 by the control unit 70C. The medical support image 114 is an example of a "schematic diagram" and a "third schematic diagram" according to the technology of the present disclosure. The importance information 102 is displayed in the medical support image 114 by the control unit 70C as an importance mark 116 instead of the importance mark 104 described in the above embodiment. The medical support image 114 is a schematic diagram showing a schematic development of the stomach. The medical support image 114 is divided into multiple regions by major category and by minor category. The importance marks 116 are oval marks distributed in the medical support image 114 at locations corresponding to the multiple regions described in the above embodiment.

[0161] In the example shown in Fig. 12, a first importance mark 116A is shown instead of the first importance mark 104A described in the above embodiment. Also, in the example shown in Fig. 12, a second importance mark 116B is shown instead of the second importance mark 104B described in the above embodiment. Furthermore, in the example shown in Fig. 12, a third importance mark 116C is shown instead of the third importance mark 104C described in the above embodiment.

[0162] The second importance mark 116B is displayed in a more emphasized state than the third importance mark 116C. Furthermore, the first importance mark 116A is displayed in a more emphasized state than the second importance mark 116B. The first importance mark 116A, the second importance mark 116B, and the third importance mark 116C are all in different colors, with the color of the second importance mark 116B being darker than the color of the third importance mark 116C, and the color of the first importance mark 116A being darker than the color of the second importance mark 116B.

[0163] In the above embodiment, an example was given in which the circular mark 108 corresponding to the part recognized by the recognition unit 70B was filled in with a specific color, but in the example shown in Fig. 12, the importance mark 116 corresponding to the part recognized by the recognition unit 70B is erased. As a result, the part in the medical support image 114 to which the importance mark 116 is added is displayed in a more emphasized state than the part in the medical support image 114 from which the importance mark 116 has been erased.

[0164] This allows the doctor 14 to easily visually understand that the areas in the medical support image 114 where the importance marks 116 remain correspond to areas not recognized by the recognition unit 70B, and the areas where the importance marks 116 have been erased correspond to areas recognized by the recognition unit 70B. Note that the importance marks 116 in the medical support image 114 are an example of a "first image" according to the technology of the present disclosure, and the areas in the medical support image 114 where the importance marks 116 have been erased are an example of a "second image" according to the technology of the present disclosure.

[0165] 12 , the doctor 14 can visually determine which parts of the stomach are not recognized by the recognition unit 70B from the positions of the importance marks 116 in the medical support image 114. Furthermore, by determining whether the first importance mark 116A, the second importance mark 116B, or the third importance mark 116C remains in the medical support image 114, the doctor 14 can visually determine whether the part is likely to be overlooked by the recognition unit 70B. Thus, even when the medical support image 114 is displayed on the screen 37, the same effects as those of the above embodiment can be expected.

[0166] 12, the control unit 70C displays a reference image 118 on the screen 37 alongside the medical support image 114. The reference image 118 is divided into a plurality of regions 120. In the example shown in FIG. 12, the donon, upper body, middle body, lower body, angle of the stomach, antrum, and pylorus are shown as examples of the plurality of regions 120. The plurality of regions 120 are displayed on the screen 37 so as to be contrasted with the locations of the sites classified into major categories in the medical support image 114. The reference image 118 also displays an insertion section image 122 that allows the current position of the insertion section 44 of the endoscope main body 18 to be identified. The insertion section image 122 is an image that simulates the insertion section 44. The shape and position of the insertion section image 122 are linked to the shape and position of the actual insertion section 44.

[0167] The actual shape and position of the insertion portion 44 are identified by executing AI processing. For example, the control unit 70C identifies the actual shape and position of the insertion portion 44 by performing processing using a trained model on the operation content of the insertion portion 44 and one or more frames of the endoscopic image 40, and generates an insertion portion image 122 based on the identification result and displays it superimposed on the reference image 118 on the screen 37.

[0168] Here, the trained model used by the control unit 70C is obtained by performing machine learning on a neural network using teacher data in which, for example, the operation content of the insertion unit 44 and an image corresponding to one or more frames of the endoscopic image 40 are used as example data, and the shape and position of the insertion unit 44 are used as correct answer data.

[0169] 8 shows an example in which a medical support image 41 is displayed on the screen 37, the example in Fig. 11 shows an example in which a medical support image 110 is displayed on the screen 37, and the example in Fig. 12 shows an example in which a medical support image 114 is displayed on the screen 37, but these are merely examples. For example, the medical support images 41, 110, and 114 may be selectively displayed, or two or more of the medical support images 41, 110, and 114 may be displayed side by side (i.e., in a state in which they can be compared).

[0170] In the above embodiment, an example was described in which the importance 98 assigned to multiple regions was determined based on past test data performed on the multiple regions. However, the technology of the present disclosure is not limited to this. For example, the importance 98 assigned to multiple regions may be determined based on the position of the unrecognized region in the stomach. A region spatially farthest from the position of the tip portion 46 is more likely to be missed by the recognition unit 70B than a region spatially closer to the position of the tip portion 46. Therefore, an example of the position of the unrecognized region in the stomach is the position of the unrecognized region spatially farthest from the position of the tip portion 46. In this case, the position of the unrecognized region spatially farthest from the position of the tip portion 46 changes depending on the position of the tip portion 46, so the importance 98 assigned to the multiple regions changes depending on the position of the tip portion 46 and the position of the unrecognized region in the stomach. In this way, by determining the importance 98 assigned to the multiple regions based on the position of the unrecognized region in the stomach, it is possible to prevent the recognition unit 70B from missing a region with a high importance 98 determined based on the position of the unrecognized region in the stomach.

[0171] In the above embodiment, an example was described in which the importance 98 assigned to the multiple parts was determined according to an external instruction, but the technology of the present disclosure is not limited to this. For example, the importance 98 corresponding to a part of the multiple parts that is scheduled to be recognized by the recognition unit 70B before a specified part (e.g., a part corresponding to a predetermined checkpoint) may be set higher than the importance 98 corresponding to a part of the multiple parts that is scheduled to be recognized after the specified part. This makes it possible to prevent parts that are scheduled to be recognized by the recognition unit 70B before the specified part from being overlooked.

[0172] In the above embodiment, an example was described in which unrecognized parts were set regardless of whether they were classified into major categories or minor categories among the multiple parts, but the technology of the present disclosure is not limited to this. For example, because the recognition unit 70B is more likely to miss recognition of parts classified into minor categories than to miss recognition of parts classified into major categories, the unrecognized parts may be set only for parts classified into minor categories among the multiple parts. This makes it less likely that the recognition unit 70B will miss recognition compared to a case in which the recognition unit 70B misses recognition of both major categories and minor categories.

[0173] In the above embodiment, an example was given in which, when a part classified into a minor category is not recognized by the recognition unit 70B, unrecognized information 100 is output on the condition that a part classified into a major category that is scheduled to be recognized by the recognition unit 70B after the part not recognized by the recognition unit 70B is recognized by the recognition unit 70B, but the technology of the present disclosure is not limited to this.

[0174] For example, when a site classified into a minor category is not recognized by the recognition unit 70B, the unrecognized information 100 may be output on the condition that a site classified into a minor category that is scheduled to be recognized by the recognition unit 70B after the site not recognized by the recognition unit 70B (i.e., the site classified into a minor category) is recognized by the recognition unit 70B. This allows the doctor 14 to understand that a site within the observation target 21 has been overlooked in recognition when there is a high possibility that a site within the observation target 21 (here, as an example, a site classified into a minor category) has been overlooked.

[0175] Here, the plurality of parts classified into the small categories among the plurality of parts are an example of the "plurality of small category parts" according to the technology of the present disclosure. A part among the plurality of parts classified into the small categories that was not recognized by the recognition unit 70B is an example of the "first small category part" according to the technology of the present disclosure. A part classified into the small category that is scheduled to be recognized by the recognition unit 70B after the part that was not recognized by the recognition unit 70B (i.e., the part classified into the small category) is an example of the "second small category part" according to the technology of the present disclosure.

[0176] Furthermore, for example, when a site classified into a minor category is not recognized by the recognition unit 70B, the unrecognized information 100 may be output on the condition that a plurality of sites classified into a minor category that are scheduled to be recognized by the recognition unit 70B after the site not recognized by the recognition unit 70B (i.e., the site classified into a minor category) have been recognized by the recognition unit 70B. In this case as well, in a situation where there is a high possibility that a site within the observation target 21 (here, as an example, a site classified into a minor category) has been overlooked, the doctor 14 can be made to understand that a site within the observation target 21 has been overlooked.

[0177] Here, the multiple areas classified into a subcategory that are scheduled to be recognized by the recognition unit 70B later than the areas that were not recognized by the recognition unit 70B (i.e., the areas classified into a subcategory) are an example of the ``multiple second subcategory areas'' related to the technology disclosed herein.

[0178] In the above embodiment, an example of the unrecognized information 100 being output from the control unit 70C to the display device 13 has been described, but the technology of the present disclosure is not limited to this. For example, the unrecognized information 100 may be stored in the headers of various images such as the endoscopic image 40. For example, if a region not recognized by the recognition unit 70B is classified into a minor category, information indicating that the region is classified into the minor category and / or information enabling identification of the region may be stored in the headers of various images such as the endoscopic image 40. Furthermore, for example, if a region not recognized by the recognition unit 70B is classified into a major category, information indicating that the region is classified into the major category and / or information enabling identification of the region may be stored in the headers of various images such as the endoscopic image 40.

[0179] In addition, information regarding the recognition order including major and minor classifications (i.e., the order of the areas recognized by the recognition unit 70B) and / or information regarding the ultimately unrecognized areas (i.e., the areas not recognized by the recognition unit 70B) may be transmitted to an examination system communicatively connected to the endoscope 12 and stored as examination data by the examination system or included in the examination diagnosis report.

[0180] In the above embodiment, an example has been described in which the camera 48 sequentially images a plurality of regions of the greater curvature-side path 106A from the upstream side of the stomach (i.e., the entrance side of the stomach) to the downstream side (i.e., the exit side of the stomach), and then the camera 48 sequentially images the lesser curvature-side path 106B from the downstream side to the upstream side of the stomach (i.e., an example in which regions are imaged according to the planned recognition order 97), but the technology of the present disclosure is not limited to this. For example, when the recognition unit 70B sequentially recognizes a first region upstream of the insertion direction of the insertion unit 44 inserted into the stomach (e.g., the posterior wall of the upper stomach body) to a second region downstream of the insertion direction (e.g., the posterior wall of the lower stomach body), the processor 70 estimates that imaging is being performed according to a first path (here, as an example, the greater curvature-side path 106A) defined from the upstream side to the downstream side of the insertion unit 44, and outputs the unrecognized information 100 according to the first path. Furthermore, for example, when the recognition unit 70B recognizes in order from a third region downstream in the insertion direction of the insertion unit 44 inserted into the stomach (e.g., the posterior wall of the lower stomach body) to a fourth region upstream (e.g., the posterior wall of the upper stomach body), the processor 70 estimates that imaging is being performed along a second path (here, as an example, the lesser curvature side path 106B) defined from the downstream side to the upstream side of the insertion unit 44, and outputs the unrecognized information 100 along the second path. This makes it possible to easily identify whether a region on the greater curvature side path 106A has not been recognized by the recognition unit 70B or a region on the lesser curvature side path 106B has not been recognized by the recognition unit 70B.

[0181] Here, the greater curvature-side path 106A is given as an example of the first path, and the lesser curvature-side path 106B is given as an example of the second path, but the first path may be the lesser curvature-side path 106B, and the second path may be the greater curvature-side path 106A. Here, the upstream side in the insertion direction refers to the entrance side of the stomach (i.e., the esophagus side), and the downstream side in the insertion direction refers to the exit side of the stomach (i.e., the duodenum side).

[0182] In the above embodiment, an example was described in which medical support processing was performed by the processor 70 of the computer 64 included in the endoscope 12, but the technology of the present disclosure is not limited to this, and the device that performs medical support processing may be provided outside the endoscope 12. Examples of devices that may be provided outside the endoscope 12 include at least one server and / or at least one personal computer that are communicably connected to the endoscope 12. Furthermore, medical support processing may be performed in a distributed manner by multiple devices.

[0183] In the above embodiment, an example has been described in which the medical support processing program 76 is stored in the NVM 74, but the technology of the present disclosure is not limited to this. For example, the medical support processing program 76 may be stored in a portable non-transitory storage medium such as an SSD or USB memory. The medical support processing program 76 stored in the non-transitory storage medium is installed in the computer 64 of the endoscope 12. The processor 70 executes medical support processing in accordance with the medical support processing program 76.

[0184] Alternatively, the medical support processing program 76 may be stored in a storage device such as another computer or server connected to the endoscope 12 via a network, and the medical support processing program 76 may be downloaded and installed on the computer 64 in response to a request from the endoscope 12.

[0185] It is not necessary to store the entire medical support processing program 76 in the storage device of another computer or server device connected to the endoscope 12, or in the NVM 74; only a portion of the medical support processing program 76 may be stored therein.

[0186] The hardware resources that execute the medical support processing can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes medical support processing by executing software, i.e., a program. Examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with a circuit configuration specifically designed to execute specific processing. Each processor has built-in or connected memory, and executes medical support processing by using the memory.

[0187] The hardware resource for executing the medical support processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the medical support processing may be a single processor.

[0188] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute medical support processing, on a single IC chip, as typified by SoC. In this way, medical support processing is realized using one or more of the various processors described above as hardware resources.

[0189] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0190] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0191] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0192] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A processor is provided. The processor, Recognizing a plurality of regions within an object of observation based on a plurality of medical images including the object of observation; When an unrecognized site exists in the observation target among the plurality of sites, unrecognized information capable of identifying the existence of the unrecognized site is output. Medical support equipment.

2. A processor comprising: The processor, Recognizing a plurality of regions within an object of observation based on a plurality of medical images including the object of observation; outputting unrecognized information capable of identifying the presence of an unrecognized site when an unrecognized site in the observation target is present among the plurality of sites; the plurality of sites includes a subsequent site that is scheduled to be recognized by the processor after the unrecognized site; The processor outputs the unrecognized information on the condition that the subsequent portion is recognized. Medical support equipment.

3. The processor outputs the unrecognized information based on a first order, which is an order in which the plurality of sites are recognized by the processor, and a second order, which is an order in which a plurality of planned sites including the unrecognized site, which are scheduled to be recognized by the processor, are recognized by the processor. The medical support device according to claim 1 .

4. Importance is assigned to the plurality of parts, The unrecognized information includes importance information that allows the importance to be specified. The medical support device according to claim 1 .

5. The importance is determined according to an externally given instruction. The medical support device according to claim 4.

6. The importance is determined according to past inspection data performed on the plurality of regions. The medical support device according to claim 4.

7. The importance is determined according to the location of the unrecognized site within the object of observation. The medical support device according to claim 4.

8. A processor comprising: The processor, Recognizing a plurality of regions within an object of observation based on a plurality of medical images including the object of observation; outputting unrecognized information capable of identifying the presence of an unrecognized site when an unrecognized site in the observation target is present among the plurality of sites; Importance is assigned to the plurality of parts, The unrecognized information includes importance information that can identify the importance, The importance level corresponding to a part of the plurality of parts that is scheduled to be recognized by the processor before a designated part is higher than the importance level corresponding to a part of the plurality of parts that is scheduled to be recognized after the designated part. Medical support equipment.

9. The importance level corresponding to a part determined as a part typically prone to oversight of recognition among the plurality of parts is higher than the importance level corresponding to a part determined as a part typically unlikely to oversight of recognition among the plurality of parts. The medical support device according to claim 4.

10. The plurality of sites are classified into major categories and minor categories included in the major categories, The importance level corresponding to the site of the plurality of sites classified into the small classification is higher than the importance level corresponding to the site of the plurality of sites classified into the large classification. The medical support device according to claim 4.

11. The plurality of sites are classified into major categories and minor categories included in the major categories, The unrecognized site is a site that is classified into the subcategory among the plurality of sites. The medical support device according to claim 1 .

12. A processor comprising: The processor, Recognizing a plurality of regions within an object of observation based on a plurality of medical images including the object of observation; outputting unrecognized information capable of identifying the presence of an unrecognized site when an unrecognized site in the observation target is present among the plurality of sites; The plurality of sites are classified into major categories and minor categories included in the major categories, the unrecognized site is a site classified into the subcategory among the plurality of sites, The major classification is divided into a first major classification and a second major classification, the portions classified into the second major category are scheduled to be recognized by the processor later than the portions classified into the first major category; the unrecognized site is a site belonging to the minor category included in the first major category among the plurality of sites, The processor outputs the unrecognized information on the condition that a part classified into the second major category among the plurality of parts is recognized. Medical support equipment.

13. A processor comprising: The processor, Recognizing a plurality of regions within an object of observation based on a plurality of medical images including the object of observation; outputting unrecognized information capable of identifying the presence of an unrecognized site when an unrecognized site in the observation target is present among the plurality of sites; The plurality of sites are classified into major categories and minor categories included in the major categories, the unrecognized site is a site classified into the subcategory among the plurality of sites, The plurality of sites includes a plurality of sub-classification sites classified into the sub-classifications, The plurality of small-classified regions include a first small-classified region and a second small-classified region that is scheduled to be recognized by the processor after the first small-classified region, the unrecognized site is the first subclass site, The processor outputs the unrecognized information on the condition that the second small classification portion is recognized. Medical support equipment.

14. A processor comprising: The processor, Recognizing a plurality of regions within an object of observation based on a plurality of medical images including the object of observation; outputting unrecognized information capable of identifying the presence of an unrecognized site when an unrecognized site in the observation target is present among the plurality of sites; The plurality of sites are classified into major categories and minor categories included in the major categories, the unrecognized site is a site classified into the subcategory among the plurality of sites, The plurality of sites includes a plurality of sub-classification sites belonging to the sub-classification, The plurality of small-classified regions includes a first small-classified region and a plurality of second small-classified regions that are scheduled to be recognized by the processor after the first small-classified region, the unrecognized site is the first subclass site, The processor outputs the unrecognized information on the condition that the plurality of second small classification parts are recognized. Medical support equipment.

15. The output destination of the unrecognized information includes a display device. The medical support device according to claim 1 .

16. the unrecognized information includes a first image capable of identifying the unrecognized portion and a second image capable of identifying other portions of the plurality of portions other than the unrecognized portion, The first image and the second image are displayed on the display device in a distinguishable manner. The medical support device according to claim 15.

17. The display device displays a schematic diagram in which the observation target is divided into a plurality of regions corresponding to the plurality of parts, and the first image and the second image are displayed in a distinguishable manner within the schematic diagram. The medical support device according to claim 16.

18. the observation target is a hollow organ, The schematic diagram is a first schematic diagram showing a schematic aspect of at least one path for observing the hollow organ, a second schematic diagram showing a schematic aspect of the hollow organ seen through, and / or a third schematic diagram showing a schematic aspect of the hollow organ in an expanded state. The medical support device according to claim 17.

19. The display device displays the first image in a more emphasized state than the second image. The medical support device according to claim 16.

20. Importance is assigned to the plurality of parts, The display mode of the first image varies depending on the importance. The medical support device according to claim 16.

21. The display mode of the first image differs depending on the type of the unrecognized portion. The medical support device according to claim 16.

22. the medical image is an image obtained from an endoscope inserted into a body; The processor, when recognizing a first portion on the upstream side to a second portion on the downstream side in an insertion direction of the endoscope inserted into the body, outputting the unrecognized information along a first path defined from the upstream side to the downstream side in the insertion direction, When the third portion on the downstream side in the insertion direction is recognized in order from the fourth portion on the upstream side in the insertion direction, the unrecognized information is output along a second path defined from the downstream side to the upstream side in the insertion direction. The medical support device according to claim 1 .

23. A medical support device according to any one of claims 1 to 22, an image acquisition device for acquiring an endoscopic image as the medical image; An endoscope comprising:

24. Recognizing a plurality of regions within an object of observation based on a plurality of medical images including the object of observation; and and outputting unrecognized information capable of identifying the presence of an unrecognized site when an unrecognized site in the observation target is present among the plurality of sites. Medical assistance methods.

25. Recognizing a plurality of regions within an object of observation based on a plurality of medical images including the object of observation; and A program for causing a computer to execute a process including: when an unrecognized site is present in the object of observation among the plurality of sites, outputting unrecognized information capable of identifying the presence of the unrecognized site.

Images