Medical support device, endoscope system, medical support method, and program

Abstract

To provide a medical support device, an endoscope system, a medical support method, and a program that allow a user or the like to accurately grasp the position of a lumen in a medical image.SOLUTION: The medical support device includes a processor. The processor inputs, to the learned model, a medical image generated by imaging an inside of a luminal organ including a lumen to generate a plurality of certainty factors corresponding to a plurality of division regions obtained by dividing the medical image or an image corresponding to the medical image along a circumferential direction, the plurality of certainty factors indicating that the lumen appears in the plurality of division regions. Further, the processor outputs, based on the plurality of sectional regions and the plurality of certainty factors, lumen identification information capable of identifying a lumen presence region in which the presence position of the lumen is identified more finely than in the sectional regions in the medical image or in the image corresponding to the medical image.SELECTED DRAWING: Figure 18

Description

[Technical Field]

[0001] The present disclosure relates to a medical support device, an endoscope system, a medical support method, and a program. [Background technology]

[0002] Patent Document 1 discloses an endoscope insertion direction detection method including a first step of inputting an endoscopic image, a second step of detecting the direction of light-dark changes in the endoscopic image, and a third step of generating information related to the insertion direction of the endoscope based on the detection results. Patent Document 1 also discloses an endoscope insertion direction detection method including a first step of setting candidate insertion directions that serve as candidates for the insertion direction of the endoscope, a second step of inputting an endoscopic image, a third step of detecting the direction of light-dark changes in the endoscopic image, a fourth step of evaluating the similarity between the multiple candidate insertion directions and the directions of light-dark changes, and a fifth step of determining the insertion direction of the endoscope based on the evaluation results.

[0003] Patent Document 2 discloses a mobility assistance system that includes a multiple operation information calculation unit that calculates multiple operation information indicating multiple operations that differ over time, corresponding to a multiple operation target scene, which is a scene that requires multiple operations that differ over time, based on an image captured by an imaging unit arranged in an insertion unit, and a presentation information generation unit that generates presentation information for the insertion unit based on the multiple operation information calculated by the multiple operation information calculation unit. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-093328 [Patent Document 2] International Publication No. 2020 / 194472 Summary of the Invention

[0005] One embodiment of the present disclosure provides a medical support device, an endoscopic system, a medical support method, and a program that enable a user or the like to accurately grasp the position of a lumen shown in a medical image within the medical image. [Means for solving the problem]

[0006] A first aspect of the present disclosure is a medical support device that includes a processor, and the processor inputs a medical image generated by imaging the inside of a hollow organ, including the lumen, into a trained model, thereby generating a plurality of confidence levels corresponding to a plurality of partitioned areas obtained by dividing the medical image or an image corresponding to the medical image along a circumferential direction, where the plurality of partitioned areas depict a lumen, and outputs lumen identification information that can identify a lumen existence area in the medical image or an image corresponding to the medical image in which the location of the lumen is identified more precisely than in the partitioned areas, based on the plurality of partitioned areas and the plurality of confidence levels.

[0007] A second aspect of the present disclosure is a medical support device according to the first aspect, in which the lumen presence region is a region in which the position where the lumen is captured in the medical image can be identified with higher resolution along the circumferential direction than the multiple divided regions.

[0008] A third aspect of the present disclosure is a medical support device according to the first or second aspect, in which the direction from a reference position of a medical image or an image corresponding to the medical image to the respective locations of a plurality of divided regions is determined by a plurality of first vectors, the direction from the reference position to the lumen existence region is determined by a second vector, the second vector is the sum of at least two third vectors obtained by assigning a confidence level as a weight to at least two of the plurality of first vectors, and the lumen identification information is information determined based on the second vectors.

[0009] A fourth aspect of the present disclosure is a medical support device according to any one of the first to third aspects, in which a processor outputs a medical image or an image corresponding to the medical image, and the lumen identification information is updated in accordance with the output timing of the medical image or the image corresponding to the medical image.

[0010] A fifth aspect of the present disclosure is a medical support device according to any one of the first to fourth aspects, in which output of lumen-specific information is achieved by displaying the lumen-specific information on a screen.

[0011] A sixth aspect of the present disclosure is a medical support device according to the fifth aspect, in which a medical image or an image corresponding to the medical image is displayed on a screen, and the lumen identification information displayed on the screen is updated in accordance with the display timing of the medical image or the image corresponding to the medical image.

[0012] A seventh aspect of the present disclosure is a medical support device according to the fifth or sixth aspect, in which medical images and / or images corresponding to the medical images are displayed on a screen so that lumen identification information can be compared.

[0013] An eighth aspect of the present disclosure is the medical support device according to the seventh aspect, in which the lumen-specifying information is superimposed on the medical image and / or an image corresponding to the medical image.

[0014] A ninth aspect of the present disclosure is a medical support device according to any one of the first to eighth aspects, wherein the direction from a reference position of a medical image or an image corresponding to the medical image to the respective locations of a plurality of divided areas is determined by a plurality of first vectors, the direction from the reference position to the lumen existence area is determined by a second vector, the second vector is the sum of at least two third vectors obtained by assigning a confidence level as a weight to at least two of the plurality of first vectors, and the lumen identification information includes a mark that can identify an area within the medical image or an image corresponding to the medical image that has been determined as the lumen existence area based on the second vectors.

[0015] A tenth aspect of the present disclosure is the medical support device according to the ninth aspect, in which the shape of the mark is an arc, and the center of the arc is the center of the medical image or an image corresponding to the medical image.

[0016] An eleventh aspect of the present disclosure is the medical support device according to the ninth aspect, in which the shape of the mark is a shape that follows the outer edge of the medical image or the outer edge of an image corresponding to the medical image.

[0017] A twelfth aspect of the present disclosure is a medical support device according to any one of the ninth to eleventh aspects, in which a medical image or an image corresponding to the medical image is associated with a plurality of hidden markers, and a mark is displayed on the screen by displaying at least one marker among the plurality of markers that corresponds to the position of the lumen existence area.

[0018] A thirteenth aspect of the present disclosure is an endoscopic system comprising a medical support device according to any one of the first to twelfth aspects and an endoscopic scope, in which medical images are generated by imaging the inside of a tubular organ, including the lumen, using the endoscopic scope.

[0019] A fourteenth aspect of the present disclosure is a medical support method that includes inputting a medical image generated by imaging the inside of a hollow organ, including the lumen, into a trained model, thereby generating a plurality of confidence levels corresponding to a plurality of partitioned areas obtained by dividing the medical image or an image corresponding to the medical image along a circumferential direction, where the plurality of partitioned areas depict a lumen; and outputting lumen identification information that can identify a lumen existence area in the medical image or an image corresponding to the medical image in which the location of the lumen is identified more precisely than in the partitioned areas, based on the plurality of partitioned areas and the plurality of confidence levels.

[0020] A fifteenth aspect of the present disclosure is a program for causing a computer to execute processing including: inputting a medical image generated by imaging the inside of a hollow organ, including the lumen, into a trained model, thereby generating a plurality of confidence levels corresponding to a plurality of partitioned areas obtained by dividing the medical image or an image corresponding to the medical image along a circumferential direction, wherein the plurality of partitioned areas depict a lumen; and outputting lumen identification information that can identify a lumen existence area in the medical image or an image corresponding to the medical image in which the location of the lumen is identified more precisely than in the partitioned areas, based on the plurality of partitioned areas and the plurality of confidence levels. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a conceptual diagram showing an example of how the endoscope system is used by a doctor. [Figure 2] 1 is a conceptual diagram showing an example of the overall configuration of an endoscope system. [Figure 3] FIG. 2 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope system. [Figure 4] 2 is a block diagram showing an example of the main functions of a processor included in the medical support device and an example of information stored in a storage. FIG. [Figure 5] FIG. 2 is a block diagram illustrating an example of an electrical hardware configuration of the information processing apparatus. [Figure 6] FIG. 1 is a conceptual diagram illustrating an example of how teacher data is generated by an information processing device. [Figure 7] FIG. 10 is a conceptual diagram showing an example of a sample image. [Figure 8] FIG. 8 is a conceptual diagram showing an example of training data that is generated when a lumen is captured in an area other than the central area of ​​the example image shown in FIG. 7. [Figure 9] FIG. 8 is a conceptual diagram showing an example of training data generated when a lumen is captured in the central region of the example image shown in FIG. 7. [Figure 10]FIG. 10 is a conceptual diagram showing an example of processing content in an information processing device when a lumen recognition model is generated by performing machine learning on a model using training data. [Figure 11] FIG. 2 is a conceptual diagram showing an example of processing content of a recognition unit of the medical processing device. [Figure 12] FIG. 10 is a conceptual diagram showing an example of certainty information generated by a lumen recognition model when a lumen is captured in a frame. [Figure 13] FIG. 10 is a conceptual diagram showing an example of the relationship between a plurality of partitioned regions obtained by radially partitioning a map included in the certainty factor information and a plurality of directional unit vectors assigned to the plurality of partitioned regions. [Figure 14] FIG. 2 is a conceptual diagram showing an example of processing contents of a control unit of a medical processing device. [Figure 15] This is a conceptual diagram showing an example of an embodiment in which, when a lumen is displayed in an area other than the central area of ​​a frame, the frame is displayed in a first display area of ​​the screen, a mark that can identify the position within the frame of the lumen presence area that identifies the location of the lumen displayed in the frame is superimposed on the frame, and visible information is displayed in a second display area of ​​the screen as one piece of auxiliary information. [Figure 16] This is a conceptual diagram showing an example of an embodiment in which, when a lumen is shown in the central area of ​​a frame, the frame is displayed in the first display area of ​​the screen, a mark that can identify the position within the frame of the lumen presence area that identifies the location of the lumen shown in the frame is superimposed on the frame, and visible information is displayed in the second display area of ​​the screen as one piece of auxiliary information. [Figure 17] 10 is a flowchart illustrating an example of the flow of a machine learning process. [Figure 18] 10 is a flowchart showing an example of the flow of medical support processing. [Figure 19] FIG. 10 is a conceptual diagram showing an example of a form in which a mark is displayed along the outer edge of a frame displayed on a screen. [Figure 20]FIG. 10 is a conceptual diagram showing an example of a form in which at least one marker corresponding to the position of a lumen presence region is displayed among a plurality of non-display markers associated with a map and / or frame, thereby superimposing a mark on the frame. [Figure 21] 10 is a conceptual diagram showing an example of a form in which an outer contour line, which is a line that outlines the outer contour of a lumen existing region, is superimposed on a frame. FIG. [Figure 22] A conceptual diagram showing an example of a series of processes in which a processor included in a computer issues a processing execution request to an external device via a network, the external device executes processing in response to the processing execution request, and the processor included in the computer receives the processing result from the external device. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, exemplary embodiments of a medical support device, an endoscope system, a medical support method, and a program according to the present disclosure will be described with reference to the accompanying drawings. Note that the present disclosure can also be applied to a program and a computer program product.

[0023] First, the terms used in the following description will be explained.

[0024] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". GPGPU is an abbreviation for "General-Purpose computing on Graphics Processing Units". APU is an abbreviation for "Accelerated Processing Unit". TPU is an abbreviation for "Tensor Processing Unit". RAM is an abbreviation for "Random Access Memory". EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". I / F is an abbreviation for "Interface". LAN is an abbreviation for "Local Area Network". WAN is an abbreviation for "Wide Area Network". 5G is an abbreviation for "5th Generation Mobile Communication System".

[0025] In the following description, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU, a GPU, a GPGPU, an APU, and a TPU.

[0026] In the following description, a signed memory is a memory such as a RAM in which information is temporarily stored, and is used as a work memory by a processor.

[0027] In the following description, the term "storage" refers to one or more nonvolatile storage devices that store various programs, various parameters, etc. Examples of nonvolatile storage devices include flash memory, magnetic disks, and magnetic tapes. Another example of storage is cloud storage.

[0028] In the following embodiments, the external I / F with a symbol controls the exchange of various information between multiple devices connected to each other. An example of the external I / F is a USB interface. A communication I / F including a communication processor, an antenna, etc. may be applied to the external I / F. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] Fig. 1 is a conceptual diagram showing an example of an embodiment in which an endoscope system 10 is used. As shown in Fig. 1, the endoscope system 10 is used by a doctor 12 in an endoscopic examination or the like. The endoscopic examination is assisted by a staff member 14, such as a nurse.

[0031] The endoscope system 10 is communicably connected to a communication device (not shown), and information obtained by the endoscope system 10 is transmitted to the communication device. Examples of the communication device include a server that manages various information such as electronic medical records, a personal computer, and / or a tablet terminal. The communication device receives the information transmitted from the endoscope system 10 and performs processing using the received information (for example, processing to store the information in an electronic medical record, etc.).

[0032] The endoscopic system 10 includes an endoscope 16, a display device 18, a light source device 20, a control device 22, and a medical support device 24. In this embodiment, the endoscopic system 10 is an example of an "endoscopic system" according to the present disclosure, the endoscope 16 is an example of an "endoscopic scope" according to the present disclosure, and the medical support device 24 is an example of a "medical support device" according to the present disclosure.

[0033] The endoscope system 10 is a modality for performing medical examinations on a large intestine 28, which is a hollow organ contained in the body of a subject 26 (e.g., a patient), using an endoscope 16. In this embodiment, the large intestine 28 is an object to be observed by a doctor 12.

[0034] The endoscope 16 is used by the doctor 12 and inserted into the body of the subject 26. In this embodiment, the endoscope 16 is inserted into the large intestine 28 of the subject 26. In this embodiment, the large intestine 28 is an example of a "hollow organ" according to the present disclosure.

[0035] The endoscope system 10 causes an endoscope 16 inserted into the large intestine 28 of a subject 26 to capture images of the inside of the large intestine 28, including a lumen 42, and performs various medical procedures on the large intestine 28 as necessary. The large intestine 28 has a lumen 42. The endoscope 16 is inserted into the lumen 42. The position of the lumen 42 within the large intestine 28 can be medically identified based on the morphological pattern (e.g., the shape and orientation of the folds 43) of a plurality of folds 43, which are characteristic regions within the large intestine 28. As will be described in detail below, in this embodiment, the position of the lumen 42 is recognized by AI that has undergone machine learning of various information, such as the morphological pattern of the plurality of folds 43, and the recognition result is provided to the doctor 12 as visually comprehensible information. In this embodiment, the lumen 42 is an example of a "lumen" according to the present disclosure.

[0036] The endoscope system 10 captures an image of the inside of the large intestine 28, including the lumen 42, to obtain and output an image showing the state of the inside of the large intestine 28, including the lumen 42. In this embodiment, the endoscope system 10 has an optical imaging function of capturing an image of reflected light obtained by irradiating the inside of the large intestine 28 with light 30 and reflecting it off the intestinal wall 32 of the large intestine 28.

[0037] Although an endoscopic examination of the large intestine 28 is illustrated here, this is merely one example, and the present disclosure also applies to an endoscopic examination of a hollow organ such as the esophagus, stomach, duodenum, or trachea.

[0038] The light source device 20, the control device 22, and the medical support device 24 are installed on a wagon 34. The wagon 34 has a plurality of stands arranged vertically, and the medical support device 24, the light source device 20, and the control device 22 are installed from the lower stand to the upper stand. In addition, the display device 18 is installed on the top stand of the wagon 34.

[0039] The control device 22 controls the entire endoscope system 10. The control device 22 performs various processes on images obtained by imaging the intestinal wall 32 with the endoscope 16. Furthermore, under the control of the control device 22, the medical support device 24 executes AI processing and the like on the images that have been subjected to various processes by the control device 22, and outputs various information including the results of the AI ​​processing and the like. Examples of destinations to which the various information may be output include the display device 18, a stationary storage medium (for example, a storage device mounted on the endoscope system 10, or a storage device of a server or the like communicatively connected to the endoscope system 10), and / or a portable storage medium (for example, a memory card, a USB flash drive, or the like).

[0040] The display device 18 displays various information (for example, various information output from the medical support device 24). Examples of the display device 18 include a liquid crystal display and an EL display. Alternatively, instead of the display device 18, or together with the display device 18, a tablet terminal with a display may be used.

[0041] A screen 35 is displayed on the display device 18. The screen 35 includes a plurality of display areas. The plurality of display areas are arranged side by side within the screen 35. In the example shown in FIG. 1, a first display area 35A and a second display area 35B are shown as examples of the plurality of display areas. The size of the first display area 35A is larger than the size of the second display area 35B. The first display area 35A is used as the main display area, and the second display area 35B is used as the sub-display area. The size relationship between the first display area 35A and the second display area 35B is not limited to this, and may be any size relationship that fits within the screen 35.

[0042] The first display area 35A displays an endoscopic video 39. The endoscopic video 39 is obtained by performing various processes on a plurality of images in time series obtained by capturing images of the inside of the large intestine 28 of the subject 26 using the endoscope 16. The intestinal wall 32 shown in the endoscopic video 39 includes a lumen 42 as a region of interest (i.e., a region to be observed) that is being gazed upon by the doctor 12, and the doctor 12 can visually recognize the appearance of the intestinal wall 32, including the lumen 42, through the endoscopic video 39.

[0043] The image displayed in the first display area 35A is one frame 40 included in a moving image that includes a plurality of frames 40 arranged in chronological order. That is, the plurality of frames 40 arranged in chronological order are displayed in the first display area 35A at a predetermined frame rate (e.g., a dozen or so frames per second or several tens of frames per second). In this embodiment, the frame 40 is an example of a "medical image" according to the present disclosure.

[0044] An example of a moving image displayed in the first display area 35A is a moving image in a live view format. The live view format is merely one example, and a moving image that is temporarily stored in a memory or the like and then displayed, such as a moving image in a post-view format, may also be used. Furthermore, each frame included in a recording moving image stored in a memory or the like may be played back and displayed on the screen 35 (for example, in the first display area 35A) as an endoscopic moving image 39.

[0045] The display position of the second display area 35B is the lower right corner as viewed from the front within the screen 35. The display position of the second display area 35B may be anywhere within the screen 35 of the display device 18, but it is preferable that it be displayed in a position that can be compared with the endoscopic video image 39. Auxiliary information 44 that assists the doctor 12 in making medical decisions during an endoscopic examination is displayed in the second display area 35B. The auxiliary information 44 is information that is referenced by the doctor 12. Examples of the auxiliary information 44 include various types of information regarding the subject 26 into whose body the endoscope 16 is inserted, and / or various types of information obtained by performing medical support processing, which will be described later.

[0046] Fig. 2 is a conceptual diagram showing an example of the overall configuration of the endoscope system 10. As shown in Fig. 2, the endoscope 16 includes an operation unit 46 and an insertion unit 48. The insertion unit 48 is partially curved by operating the operation unit 46. The insertion unit 48 is inserted into the large intestine 28 (see Fig. 1) while curving in accordance with the shape of the large intestine 28 (see Fig. 1) in accordance with the operation of the operation unit 46 by the doctor 12 (see Fig. 1).

[0047] A camera 52, an illumination device 54, and a treatment tool opening 56 are provided at the distal end 50 of the insertion section 48. A part of the camera 52 (e.g., an imaging optical system) and a part of the illumination device 54 (e.g., an irradiation optical system) are exposed from a distal end surface 50A of the distal end 50.

[0048] The camera 52 is mounted on the endoscope 16 and inserted into the body cavity of the subject 26 to capture an image of the observation area. An example of the camera 52 is a CMOS camera. However, this is merely an example, and other types of cameras, such as a CCD camera, may also be used. In this embodiment, the camera 52 captures an image of the inside of the large intestine 28, including the lumen 42, to generate an image showing the inside of the large intestine 28, including the lumen 42. The image generated by the camera 52 has a circular outline. For example, the image generated by the camera 52 is processed by the control device 22 into a shape in which the upper and lower ends are masked. As a result, an image having linear upper and lower edges and arc-shaped left and right edges is generated as the frame 40, as shown in FIG. 1 .

[0049] The illumination device 54 has illumination windows 54A and 54B. The illumination device 54 emits light 30 (see FIG. 1 ) through the illumination windows 54A and 54B. Examples of the light 30 emitted from the illumination device 54 include visible light (e.g., white light) and invisible light (e.g., near-infrared light). The illumination device 54 also emits special light through the illumination windows 54A and 54B. Examples of the special light include light for BLI and / or light for LCI. The camera 52 captures images of the inside of the large intestine 28 by an optical method while the light 30 is being emitted inside the large intestine 28 by the illumination device 54.

[0050] The treatment tool opening 56 is an opening for allowing a treatment tool 58 to protrude from the distal end portion 50. The treatment tool opening 56 is also used as a suction port for sucking blood, internal waste, etc., and as a delivery port for delivering fluid.

[0051] A treatment tool insertion port 60 is formed in the operation section 46, and the treatment tool 58 is inserted into the insertion section 48 from the treatment tool insertion port 60. The treatment tool 58 passes through the insertion section 48 and protrudes to the outside from the treatment tool opening 56. In the example shown in FIG. 2, a puncture needle is shown as the treatment tool 58 protruding from the treatment tool opening 56. Here, a puncture needle is shown as the treatment tool 58, but this is merely one example, and the treatment tool 58 may also be a grasping forceps, a papillotomy knife, a snare, a catheter, a guidewire, a cannula, and / or a puncture needle with a guide sheath, etc.

[0052] The endoscope 16 is connected to a light source device 20 and a control device 22 via a universal cord 62. A medical support device 24 and a reception device 64 are connected to the control device 22. A display device 18 is also connected to the medical support device 24. That is, the control device 22 is connected to the display device 18 via the medical support device 24.

[0053] Note that, because the medical support device 24 is exemplified here as an external device for expanding the functions performed by the control device 22, an example in which the control device 22 and the display device 18 are indirectly connected via the medical support device 24 is given, but this is merely one example. For example, the display device 18 may be directly connected to the control device 22. In this case, for example, the functions of the medical support device 24 may be installed in the control device 22, or the control device 22 may be equipped with a function for causing a server (not shown) to execute the same processing as that executed by the medical support device 24 (for example, the medical support processing described below) and receiving and using the processing results from the server.

[0054] The reception device 64 receives instructions from the doctor 12 and outputs the received instructions as an electrical signal to the control device 22. Examples of the reception device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and / or a remote control device.

[0055] The control device 22 controls the light source device 20, exchanges various signals with the camera 52, and exchanges various signals with the medical support device 24.

[0056] The light source device 20 emits light under the control of the control device 22 and supplies light 30 to the illumination device 54. The illumination device 54 has a built-in light guide, and the light 30 supplied from the light source device 20 passes through the light guide and is emitted from illumination windows 54A and 54B. The control device 22 causes the camera 52 to capture an image while the light 30 is being emitted from the illumination windows 54A and 54B. The control device 22 processes the outline of the image obtained by the camera 52 when capturing an image, adjusts the image quality, etc., and generates a plurality of frames 40 in chronological order. The control device 22 then outputs the generated endoscopic video 39 including the plurality of frames 40 in chronological order to a predetermined output destination (e.g., the medical support device 24).

[0057] The medical support device 24 supports medical care (here, as an example, endoscopic examination) by performing various processes on the endoscopic moving image 39 input from the control device 22. The medical support device 24 outputs the endoscopic moving image 39 that has been subjected to various processes to a predetermined output destination (for example, the display device 18).

[0058] Although the embodiment in which the endoscopic moving image 39 output from the control device 22 is output to the display device 18 via the medical support device 24 has been described above, this is merely one example. For example, the control device 22 and the display device 18 may be connected, and the endoscopic moving image 39 that has undergone various processes by the medical support device 24 may be displayed on the display device 18 via the control device 22.

[0059] Fig. 3 is a block diagram showing an example of the hardware configuration of the electrical system of the endoscope system 10. As shown in Fig. 3, the control device 22 includes a computer 66, a bus 68, and an external I / F 70. The computer 66 includes a processor 72, a memory 74, and a storage 76. The processor 72, the memory 74, the storage 76, and the external I / F 70 are connected to the bus 68. The processor 72 controls the entire control device 22. The memory 74 and the storage 76 are used by the processor 72.

[0060] The external I / F 70 controls the exchange of various information between the processor 72 and one or more devices (hereinafter also referred to as "first external devices") that exist outside the control device 22.

[0061] The external I / F 70 is connected to the camera 52 as one of the first external devices, and the external I / F 70 controls the exchange of various information between the camera 52 and the processor 72. The processor 72 controls the camera 52 via the external I / F 70. The processor 72 also generates an endoscopic video 39 (see FIG. 1) by performing various processes on images obtained by the camera 52 capturing images of the inside of the large intestine 28 (see FIG. 1), and acquires the generated endoscopic video 39 via the external I / F 70.

[0062] The light source device 20 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 controls the exchange of various information between the light source device 20 and the processor 72. The light source device 20 supplies light to the illumination device 54 under the control of the processor 72. The illumination device 54 irradiates the light supplied from the light source device 20.

[0063] A reception device 64 is connected to the external I / F 70 as one of the first external devices, and the processor 72 acquires instructions accepted by the reception device 64 via the external I / F 70 and executes processing according to the acquired instructions.

[0064] The medical support device 24 includes a computer 78 and an external I / F 80. The computer 78 includes a processor 82, a memory 84, and a storage 86. The processor 82, the memory 84, the storage 86, and the external I / F 80 are connected to a bus 88. In this embodiment, the computer 78 is an example of a "computer" according to the present disclosure, and the processor 82 is an example of a "processor" according to the present disclosure.

[0065] The hardware configuration of the computer 78 (i.e., the processor 82, the memory 84, and the storage 86) is basically the same as the hardware configuration of the computer 66, so a description of the hardware configuration of the computer 78 will be omitted here.

[0066] The external I / F 80 controls the exchange of various information between the processor 82 and one or more devices (hereinafter also referred to as "second external devices") that exist outside the medical support device 24.

[0067] The control device 22 is connected to the external I / F 80 as one of the second external devices. In the example shown in Fig. 3, the external I / F 70 of the control device 22 is connected to the external I / F 80. The external I / F 80 controls the exchange of various information between the processor 82 of the medical support device 24 and the processor 72 of the control device 22. For example, the processor 82 acquires endoscopic moving images 39 (see Fig. 1) from the processor 72 of the control device 22 via the external I / Fs 70 and 80, and performs various processes on the acquired endoscopic moving images 39. The various processes performed by the processor 82 include AI-based processes (for example, processes using a lumen recognition model 92, which will be described later).

[0068] The display device 18, which serves as one of the second external devices, is connected to the external I / F 80. The processor 82 controls the display device 18 via the external I / F 80, thereby causing the display device 18 to display various information (for example, an endoscopic moving image 39 on which various processes have been performed).

[0069] 4 is a block diagram showing an example of the main functions of the processor 82 included in the medical support device 24 and an example of information stored in the storage 86. As shown in FIG. 4, a medical support program 90 is stored in the storage 86. In this embodiment, the medical support program 90 is an example of a "program" according to the present disclosure.

[0070] The processor 82 performs medical support processing by reading the medical support program 90 from the storage 86 and executing the read medical support program 90 on the memory 84. The medical support processing is realized by the processor 82 operating as a recognition unit 82A and a control unit 82B in accordance with the medical support program 90 executed on the memory 84.

[0071] The storage 86 stores a lumen recognition model 92. As will be described in detail later, the lumen recognition model 92 is a trained model used in AI-based processing and is used by the recognition unit 82A. In this embodiment, the lumen recognition model 92 is an example of a "trained model" according to the present disclosure.

[0072] Fig. 5 is a block diagram showing an example of the hardware configuration of an electrical system of an information processing device 100 used to generate a lumen recognition model 92. As shown in Fig. 5, the information processing device 100 includes a computer 102 and an external I / F 104. The computer 102 includes a processor 106, a memory 108, and a storage 110. The processor 106, the memory 108, the storage 110, and the external I / F 104 are connected to a bus 112.

[0073] The hardware configuration of the computer 102 (i.e., the processor 106, memory 108, and storage 110) is basically the same as the hardware configuration of the computer 66, so a description of the hardware configuration of the computer 102 will be omitted here.

[0074] The information processing device 100 includes a receiving device 116. The receiving device 116 is a keyboard and / or a mouse, etc., and receives instructions from a user of the information processing device 100, etc. The receiving device 116 is connected to the bus 112. The processor 106 acquires the instructions accepted by the receiving device 116 and operates in accordance with the acquired instructions.

[0075] The display device 118 displays various information including images. Examples of the display device 118 include a liquid crystal display and an EL display. The display device 118 is connected to the bus 112. The processor 106 causes the display device 118 to display the results obtained by executing various processes.

[0076] The external I / F 104 controls the exchange of various information between the processor 106 and one or more devices (hereinafter also referred to as "third external devices") existing outside the information processing device 100. A medical support device 24 is connected to the external I / F 104 as one of the third external devices. In the example shown in FIG. 5, an external I / F 80 of the medical support device 24 is connected to the external I / F 104. The external I / F 104 controls the exchange of various information between the processor 82 of the medical support device 24 (see FIGS. 3 and 4) and the processor 106 of the information processing device 100. For example, the information processing device 100 generates a lumen recognition model 92 and transmits the generated lumen recognition model 92 to the medical support device 80 via the external I / Fs 80 and 104 in response to a request from the medical support device 24.

[0077] The storage 110 stores a machine learning processing program 120. The processor 106 performs machine learning processing by reading the machine learning processing program 120 from the storage 110 and executing the read machine learning processing program 120 on the memory 108. The machine learning processing is realized by the processor 106 operating as a teacher data generation unit 106A and a learning execution unit 106B in accordance with the machine learning processing program 120 executed on the memory 108.

[0078] The storage 110 stores an example image set 122. As will be described in detail later, the example image set 122 is used by the training data generation unit 106A.

[0079] Fig. 6 is a conceptual diagram showing an example of the processing content of the teacher data generation unit 106A. As shown in Fig. 6, the information processing device 100 is used by an annotator 124. The annotator 124 refers to an operator who adds annotations for machine learning to given data (i.e., an operator who performs labeling).

[0080] 6, a keyboard 116A and a mouse 116B are shown as an example of the reception device 116. The annotator 124 gives instructions to the computer 102 via the keyboard 116A and the mouse 116B.

[0081] The example image set 122 includes a plurality of example images 122A each depicting different content. The example images 122A are images that have been determined in advance as medical images to be used in object recognition processing (e.g., processing in which the recognition unit 82A recognizes the lumen 42 based on the frame 40 and the lumen recognition model 92). The images that have been determined in advance as medical images to be used in object recognition processing are images that correspond to the frame 40. In other words, the image that corresponds to the frame 40 can also be said to be an image that simulates the frame 40. In further other words, the image that simulates the frame 40 can also be said to be an image that shows a sample of the frame 40. Here, a first example of an image that shows a sample of the frame 40 is an image obtained by actually capturing an image of the inside of the large intestine with a camera. A second example of an image that shows a sample of the frame 40 is a virtually created image (e.g., an image generated by a generation AI such as Stable Diffusion or Midjourney).

[0082] The teacher data generation unit 106A acquires an example image 122A from the example image set 122 in accordance with instructions received by the reception device 116. The teacher data generation unit 106A displays the example image 122A on a screen 118A of the display device 118. With the example image 122A displayed on the screen 118A, the annotator 124 instructs the teacher data generation unit 106A, via the reception device 116, the lumen corresponding position, which is the position within the example image 122A of the lumen shown in the example image 122A. The teacher data generation unit 106A generates teacher data 128 by associating the correct answer data 126 with the example image 122A based on the lumen corresponding position instructed via the reception device 116. Corresponding of the supervised answer data 126 to the example image 122A is realized by adding, as the supervised answer data 126, an annotation that can identify the lumen corresponding position to the lumen corresponding position in the example image 122A.

[0083] In this way, the teacher data generation unit 106A generates multiple teacher data 128 by repeatedly performing the process of associating correct answer data 126 with each of all example images 122A included in the example image set 122 in accordance with instructions given by the annotator 124.

[0084] Fig. 7 is a conceptual diagram showing an example of the configuration of example image 122A. As shown in Fig. 7, example image 122A shows the inside of large intestine 132. In the example shown in Fig. 7, example image 122A shows intestinal wall 136 with multiple folds 134 formed therein and lumen 138.

[0085] The sample image 122A is divided into a plurality of segmented regions 130A. The plurality of segmented regions 130A includes eight segmented regions 130A1 to 130A8. The segmented regions 130A1 to 130A8 are regions that exist radially from the center C1 of the sample image 122A toward the outer edge of the sample image 122A, and are arranged along the circumferential direction CD1 of the sample image 122A (in other words, around the center C1).

[0086] 8 and 9 are conceptual diagrams showing an example of a method in which the training data generating unit 106A generates training data 128 by associating the correct answer data 126 with the example image 122A.

[0087] As shown in FIG. 8 , with the example image 122A displayed on the screen 118A, the annotator 124 instructs the teacher data generation unit 106A, via the reception device 116, on a lumen-corresponding position 139, which is the position within the example image 122A of a lumen 138 appearing in the example image 122A. In accordance with the instruction received by the reception device 116, the teacher data generation unit 106A superimposes a circular frame 140 on the example image 122A and places the frame 140 at a position surrounding the lumen 138 appearing in the example image 122A. The frame 140 is a mark that defines the lumen-corresponding position 139 within the example image 122A. In other words, the position of the area surrounded by the frame 140 within the example image 122A is the lumen-corresponding position 139. The size and position of the frame 140 can be freely changed within the screen 118A in accordance with the instruction received by the reception device 116. Here, the shape of the frame 140 is circular, but it may be a shape other than circular. The size of the frame 140 can be changed according to an instruction received by the reception device 116.

[0088] With the frame 140 placed at a position surrounding the lumen 138, the annotator 124 issues a confirmation instruction, which is an instruction to confirm the lumen-corresponding position 139, to the teacher data generation unit 106A via the reception device 116. In response to this, the teacher data generation unit 106A confirms the lumen-corresponding position 139.

[0089] The teacher data generation unit 106A identifies, from the multiple segmented regions 130A, the segmented region 130A that has the largest overlapping area with the frame 140 that defines the lumen-corresponding position 139. Then, the teacher data generation unit 106A generates teacher data 128 by associating the correct answer data 126 with the identified segmented region 130A (segmented region 130A2 in the example shown in FIG. 8) as an annotation that can identify the segmented region 130A in which the lumen 138 is captured.

[0090] Figure 8 shows an example of a method for generating teacher data 128 when lumen 138 is captured in an area other than the central area in sample image 122A, while Figure 9 shows an example of a method for generating teacher data 128 when lumen 138 is captured in the central area in sample image 122A. As shown in Figure 9, when lumen 138 is captured in the central area in sample image 122A, teacher data generation unit 106A generates teacher data 128 by associating correct answer data 126 with each of all segmented areas 130A (i.e., segmented areas 130A1 to 130A8).

[0091] 10 is a conceptual diagram showing an example of an embodiment in which a lumen recognition model 92 is generated by a learning execution unit 106B performing machine learning using teacher data 128. As shown in FIG. 10, in the information processing device 100, the learning execution unit 106B acquires teacher data 128 generated by the teacher data generation unit 106A. Then, the learning execution unit 106B performs machine learning using the teacher data 128.

[0092] In the example shown in FIG. 10 , the learning execution unit 106B has a model 142. An example of the model 142 is a neural network. An example of the neural network is a convolutional neural network. The learning execution unit 106B inputs an example image 122A included in the training data 128 to the model 142. When the example image 122A is input, the model 142 performs inference and outputs an inference result 144. The learning execution unit 106B calculates an error 146 between the inference result 144 and the correct answer data 126 included in the training data 128.

[0093] The learning execution unit 106B calculates a plurality of adjustment values ​​148 that minimize the error 146. Then, the learning execution unit 106B optimizes the model 142 by adjusting a plurality of optimization variables in the model 142 using the plurality of adjustment values ​​148. Here, the plurality of optimization variables refers to, for example, a plurality of connection weights and a plurality of offset values ​​included in the model 142.

[0094] The learning execution unit 106B repeatedly performs a learning process using multiple teacher data 128, including inputting example images 122A into the model 142, calculating an error 146, calculating multiple adjustment values ​​148, and adjusting multiple optimization variables in the model 142. That is, the learning execution unit 106B optimizes the model 142 by adjusting multiple optimization variables in the model 142 using multiple adjustment values ​​148 calculated to minimize the error 146 for each of the multiple example images 122A included in the multiple teacher data 128. By optimizing the model 142 in this manner, a lumen recognition model 92 is generated. The lumen recognition model 92 is transmitted from the information processing device 100 to the medical support device 24 via the external I / Fs 80 and 104 (see FIG. 5) and received by the medical support device 24. Then, in the medical support device 24, the processor 82 stores the lumen recognition model 92 in the storage 86 (see FIG. 4). The lumen recognition model 92 stored in the storage 86 is used by the recognition unit 82A (see FIG. 4).

[0095] When the lumen recognition model 92 is actually used, a frame 40 is input to the lumen recognition model 92. Then, the lumen recognition model 92 recognizes the lumen 42 (see FIG. 1) that appears in the input frame 40. The recognition result is displayed on the screen 35. For example, the position where the lumen 42 appears in the frame 40 is visualized by displaying a mark or the like corresponding to one of eight regions in the frame 40 (i.e., eight regions corresponding to the segmented regions 130A1 to 130A8).

[0096] However, even if the area in which the lumen 42 is shown becomes visually identifiable by the display of a mark or the like, it may be difficult for the doctor 12 to visually grasp where the lumen 42 is shown within the area corresponding to the position where the mark or the like is displayed. For example, the smaller the display size of the lumen 42 is relative to the display size of the area in which the lumen 42 is shown, out of the eight areas in the frame 40, the more difficult it becomes to visually grasp where the lumen 42 is shown within the area corresponding to the position where the mark or the like is displayed.

[0097] In view of the above circumstances, in this embodiment, the medical support process is executed by the processor 82 of the medical support device 24.

[0098] Fig. 11 shows an example of the processing content of the recognition unit 82A. As shown in Fig. 11, an image 150 obtained by capturing an image of the intestinal wall 32 in the large intestine 28, including the lumen 42, using the camera 52 is acquired by the recognition unit 82A. The recognition unit 82A performs various processes on the image 150 to generate a frame 40. In the example shown in Fig. 11, the intestinal wall 32 having folds 43 and the lumen 42 are captured in the frame 40.

[0099] The recognition unit 82A executes a lumen recognition process 152 on the frame 40. The lumen recognition process 152 is a process for recognizing the lumen 42 captured in the frame 40 by using a lumen recognition model 92 stored in the storage 86 (in other words, a process for identifying the position within the frame 40 of the lumen 42 captured in the frame 40 by using the lumen recognition model 92). The recognition unit 82A acquires the frame 40 from the camera 52, and inputs the acquired frame 40 to the lumen recognition model 92, thereby causing the lumen recognition model 92 to generate certainty information 154.

[0100] 12 shows an example of the configuration of certainty information 154 generated by the lumen recognition model 92 when a lumen 42 is captured in the frame 40. As shown in FIG. 12, the certainty information 154 is information that includes a map 156 that corresponds to the frame 40. The size and shape of the map 156 are the same as those of the frame 40. However, this is merely an example, and the outer contour of the map 156 may be similar to the outer contour of the frame 40.

[0101] A confidence level 158 (e.g., the probability that a lumen 42 exists) is assigned to the map 156. Although the map 156 is illustrated here, a frame 40 may be used instead of the map 156. The map 156 has a plurality of segmented regions 160A corresponding to the plurality of segmented regions 130A (see FIGS. 7 to 9). Each of the plurality of segmented regions 160A is obtained by segmenting the map 156 along the circumferential direction CD2 (in other words, around the center C2 of the map 156). In the example shown in FIG. 12, segmented regions 160A1 to 160A8 are shown as an example of the plurality of segmented regions 160A. The segmented regions 160A1 to 160A8 are obtained by segmenting the map 156 at regular angular intervals (e.g., 45-degree intervals) along the circumferential direction CD2. In other words, the divided regions 160A1 to 160A8 can be said to be regions obtained by dividing the map 156 into eight regions radially from the center C2 of the map 156 toward the outer edge of the map 156.

[0102] In this embodiment, map 156 is an example of an "image corresponding to a medical image" according to the present disclosure. Also, in this embodiment, confidence level 158 is an example of a "confidence level" according to the present disclosure. Also, in this embodiment, circumferential direction CD2 is an example of a "circumferential direction" according to the present disclosure, and segmented regions 160A1 to 160A8 are an example of a "plurality of segmented regions" according to the present disclosure.

[0103] The map 156 is provided with a plurality of center lines CL. The center lines CL correspond to the plurality of sectional areas 160A and are arranged at equal intervals along the circumferential direction CD2. Each of the center lines CL is a virtual line extending from the center C2 in each sectional area 160A at an angle that is half the above-mentioned constant angle (for example, 22.5 degrees). In the example shown in FIG. 15, center lines CL1 to CL8 are provided as an example of the plurality of center lines CL for the sectional areas 160A1 to 160A8. The center lines CL1 to CL8 are arranged at 45-degree intervals around the center C2.

[0104] FIG. 13 shows an example in which multiple unit direction vectors 162 are assigned to the map 156. As shown in FIG. 16, the direction from the center C2 to the respective positions of the multiple segmented regions 160A is determined by the unit direction vector 162. The unit direction vector 162 is assigned to each of the multiple segmented regions 160A. The unit direction vector 162 is a unit vector indicating the direction from the center C2 to the respective positions of the segmented regions 160 (i.e., the respective positions of the segmented regions 160A1 to 160A8). In the example shown in FIG. 13, one unit direction vector 162 is assigned to each of the multiple segmented regions 160A (i.e., the segmented regions 160A1 to 160A8) along the center line CL of each segmented region 160A. In this embodiment, the center C2 is an example of a "reference position" according to the present disclosure, and the multiple unit direction vectors 162 are an example of a "multiple first vectors" according to the present disclosure.

[0105] 14 shows an example in which a plurality of directional vectors 164 are assigned to map 156. As shown in Fig. 14, control unit 82B acquires, from recognition unit 82A, certainty factor information 154 including map 156 to which a plurality of unit directional vectors 162 (see Fig. 13) are assigned. Control unit 82B then generates a plurality of directional vectors 164 based on the plurality of unit directional vectors 162 and a plurality of certainties 158 included in the certainty factor information 154.

[0106] Direction vector 164 is a vector whose magnitude is adjusted by weighting unit direction vector 162 with confidence 158 of partitioned area 160A to which unit direction vector 162 is assigned. The magnitude of direction vector 164 corresponds to the level of confidence 158, and the higher the confidence 158, the larger direction vector 164 becomes.

[0107] Here, a specific example of a method for generating the directional vector 164 will be described. For example, in a partitioned area 160A (partitioned area 160A1 in the example shown in FIG. 14) to which "0.3" has been assigned as the confidence factor 158, a vector obtained by increasing the magnitude of the unit directional vector 162 by 30% is generated as the directional vector (directional vector 164B in the example shown in FIG. 14). Also, in a partitioned area 160A (partitioned area 160A2 in the example shown in FIG. 14) to which "0.7" has been assigned as the confidence factor 158, a vector obtained by increasing the magnitude of the unit directional vector 162 by 70% is generated as the directional vector 164 (directional vector 164A in the example shown in FIG. 14). Also, in a partitioned area 160A (partitioned areas 160A3 to 160A8 in the example shown in FIG. 14) to which "0.0" has been assigned as the confidence factor 159, the magnitude of the directional vector 164 may be set to "zero," or the unit directional vector 162 may be used as the directional vector 164 as is.

[0108] It should be noted that the direction vector 164 illustrated here is merely an example, and a vector obtained by simply multiplying the unit direction vector 162 by the confidence factor 158 may also be used as the direction vector 164.

[0109] In the example shown in FIG. 14, directional vectors 164A and 164B are shown. The directional vector 164A is a vector obtained by adjusting the magnitude of the unit direction vector 162 assigned to the segmented region 160A2 by the confidence factor 158 of the segmented region 160A2 (0.7 in the example shown in FIG. 14). The directional vector 164B is a vector obtained by adjusting the magnitude of the unit direction vector 162 assigned to the segmented region 160A2 by the confidence factor 158 of the segmented region 160A1 (0.3 in the example shown in FIG. 14). The magnitude of the directional vector 164B represents the likelihood that a lumen 42 exists in the segmented region 160A to which the unit direction vector 162 on which the directional vector 164B is based is assigned. In other words, the larger the directional vector 164B, the higher the likelihood that a lumen 42 exists in the segmented region 160A to which the unit direction vector 162 on which the directional vector 164B is based is assigned.

[0110] Control unit 82B generates vector sum 166 based on the multiple directional vectors 164. Vector sum 166 is the sum of the multiple directional vectors 164. In the example shown in Fig. 14, vector sum 166 is the vector sum of directional vector 164A and directional vector 164B.

[0111] Within the map 156, the direction from the center C2 to the lumen presence region 168 is determined by the vector sum 166. The lumen presence region 168 refers to the region within the frame 40 in which the lumen 42 is captured. The segmented region 160A is a region whose position within the map 156 is constrained, whereas the lumen presence region 168 is a region whose position within the map 156 is not constrained like the segmented region 160A, and whose position changes depending on the position at which the vector sum 166 is generated. Furthermore, even if a lumen 42 exists within the segmented region 160A, it is difficult to estimate where within the segmented region 160A the lumen 42 exists. However, in the lumen presence region 168, the lumen 42 exists on a line along the vector sum 166, making it easy to estimate the location of the lumen 42. Therefore, the lumen existing region 168 identifies the location of the lumen 42 more precisely than when the location of the lumen 42 is identified in the segmented region 160A. In other words, the lumen existing region 168 can be said to be a region where the position where the lumen 42 is captured in the frame 40 (i.e., the location of the lumen 42 in the frame 40) can be identified with higher resolution along the circumferential direction CD2 than the multiple segmented regions 160A.

[0112] The control unit 82B identifies the lumen existence region 168 based on the vector sum 166. For example, the control unit 82B identifies a region of ±α degrees along the circumferential direction CD2 centered on a point other than the starting point of the vector sum 166 (for example, the end point) as the lumen existence region 168. An example of ±α degrees is ±22.5 degrees. Note that ±22.5 degrees is merely an example, and the range may be narrower or wider than ±22.5 degrees. Furthermore, α degrees may be a fixed value, or may be a variable value that is changed depending on an instruction accepted by the accepting device 64 or various conditions (for example, the type of operation mode of the endoscope system 10).

[0113] The control unit 82B generates a mark 169 that can identify the position of the lumen presence region 168 within the map 156, based on the multiple segmented regions 160A and the multiple confidence levels 158. The mark 169 is visible information that is determined based on a vector sum 166 that is generated based on the multiple segmented regions 160A and the multiple confidence levels 158. In the example shown in FIG. 14 , the shape of the mark 169 is an arc whose midpoint is the end point of the vector sum 166. The center of the arc that is the shape of the mark 169 is the center C2 of the map 156. The mark 169 indicates the range from one end to the other end of the lumen presence region 168 in the circumferential direction CD2.

[0114] In this embodiment, vector sum 166 is an example of a "second vector" according to the present disclosure. Also, in this embodiment, direction vector 164 is an example of a "third vector" according to the present disclosure. Also, in this embodiment, mark 169 is an example of "lumen-specific information" and a "mark" according to the present disclosure.

[0115] 15 shows an example of a form in which frame 40, etc. is displayed on screen 35 when a lumen 42 is captured in an area other than the central area of ​​frame 40. As shown in FIG. 15, control unit 82B acquires frame 40 input to lumen recognition model 92 from recognition unit 82A to obtain confidence information 154 including map 156 used to generate mark 169. Control unit 82B displays frame 40 acquired from recognition unit 82A in first display area 35A, and also displays mark 169 in first display area 35A so that it can be contrasted with frame 40. For example, mark 169 is displayed superimposed on frame 40.

[0116] Furthermore, control unit 82B updates mark 169 in accordance with the display timing of frame 40. For example, every time recognition unit 82A obtains certainty factor information 154, control unit 82B generates mark 169 based on certainty factor information 154 and displays it superimposed on frame 40. In this case, mark 169 displayed in first display area 35A is updated every time frame 40 is displayed. Note that mark 169 displayed in first display area 35A may be updated on the condition that frames 40 have been updated multiple times and displayed in first display area 35A (for example, a pre-specified number of frames 40 ranging from several to several hundred have been displayed in first display area 35A).

[0117] Additionally, the controller 82B displays visual information 44A in the second display region 35B as one piece of auxiliary information 44. An example of the visual information 44A is text that can identify the position of an area corresponding to the lumen existence region 168 in the frame 40, that is, text that can identify the position of the mark 169 displayed in the first display region 35A (for example, text that expresses an angle indicating the position of the vector sum 166 when the boundary line between the segmented regions 160A1 and 160A8 is set to 0 degrees). In this embodiment, the visual information 44A is an example of "lumen-identifying information" according to the present disclosure.

[0118] FIG. 16 shows an example of how the frame 40 and the like are displayed on the screen 35 when the lumen 42 is captured in the central region of the frame 40 (for example, when the center of the lumen 42 coincides with the center of the frame 40). As shown in FIG. 16, when the center of the lumen 42 coincides with the center of the frame 40, the control unit 82B displays the frame 40 in the first display region 35A and also superimposes a mark 170 on the frame 40. The mark 170 is a mark (for example, an annular mark) that surrounds the lumen 42 captured in the frame 40. When the direction vectors 164 of all the segmented regions 160A are equivalent, that is, when the vector sum 166 is zero, the control unit 82B generates the mark 170 and displays it superimposed on the frame 40. In this case, the control unit 82B also displays, as visible information 44A, information (for example, text) indicating that the lumen 42 is captured in the center of the frame 40 in the second display region 35B.

[0119] Note that, although an example in which the mark 170 is generated and superimposed on the frame 40 when the vector sum 166 is zero has been given here, this is merely one example. For example, the mark 170 may be generated and superimposed on the frame 40 when the magnitude of the vector sum 166 is less than a threshold (for example, the magnitude of the unit direction vector 162). Also, although a mark surrounding the lumen 42 shown in the frame 40 has been given here as an example of the mark 170, this is merely one example, and the mark may be one that can identify the position of the lumen 42 shown in the frame 40 (for example, a dot located at the center of the lumen 42 shown in the frame 40, or an arrow pointing to the position of the lumen 42).

[0120] Next, the operation of the information processing device 100 will be described with reference to FIG.

[0121] In the machine learning process shown in Figure 17, first, in step ST10, the teacher data generation unit 106A acquires an unprocessed example image 122A from the example image set 122 stored in the storage 110. Here, the unprocessed example image 122A refers to an example image 122A that has not yet been used in the machine learning process. The teacher data generation unit 106A displays the example image 122A acquired from the example image set 122 on the screen 118A. After the process of step ST10 is executed, the machine learning process proceeds to step ST12.

[0122] In step ST12, the teacher data generating unit 106A receives an instruction for the lumen corresponding position 139. After the processing of step ST12 is executed, the machine learning processing proceeds to step ST14.

[0123] In step ST14, the training data generating unit 106A identifies the positional relationship between the lumen corresponding position 139 received in step ST12 and the plurality of segmented regions 130A. After the process of step ST14 is executed, the machine learning process proceeds to step ST16.

[0124] In step ST16, the teacher data generation unit 106A associates the correct answer data 126 with the example image 122A acquired in step ST10 in accordance with the positional relationship determined in step ST14. For example, if the lumen-corresponding position 139 is located outside the central region of the example image 122A, the correct answer data 126 is associated with the segmented region 130A having the largest overlapping area with the lumen-corresponding position 139 in accordance with instructions given by the annotator 124. Also, for example, if the lumen-corresponding position 139 is located in the central region of the example image 122A, the correct answer data 126 is associated with each of all segmented regions 130A in accordance with instructions given by the annotator 124. In this manner, the teacher data generation unit 106A generates teacher data 128 by associating the correct answer data 126 with the example image 122A. The teacher data 128 generated in this manner is stored in a predetermined storage medium (e.g., the storage 110). After the process of step ST16 is executed, the machine learning process proceeds to step ST18.

[0125] In step ST18, the teacher data generation unit 106A determines whether or not there are any unprocessed example images 122A. If there are any unprocessed example images 122A in step ST18, the determination is negative, and the machine learning process proceeds to step ST10. If there are no unprocessed example images 122A in step ST18, the determination is positive, and the machine learning process proceeds to step ST20.

[0126] In step ST20, the learning execution unit 106B generates a lumen recognition model 92 by executing machine learning using a plurality of training data 128 obtained by repeatedly executing the processes of steps ST10 to ST18 (see FIG. 10). The lumen recognition model 92 is stored in the storage 86 of the medical support device 24 (see FIG. 4). After the process of step ST20 is executed, the machine learning process ends.

[0127] Next, the operation of the portion of the endoscope system 10 according to the present disclosure will be described with reference to Fig. 18. The flow of the medical support process shown in Fig. 18 is an example of a "medical support method" according to the present disclosure. For convenience, the following description will be given on the assumption that a lumen recognition model 92 is stored in the storage 86.

[0128] 22, in step ST50, the recognition unit 82A acquires an image 150 from the camera 52 and generates a frame 40 by performing various processes on the acquired image 150. After the processing of step ST50 is executed, the medical support processing proceeds to step ST52.

[0129] In step ST52, the recognition unit 82A generates certainty information 154 by executing lumen recognition processing 152 on the frame 40 generated in step ST50 using the lumen recognition model 92 stored in the storage 86. After the processing of step ST52 is executed, the medical support processing proceeds to step ST54.

[0130] In step ST54, the control unit 82B generates a plurality of direction vectors 164 by assigning the confidence 158 of the sectioned area 160A to which each unit direction vector 162 is assigned as a weight to each unit direction vector 162 of each sectioned area 160A of the map 156 included in the confidence information 154 generated in step ST52. After the processing of step ST54 is executed, the medical support processing proceeds to step ST56.

[0131] In step ST56, the control section 82B generates a vector sum 166, which is the sum of the plurality of direction vectors 164 generated in step ST54. After the processing of step ST56 is executed, the medical support processing proceeds to step ST58.

[0132] In step ST58, the control unit 82B identifies the lumen existing region 168 based on the vector sum 166 generated in step ST56. After the processing of step ST58 is executed, the medical support processing proceeds to step ST60.

[0133] In step ST60, the control unit 82B generates a mark 169 that can identify the position of the lumen existing region 168 identified in step ST58. After the processing of step ST60 is executed, the medical support processing proceeds to step ST62.

[0134] In step ST62, the control unit 82B displays the frame 40 generated in step ST50 in the first display area 35A. After the process of step ST62 is executed, the medical support process proceeds to step ST64.

[0135] In step ST64, the control unit 82B superimposes the mark 169 generated in step ST60 on the frame 40 displayed in the first display area 35A. After the processing of step ST64 is executed, the medical support processing proceeds to step ST66.

[0136] In step ST66, the control unit 82B determines whether a condition for terminating the medical support process is satisfied. An example of the condition for terminating the medical support process is a condition that an instruction to terminate the medical support process is given to the endoscope system 10 (for example, a condition that an instruction to terminate the medical support process is accepted by the acceptance device 64).

[0137] In step ST66, if the condition for terminating the medical support process is not satisfied, the determination is negative and the medical support process proceeds to step ST50. In step ST66, if the condition for terminating the medical support process is satisfied, the determination is positive and the medical support process ends.

[0138] As described above, in the endoscope system 10, a frame 40 showing the intestinal wall 32 and the lumen 42 is input to the lumen recognition model 92, and the lumen recognition model 92 generates certainty information 154. The certainty information 154 includes a map 156 divided into a plurality of divided regions 160A. Each of the plurality of divided regions 160A is assigned a certainty 158 that a lumen 42 exists.

[0139] In the endoscope system 10, a lumen presence region 168 is generated based on a plurality of segmented regions 160A and a plurality of certainty factors 158. The lumen presence region 168 is a region in the frame 40 in which the presence position of the lumen 42 (i.e., the position where the lumen 42 is captured) is specified more precisely than in the segmented region 160A. Because the lumen presence region 168 is generated based on a plurality of segmented regions 160A and a plurality of certainty factors 158, its position is not fixed within the map 156, as is the case with the segmented region 160A. Furthermore, the position of the lumen presence region 168 within the map 156 changes finely along the circumferential direction CD2 depending on the presence position of the lumen 42. This means that the lumen presence region 168 is a region in which the presence position of the lumen 42 is defined with higher resolution along the circumferential direction CD2 than in the plurality of segmented regions 160A.

[0140] In the endoscope system 10, a mark 169 is generated as information that can identify a lumen existence region 168. Then, in the first display region 35A of the screen 35, a frame 40 that was input to the lumen recognition model 92 to generate the certainty information 154 is displayed. The mark 169 is also superimposed on the frame 40. This means that the mark 169 is expressed on the frame 40 with higher resolution than when the certainty 158 of each of the multiple segmented regions 160A is simply superimposed on the frame 40 or when visible information (for example, a mark) that simply indicates the level of the certainty 158 is displayed in the first display region 35A.

[0141] Therefore, by visually checking the mark 169 superimposed on the frame 40, the doctor 12 can accurately grasp the position of the lumen 42 shown in the frame 40 within the frame 40, compared to when the confidence level 158 of each of the multiple division areas 160A is simply superimposed on the frame 40, or when visible information indicating the level of the confidence level 158 is simply displayed in the first display area 35A.

[0142] Furthermore, in the endoscope system 10, the mark 169 is displayed so as to be contrasted with the frame 40. That is, the mark 169 is displayed superimposed on the frame 40. Therefore, the doctor 12 can visually grasp the positional relationship between the frame 40 and the mark 169.

[0143] Furthermore, in the endoscope system 10, the direction from the center C2 of the map 156 to the respective positions of the plurality of segmented regions 160A is determined by a plurality of unit direction vectors 162. Furthermore, the direction from the center C2 of the map 156 to a lumen existence region 168 is determined by a vector sum 166. The vector sum 166 is the sum of a plurality of direction vectors 164 obtained by assigning confidence factors 158 as weights to the plurality of unit direction vectors 162. A mark 169 superimposed and displayed on the frame 40 is generated based on the vector sum 166.

[0144] Here, the multiple directional vectors 164 change according to the confidence level 158 assigned to each of the multiple segmented regions 160A. The vector sum 166 changes according to the multiple directional vectors 164. It can be said that the vector sum 166 is a vector indicating the direction from the center C2 of the map 156 to the location of the lumen 42. This means that, compared to when the confidence levels 158 of each of the multiple segmented regions 160A are simply superimposed on the frame 40 or when visual information indicating the level of the confidence levels 158 is simply displayed in the first display region 35A, the mark 169 generated based on the vector sum 166 is visual information that precisely represents the location within the frame 40 of the lumen 42 shown in the frame 40.

[0145] Therefore, by visually checking the mark 169 superimposed on the frame 40, the doctor 12 can accurately grasp the position of the lumen 42 shown in the frame 40 within the frame 40, compared to when the confidence level 158 of each of the multiple division areas 160A is simply superimposed on the frame 40, or when visible information indicating the level of the confidence level 158 is simply displayed in the first display area 35A.

[0146] Furthermore, in the endoscope system 10, the marks 169 and / or 170 displayed in the first display area 35A are updated in accordance with the display timing of the frame 40. Furthermore, the visual information 44A displayed in the second display area 35B is also updated in accordance with the display timing of the frame 40. Therefore, the doctor 12 can visually recognize the marks 169 and the visual information 44A that correspond to the content of the frame 40 displayed in the first display area 35A (i.e., the marks 169 and the visual information 44A that can identify the position within the frame 40 of the lumen 42 shown in the frame 40 displayed in the first display area 35A).

[0147] Furthermore, in the endoscope system 10, the shape of the mark 169 is an arc, and the center of the arc is the center of the frame 40. The mark 169 is not superimposed on the entire frame 40, but is only superimposed on a portion of the frame 40. By superimposing and displaying such a mark 169 on the frame 40, the visibility of the frame 40 displayed in the first display area 35A is ensured, and the doctor 12 can accurately grasp the position within the frame 40 of the lumen 42 shown in the frame 40.

[0148] In the above embodiment, the mark 169 has been exemplified as having an arc shape, but this is merely an example and the mark may have another shape.

[0149] 19 , instead of mark 169, mark 169A having a shape that follows the outer edge of frame 40 may be generated and displayed in first display area 35A. One example of a method for generating mark 169A is a method in which mark 169 is projected from the center C2 side onto the outer edge of map 156, thereby generating mark 169A having a shape that follows the outer edge of map 156. Frame 40 input to lumen recognition model 92 for generating certainty information 154 including map 156 is displayed in first display area 35A, and mark 169A is displayed in synchronization with the display timing of frame 40.

[0150] Here, since mark 169 is formed inside map 156, an example of a method for generating mark 169A has been given in which mark 169 is projected onto the outer edge of map 156 from the center C2 side. However, if mark 169 is formed outside map 156 (for example, if mark 169 is formed below the upper end of map 156 or below the lower end of map 156), mark 169 may be projected onto the outer edge of map 156 from the outside of map 156 toward the center C2 side.

[0151] In this way, when a mark shaped along the outer edge of the frame 40 is displayed in the first display area 35A, the display of objects that visually obstruct the frame 40 in the first display area 35A is suppressed, thereby ensuring the visibility of the frame 40 displayed in the first display area 35A while allowing the doctor 12 to accurately grasp the position of the lumen 42 shown in the frame 40 within the frame 40.

[0152] Furthermore, in the same manner as the mark 169 is updated in accordance with the display timing of the frame 40, the mark 169A may also be updated in accordance with the display timing of the frame 40. In this way, the doctor 12 can visually recognize the mark 169A that corresponds to the content of the frame 40 displayed in the first display area 35A.

[0153] In the above embodiment, an example has been given in which mark 169 is displayed superimposed on frame 40, but this is merely one example, and mark 169 may be displayed outside frame 40. If mark 169 is displayed outside frame 40, there is no object that visually obstructs frame 40 displayed in first display area 35A, and therefore the visibility of frame 40 displayed in first display area 35A can be improved.

[0154] In the above embodiment, an example has been given in which the control unit 82B generates an arc-shaped mark 169 that can identify the range from one end to the other end of the lumen presence region 168 in the circumferential direction CD2 and displays the generated mark 169 in the first display region 35A, but this is merely one example. For example, as shown in Fig. 20 , the control unit 82B may display the mark 169 in the first display region 35A by displaying in the first display region 35A at least one marker 171 that can identify the range from one end to the other end of the lumen presence region 168 in the circumferential direction CD2, out of multiple non-display markers 171 associated with the map 156. For example, the number of the markers 171 (in other words, the number of divisions) may be "40" (=8×5) obtained by dividing each of the eight segmented regions 160A (see FIGS. 12 to 14) into five equal parts, or "80" (=8×10) obtained by dividing each of the eight segmented regions 160A into ten equal parts. The number of the markers 171 may also be a number other than these. The greater the number of markers 171, the higher the resolution of the display of the marks 169. In other words, the greater the number of markers 171, the more precisely the control unit 82B can generate and display the marks 169.

[0155] In the example shown in FIG. 20, as an example of the hidden markers 171, a plurality of arc-shaped markers are shown arranged at regular intervals along a circle whose center coincides with the center C2 of the map 156.

[0156] In the example shown in Figure 20, the display of the mark 169 in the first display area 35A is achieved by displaying at least one marker 171 that can identify the range from one end to the other end of the lumen existence area 168 in the circumferential direction CD2.

[0157] 20, the mark 169 is superimposed on the frame 40 displayed in the first display region 35A, but depending on the position of the lumen existence region 168, the mark 169 may also be displayed outside the frame 40. For example, by displaying at least one marker 171 at a position outside the map 156 (in the example shown in FIG. 20, above the upper end of the map 156 in a front view and below the lower end of the map 156 in a front view), the mark 169 is displayed outside the frame 40 displayed in the first display region 35A.

[0158] 20 shows, as an example of the plurality of hidden markers 171, a plurality of arc-shaped markers arranged at regular intervals along a circle whose center coincides with the center C2 of the map 156. However, this is merely an example. For example, the plurality of hidden markers 171 may be a plurality of markers arranged at regular intervals along the outer edge of the map 156.

[0159] 20 shows a plurality of arc-shaped markers arranged at regular intervals along a circle whose center coincides with the center C2 of the map 156 and whose part overlaps the map 156, but this is merely an example. For example, a plurality of arc-shaped markers may be arranged at regular intervals along a circle whose center coincides with the center C2 of the map 156 and that surrounds the map 156. In this case, the mark 169 is displayed outside the frame 40 by displaying at least one marker 171 that can identify the range from one end to the other end of the lumen presence region 168 in the circumferential direction CD2.

[0160] In this way, the display of mark 169 is realized by displaying at least one marker 171 at a position corresponding to the position of lumen existence region 168 among multiple hidden markers 171, thereby reducing the processing load required to display mark 169.

[0161] In the example shown in Fig. 20, an example form in which the mark 169 is displayed has been given, but this is merely one example. For example, as shown in Fig. 21, instead of the mark 169, an outer contour line 172 may be displayed in the first display region 35A. The outer contour line 172 is a line that outlines the outer contour of the lumen existence region 168. In the example shown in Fig. 21, the outer contour line 172 is superimposed on the frame 40 displayed in the first display region 35A. By superimposing the outer contour line 172 on the frame 40 in this way, the doctor 12 can visually recognize that the lumen 42 is included in the outer contour line 172.

[0162] 21, a line segment 173 along the vector sum 166 (for example, a line segment extending from the center C2 along the vector sum 166) may be superimposed and displayed on the frame 40. In this case, the doctor 12 can visually recognize that the lumen 42 is shown on the line segment 173 superimposed and displayed on the frame 40.

[0163] In the above embodiment, eight segmented regions 160A are illustrated, but the number of segmented regions 160A may be less than eight, or may be nine or more. The number of segmented regions 130A may also be determined to match the number of segmented regions 160A.

[0164] In the above embodiment, an example in which the frame 40 is displayed on the screen 35 and the mark 169 is displayed so as to be contrasted with the frame 40 (for example, an example in which the mark 169 is superimposed on the frame 40) has been given, but this is merely one example. For example, the map 156 may be displayed on the screen 35 and the mark 169 may be displayed so as to be contrasted with the map 156. One example of a contrastable display is superimposing the mark 169 on the map 156. Note that the map 156 is an example of an "image corresponding to a medical image" according to the present disclosure.

[0165] In the above embodiment, an example in which the visual information 44A is displayed in the second display area 35B has been described, but this is merely one example. For example, audible information (e.g., an electronic sound or speech sound) that can identify the position of the lumen 42 shown in the frame 40 within the frame 40 may be output from a speaker (not shown). Furthermore, information in which the frame 40 and the mark 169, etc. are combined, and / or the visual information 44A may be printed on a medium by a printer. Furthermore, information in which the frame 40 and the mark 169, etc. are combined, the visual information 44A, and / or the above-mentioned audible information may be stored in a storage medium (e.g., storage 76, storage 86, or storage provided in an external device such as a server).

[0166] In the above embodiment, an example in which the medical support processing is performed by the computer 78 has been described, but the present disclosure is not limited to this, and at least a part of the processing included in the medical support processing may be performed by a device provided outside the computer 78. An example of this case will be described below with reference to FIG.

[0167] Fig. 22 is a conceptual diagram showing an example of the configuration of an endoscope system 174. In the example shown in Fig. 22, the endoscope system 174 is an example of the "endoscope system" according to the present disclosure. The endoscope system 174 differs from the endoscope system 10 described in the above embodiment in that it includes an external device 176.

[0168] For example, the external device 176 is a server, and is communicatively connected to the computer 78 via a network 178 (for example, a WAN and / or a LAN, etc.). Although a server is exemplified here, at least one personal computer or the like may be used as the external device 176 instead of a server.

[0169] An example of the external device 176 is at least one server that directly or indirectly transmits and receives data to and from the computer 78 via the network 178. The external device 176 receives a processing execution instruction provided via the network 178 from the processor 82 of the computer 78. The external device 176 then executes processing in accordance with the received processing execution instruction and transmits the processing result to the computer 78 via the network 178. In the computer 78, the processor 82 receives the processing result transmitted from the external device 176 via the network 178 and executes processing using the received processing result.

[0170] An example of the processing execution instruction is an instruction to cause the external device 176 to execute at least a part of the medical support processing. A first example of at least a part of the medical support processing (i.e., a process to be executed by the external device 176) is the lumen recognition processing 152. In this case, the external device 176 executes the lumen recognition processing 152 in accordance with the processing execution instruction provided from the processor 82 via the network 178, and transmits information including the certainty factor information 154 as a first processing result to the computer 78 via the network 178. In the computer 78, the processor 82 receives the first processing result and executes processing similar to that of the above embodiment using the received first processing result.

[0171] A second example of at least a part of the medical support processing (i.e., processing to be executed by the external device 176) is processing by the control unit 82B. In this case, the external device 176 executes the processing by the control unit 82B in accordance with a processing execution instruction provided from the processor 82 via the network 178, and transmits a second processing result (e.g., the mark 169 and / or the visible information 44A, etc.) to the computer 78 via the network 178. In the computer 78, the processor 82 receives the second processing result and executes processing similar to that of the above embodiment (e.g., display using the display device 18, etc.) using the received second processing result.

[0172] The external device 176 may be realized by cloud computing. Cloud computing is merely an example, and the external device 176 may be realized by network computing such as fog computing, edge computing, or grid computing.

[0173] In the above embodiment, an example has been described in which the medical support program 90 is stored in the storage 86, but the present disclosure is not limited to this. For example, the medical support program 90 may be stored in a portable, computer-readable, non-transitory storage medium such as an SSD or a USB flash drive. The medical support program 90 stored in the non-transitory storage medium is installed in the computer 78 of the endoscope system 10. The processor 82 executes medical support processing in accordance with the medical support program 90.

[0174] Alternatively, the medical support program 90 may be stored in a storage device such as another computer or server connected to the endoscopic system 10 via a network, and the medical support program 90 may be downloaded and installed on the computer 78 in response to a request from the endoscopic system 10.

[0175] It is not necessary to store the entire medical support program 90 in a storage device such as another computer or server device connected to the endoscopic system 10, or to store the entire medical support program 90 in the storage 86; only a portion of the medical support program 90 may be stored.

[0176] The hardware resources that execute the medical support processing can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes medical support processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations specifically designed to execute specific processes. Each processor has built-in or connected memory, and executes medical support processing by using the memory.

[0177] The hardware resource that executes the medical support processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the medical support processing may be a single processor.

[0178] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute medical support processing, on a single IC chip, as typified by SoCs. In this way, medical support processing is realized using one or more of the above-mentioned various processors as hardware resources.

[0179] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0180] The above-described description and illustrations are a detailed explanation of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or elements may be replaced with other parts from the above-described description and illustrations, as long as they do not deviate from the gist of the present disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the parts related to the present disclosure, the above-described description and illustrations omit explanations of common general technical knowledge that do not require particular explanation to enable the implementation of the present disclosure.

[0181] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0182] 10,174 Endoscopy Systems 12 Doctors 14 Staff 16 Endoscope 18,118 Display device 20 Light source device 22 Control device 24 Medical support equipment 26 Subject 28 Large intestine 30 light 32,132 intestinal wall 34 Wagon 35,118A screen 35A 1st display area 35B 2nd display area 39 Endoscopic video images 40 frames 42,138 lumen 43,134 folds 44 Supplementary Information 44A Visible Information 46 Control section 48 Insertion section 50 Tip 50A tip surface 52 Camera 54 Lighting equipment 54A, 54B Lighting window 56 Treatment opening 58 Treatment tools 60 Treatment tool insertion port 62 Universal Code 64,116 Reception equipment 66,78,102 Computer 68, 88, 112 buses 70, 80, 104 External I / F 72, 82, 106 processors 74, 84, 108 memory 76,86,110 Storage 82A recognition part 82B Control section 90 Medical Assistance Program 92 Lumen Recognition Model 100 Information processing device 106A Teacher data generation unit 106B Learning Execution Department 116A Keyboard 116B Mouse 120 Machine Learning Processing Program 122 example image sets 122A Example image 124 Annotators 126 correct data 128 training data 130A,130A1~130A8,160A,160A1~160A8 Segmented area 139 Lumen Corresponding Position 140 slots 142 models 144 Inference results 146 error 148 Adjustment Value 150 images 152 Lumen recognition processing 154 Confidence Information 156 Maps 158 Confidence 162 Unit Direction Vector 164,164A,164B direction vector 166 Vector Sum 168 Luminal area 169, 169A, 170 marks 171 Marker 172 Outer Contour 173 line segments 176 External device 178 Network C1,C2 center CD1,CD2 circumferential direction CL center line CL1~CL8 Center line

Claims

1. a processor; The processor: By inputting a medical image generated by imaging the inside of a hollow organ including the lumen into the trained model, a plurality of certainty factors corresponding to a plurality of segmented regions obtained by dividing the medical image or an image corresponding to the medical image along a circumferential direction are generated, and a plurality of certainty factors that the lumen is captured in the plurality of segmented regions are generated; Outputting lumen identification information that can identify a lumen existence region in which the lumen existence position is identified more precisely than the segmented region in the medical image or an image corresponding to the medical image, based on the plurality of segmented regions and the plurality of certainty factors. Medical support equipment.

2. The lumen existing region is a region where the position where the lumen is captured in the medical image can be identified with a higher resolution along the circumferential direction than the plurality of segmented regions. The medical support device according to claim 1 .

3. directions from a reference position of the medical image or an image corresponding to the medical image to positions where the plurality of segmented regions exist are determined by a plurality of first vectors; a direction from the reference position to the lumen presence region is determined by a second vector; the second vector is a sum of at least two third vectors obtained by assigning the confidence levels as weights to at least two first vectors among the plurality of first vectors, The lumen-specific information is information determined based on the second vector. The medical support device according to claim 1 .

4. The processor outputs the medical image or an image corresponding to the medical image; The lumen-specific information is updated in accordance with the output timing of the medical image or an image corresponding to the medical image. The medical support device according to claim 1 .

5. The output of the lumen-specific information is realized by displaying the lumen-specific information on a screen. The medical support device according to claim 1 .

6. the medical image or an image corresponding to the medical image is displayed on the screen; The lumen-specific information displayed on the screen is updated in accordance with the display timing of the medical image or an image corresponding to the medical image. The medical support device according to claim 5.

7. The medical image and / or an image corresponding to the medical image is displayed on the screen so that the lumen-specifying information can be compared with the medical image. The medical support device according to claim 5.

8. The lumen-specific information is superimposed on the medical image and / or an image corresponding to the medical image. The medical support device according to claim 7.

9. directions from a reference position of the medical image or an image corresponding to the medical image to positions where the plurality of segmented regions exist are determined by a plurality of first vectors; a direction from the reference position to the lumen presence region is determined by a second vector; the second vector is a sum of at least two third vectors obtained by assigning the confidence levels as weights to at least two first vectors among the plurality of first vectors, The lumen specifying information includes a mark that can specify a region defined as the lumen existing region based on the second vector within the medical image or an image corresponding to the medical image. The medical support device according to claim 1 .

10. the mark has an arc shape; The center of the arc is the center of the medical image or an image corresponding to the medical image. The medical support device according to claim 9.

11. The shape of the mark is a shape that follows the outer edge of the medical image or the outer edge of an image corresponding to the medical image. The medical support device according to claim 9.

12. a plurality of non-display markers are associated with the medical image or an image corresponding to the medical image; the medical image or an image corresponding to the medical image is displayed on a screen; The mark is displayed on the screen by displaying at least one marker corresponding to the position of the lumen existing region among the plurality of markers. The medical support device according to claim 9.

13. A medical support device according to any one of claims 1 to 12; an endoscope, The medical image is generated by imaging the inside of the hollow organ, including the lumen, with the endoscope. Endoscopy system.

14. By inputting a medical image generated by imaging the inside of a hollow organ including the lumen into the trained model, a plurality of certainty factors corresponding to a plurality of segmented regions obtained by dividing the medical image or an image corresponding to the medical image along a circumferential direction are generated, and a plurality of certainty factors that the lumen is captured in the plurality of segmented regions are generated; and and outputting lumen identification information capable of identifying a lumen existing region in which the location of the lumen is identified more precisely than the segmented regions within the medical image or an image corresponding to the medical image, based on the plurality of segmented regions and the plurality of certainty factors. Medical support methods.

15. By inputting a medical image generated by imaging the inside of a hollow organ including the lumen into the trained model, a plurality of certainty factors corresponding to a plurality of segmented regions obtained by dividing the medical image or an image corresponding to the medical image along a circumferential direction are generated, and a plurality of certainty factors that the lumen is captured in the plurality of segmented regions are generated; and A program for causing a computer to execute a process including outputting lumen identification information capable of identifying a lumen existence area in which the location of the lumen is identified more precisely than in the divided area within the medical image or an image corresponding to the medical image, based on the multiple divided areas and the multiple certainty levels.

Images