MEDICAL SUPPORT DEVICE, ENDOSCOPE SYSTEM, MEDICAL SUPPORT PROCEDURE AND PROGRAM

Abstract

A medical support device, an endoscope system, a medical support procedure, and a program are provided that enable a user or the like to determine the position of a lumen shown in a medical image with high accuracy. A medical support device contains a processor. The process is configured to input a medical image, generated by imaging the interior of a lumen-containing organ, into a trained model to generate multiple confidence levels corresponding to several subdivided regions obtained by subdividing the medical image, or an image corresponding to the medical image, along a circumferential direction, and indicating that the lumen is shown in the multiple subdivided regions.Furthermore, the processor is configured to output lumen specification information to specify a lumen existence area in which an existence position of the lumen is specified with higher accuracy than in the subdivided areas, in the medical image or the image corresponding to the medical image, based on the multiple subdivided areas and the multiple confidence levels.

Description

BACKGROUND OF THE INVENTION 1. Field of the invention

[0001] The present disclosure relates to a medical support device, an endoscope system, a medical support procedure and a program. 2. Description of the state of the art

[0002] JP2003-093328A discloses an endoscope insertion direction detection method comprising a first step of inputting an endoscopic image, a second step of detecting a direction of brightness variation in the endoscopic image, and a third step of generating information about an insertion direction of an endoscope based on a result of the detection.Furthermore, JP2003-093328A also discloses an endoscope insertion direction detection method comprising a first step of setting insertion candidate directions that are candidates for an insertion direction of an endoscope, a second step of inputting an endoscopic image, a third step of detecting a direction of brightness variation in the endoscopic image, a fourth step of evaluating a similarity between several insertion candidate directions and the direction of brightness variation, and a fifth step of determining the insertion direction of the endoscope based on a result of the evaluation.

[0003] WO2020 / 194472A discloses a motion support system comprising a multi-operator information calculation unit that calculates multi-operator information specifying multiple operations that differ in time and corresponding to a multi-operator target scene, which is a scene requiring multiple operations that differ in time, based on a captured image acquired by an imaging unit arranged in an insertion part, and a presentation information generation unit that generates presentation information for the insertion part based on the multi-operator information calculated by the multi-operator information calculation unit. SUMMARY OF THE INVENTION

[0004] An embodiment according to the present disclosure provides a medical support device, an endoscope system, a medical support method and a program that enables a user or the like to determine the position of a lumen shown in a medical image with high accuracy.

[0005] A first aspect according to the present disclosure relates to a medical support device comprising: a processor, wherein the processor is configured to: input a medical image, generated by imaging the interior of a lumen organ containing a lumen, into a trained model to generate multiple confidence levels corresponding to multiple subdivided areas obtained by subdividing the medical image, or an image corresponding to the medical image, along a circumferential direction, and indicating that the lumen is shown in the multiple subdivided areas;and lumen specification information to specify a lumen existence area in which an existence position of the lumen is specified with higher accuracy than in the subdivided areas, in the medical image or the image corresponding to the medical image, based on the multiple subdivided areas and the multiple confidence levels.

[0006] A second aspect according to the present disclosure relates to the medical support device according to the first aspect, wherein the lumen existence area is an area in which a position where the lumen is shown in the medical image can be specified along the circumferential direction with a higher resolution than in the several subdivided areas.

[0007] A third aspect according to the present disclosure relates to the medical support device according to the first or second aspect, wherein a direction from a reference position of the medical image or the image corresponding to the medical image to an existence position of each of the several subdivided areas is determined by several first vectors, a direction from the reference position to the lumen existence area is determined by a second vector, the second vector being a sum of at least two third vectors obtained by adding the confidence level as a weight to at least two first vectors among the several first vectors, and the lumen specification information being information determined on the basis of the second vector.

[0008] A fourth aspect according to the present disclosure relates to the medical support device according to any of the first to third aspects, wherein the processor is configured to output the medical image or the image corresponding to the medical image and the lumen specification information is updated in accordance with an output time of the medical image or the image corresponding to the medical image.

[0009] A fifth aspect according to the present disclosure relates to the medical support device according to one of the first to fourth aspects, wherein the output of the lumen specification information is implemented by displaying the lumen specification information on a screen.

[0010] A sixth aspect according to the present disclosure relates to the medical support device according to the fifth aspect, wherein the medical image or the image corresponding to the medical image is displayed on the screen and the lumen specification information displayed on the screen is updated in accordance with a display time of the medical image or the image corresponding to the medical image.

[0011] A seventh aspect according to the present disclosure relates to the medical support device according to the fifth aspect or the sixth aspect, wherein the medical image and / or the image corresponding to the medical image and the lumen specification information are displayed comparably on the screen.

[0012] An eighth aspect according to the present disclosure relates to the medical support device according to the seventh aspect, wherein the lumen specification information is displayed superimposed on the medical image and / or the image corresponding to the medical image.

[0013] A ninth aspect according to the present disclosure relates to the medical support device according to any of the first to eighth aspects, wherein a direction from a reference position of the medical image or the image corresponding to the medical image to an existence position of each of the several subdivided areas is determined by several first vectors, a direction from the reference position to the lumen existence area is determined by a second vector, the second vector being a sum of at least two third vectors obtained by adding the confidence level as a weight to at least two first vectors among the several first vectors, and the lumen specification information including a marker for specifying an area determined on the basis of the second vector in the medical image or the image corresponding to the medical image as the lumen existence area.

[0014] A tenth aspect according to the present disclosure relates to the medical support device according to the ninth aspect, wherein a form of marking is an arc and a center of the arc is a center of the medical image or of the image corresponding to the medical image.

[0015] An eleventh aspect according to the present disclosure relates to the medical support device according to the ninth aspect, wherein a form of marking is a form along an outer edge of the medical image or an outer edge of the image corresponding to the medical image.

[0016] A twelfth aspect according to the present disclosure relates to the medical support device according to one of the ninth to eleventh aspects, wherein several markings which are hidden are associated with the medical image or the image corresponding to the medical image, and the marking is indicated on the screen by displaying at least one marking which corresponds to a position of the lumen existence area among the several markings.

[0017] A thirteenth aspect according to the present disclosure relates to an endoscope system comprising: the medical support device according to any one of the first to twelfth aspects; and an endoscope, wherein the medical image is produced by imaging the interior of the lumen organ containing the lumen with the endoscope.

[0018] A fourteenth aspect according to the present disclosure relates to a medical support procedure comprising: inputting a medical image, generated by mapping the interior of a lumen organ containing a lumen, into a trained model to generate multiple confidence levels corresponding to multiple subdivided regions obtained by subdividing the medical image, or an image corresponding to the medical image, along a circumferential direction, and indicating that the lumen is shown in the multiple subdivided regions; and outputting lumen specification information to specify a lumen existence region, specifying an existence position of the lumen with higher accuracy than in the subdivided regions, in the medical image, or the image corresponding to the medical image, based on the multiple subdivided regions and the multiple confidence levels.

[0019] A fifteenth aspect according to the present disclosure relates to a program that causes a computer to perform a process comprising: inputting a medical image, generated by mapping the interior of a lumen organ containing a lumen, into a trained model to generate multiple confidence levels corresponding to multiple subdivided areas obtained by subdividing the medical image, or an image corresponding to the medical image, along a circumferential direction, and indicating that the lumen is shown in the multiple subdivided areas;and output of lumen specification information to specify a lumen existence area, specifying an existence position of the lumen with higher accuracy than in the subdivided areas, in the medical image or the image corresponding to the medical image, based on the multiple subdivided areas and the multiple confidence levels. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a concept diagram showing a view example of an endoscope system being used by a doctor. Fig. Figure 2 is a concept diagram showing an example of a complete configuration of the endoscope system. Fig. Figure 3 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope system. Fig. Figure 4 is a block diagram showing an example of the main functions of a processor included in a medical support device and an example of information stored in a data storage device. Fig. Figure 5 is a block diagram showing an example of a hardware configuration of an electrical system of an information processing device. Fig. Figure 6 is a concept diagram showing a view example where training data is generated by the information processing device. Fig. Figure 7 is a concept diagram showing an example of a sample image. Fig. Figure 8 is a concept diagram that shows an example of training data generated in a case where a lumen is located in an area outside a central region of the Fig. The example image shown in 7 shows. Fig. Figure 9 is a concept diagram that shows an example of training data generated in a case where the lumen is in the midrange of the Fig. The example image shown in 7 shows. Fig. Figure 10 is a concept diagram showing an example of processing content in the information processing device in a case where a lumen detection model is created by training a model through machine learning using training data. Fig. Figure 11 is a concept diagram showing an example of processing content of a recognition unit of the medical support device. Fig. Figure 12 is a concept diagram showing an example of confidence-level information generated by the lumen detection model in a case where the lumen is shown in a single image. Fig. Figure 13 is a concept diagram showing an example of a relationship between several subdivided areas obtained by radial subdivision of a map contained in the confidence level information and several direction unit vectors added to the several subdivided areas. Fig. Figure 14 is a concept diagram showing an example of processing content of a control unit for the medical support device. Fig. Figure 15 is a concept diagram showing a view example in which, in a case where the lumen is shown in an area outside a central area of ​​the single image, the single image is displayed in a first display area of ​​a screen, a marker to specify a position of a lumen existence area is shown, in which an existence position of the lumen shown in the single image is specified, in which the single image is displayed superimposed on the single image, and visible information is shown in a second display area of ​​the screen as part of auxiliary information. Fig. Figure 16 is a concept diagram showing a view example in which, in a case where the lumen is shown in the central area of ​​the single image, the single image is displayed in the first display area of ​​the screen, the marker to specify the position of the lumen existence area, in which the existence position of the lumen shown in the single image is specified, is displayed superimposed on the single image, and the visible information is displayed in the second display area of ​​the screen as part of the auxiliary information. Fig. Figure 17 is a flowchart showing an example of a processing flow for machine learning. Fig. Figure 18 is a flowchart showing an example of a medical support processing flow. Fig. Figure 19 is a concept diagram showing a form example in which a marker is displayed along an outer edge of the single image shown on a screen. Fig. Figure 20 is a concept diagram showing a form example in which the marking is displayed superimposed on the single image by displaying at least one label corresponding to the position of the lumen existence area under several hidden labels associated with the map and / or the single image. Fig. Figure 21 is a concept diagram showing a form example where an outer contour line, which is a line outlining an outer contour of the lumen existence area, is displayed superimposed on the single image. Fig. Figure 22 is a concept diagram illustrating an example of a series of processing operations in which a processor contained within a computer issues a processing execution request to an external device via a network, the external device performs processing in response to the processing execution request, and the processor contained within the computer receives a processing result from the external device. DESCRIPTION OF PREFERRED EXECUTION FORMS

[0020] Examples of embodiments of a medical support device, an endoscope system, a medical support method, and a program according to the present disclosure are described below with reference to the accompanying drawings. It should be noted that the present disclosure can also be applied to a program and a computer program product.

[0021] First, the terms used in the following description will be described.

[0022] CPU (central processing unit) is an abbreviation for "central processing unit". GPU (graphics processing unit) is an abbreviation for "graphics processing unit". GPGPU (general-purpose computing on graphics processing units) is an abbreviation for "general-purpose computing on graphics processing units". APU (accelerated processing unit) is an abbreviation for "accelerated processing unit". TPU is an abbreviation for "tensor processing unit". RAM (random-access memory) is an abbreviation for "random access memory". EEPROM (electrically erasable programmable read-only memory) is an abbreviation for "electrically erasable programmable read-only memory". ASIC (application-specific integrated circuit) is an abbreviation for "application-specific integrated circuit". PLD (programmable logic device) is an abbreviation for "programmable logic device". FPGA is an abbreviation for "field-programmable gate array". SoC is an abbreviation for "system-on-a-chip".SSD (solid-state drive) is an abbreviation for "solid-state drive". USB is an abbreviation for "Universal Serial Bus". EL is an abbreviation for "electroluminescence". CMOS (complementary metal-oxide semiconductor) is an abbreviation for "complementary metal-oxide semiconductor". CCD (charge-coupled device) is an abbreviation for "charge-coupled device". AI is an abbreviation for "artificial intelligence". BLI is an abbreviation for "blue light imaging". LCI is an abbreviation for "linked color imaging". I / F is an abbreviation for "interface". LAN (local area network) is an abbreviation for "local area network". WAN (wide area network) is an abbreviation for "wide area network". 5G is an abbreviation for "5th generation mobile communications system".

[0023] In the following description, a processor with a reference symbol (hereinafter simply referred to as a "processor") can be a computing device or a combination of several computing devices. The processor can be one type of computing device or a combination of several types of computing devices. Examples of computing devices include a CPU, a GPU, a GPGPU, an APU, and a TPU.

[0024] In the following description, a memory with a reference symbol is a memory such as RAM that temporarily stores information and is used by the processor as working memory.

[0025] In the following description, a data storage device, with a reference symbol, is one or more non-volatile storage devices that store various programs, parameters, and the like. Examples of non-volatile storage devices include flash memory, magnetic disks, and magnetic tape. Another example of data storage is cloud storage.

[0026] In the embodiment described below, an external interface, designated by a reference numeral, controls the transmission and reception of various types of information between multiple interconnected devices. Examples of the external interface include a USB interface. A communication interface, containing a communication processor, an antenna, and the like, can be applied to the external interface. The communication interface controls communication between multiple computers. Examples of a communication standard applied to the communication interface include a wireless communication standard, such as 5G, Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0027] In the embodiment described below, “A and / or B” is synonymous with “at least one of A and B”. That is, “A and / or B” may possibly mean only A, only B, or a combination of A and B. Furthermore, in the present description, the same concept as “A and / or B” is applied to a case in which the connection of three or more matters is expressed by “and / or”.

[0028] Fig. Figure 1 is a concept diagram showing a view example using an endoscope system 10. As shown in Fig. As shown in Figure 1, the endoscope system 10 is used by a physician 12 during an endoscopy and the like. The endoscopy is assisted by a staff member 14, such as a nurse.

[0029] The endoscope system 10 is communicatively connected to a communication device (not shown), and information received by the endoscope system 10 is transmitted to the communication device. Examples of the communication device include a server, a personal computer, and / or a tablet terminal that manage various types of information, such as electronic health records. The communication device receives the information transmitted by the endoscope system 10 and performs processing using the received information (for example, processing and storing the information in the electronic health record, or the like).

[0030] The endoscope system 10 comprises an endoscope 16, a display device 18, a light source device 20, a control device 22, and a medical support device 24. In the present embodiment, the endoscope system 10 is an example of an “endoscope system” according to the present disclosure, the endoscope 16 is an example of an “endoscope” according to the present disclosure, and the medical support device 24 is an example of a “medical support device” according to the present disclosure.

[0031] The endoscope system 10 is a modality for performing a medical examination on a large intestine 28, which is a lumen organ contained in the body of an examinee 26 (for example, a patient), by using the endoscope 16. In the present embodiment, the large intestine 28 is an object observed by the physician 12.

[0032] The endoscope 16 is used by the physician 12 and inserted into the body of the subject 26. In the present embodiment, the endoscope 16 is inserted into the large intestine 28 of the subject 26. In the present embodiment, the large intestine 28 is an example of a "lumen organ" as described in the present disclosure.

[0033] The endoscope system 10 visualizes the interior of the colon 28, including a lumen 42, using the endoscope 16, which is inserted into the colon 28 of the subject 26, and performs various medical procedures on the colon 28 as needed. The colon 28 contains the lumen 42. The endoscope 16 is inserted into the lumen 42. The position of the lumen 42 in the colon 28 can be medically specified based on a pattern of multiple folds 43 (for example, the shape, orientation, and the like of the multiple folds 43), which are characteristic regions in the colon 28.In the present embodiment, as detailed below, the position of the lumen 42 is detected by AI trained by machine learning using various types of information, such as the shape pattern of the multiple folds 43, and a result of the detection is provided to the physician 12 as visually identifiable information. In the present embodiment, the lumen 42 is an example of a “lumen” as defined in the present disclosure.

[0034] The endoscope system 10 acquires an image showing a view containing the lumen 42 in the colon 28 by imaging the interior of the colon 28 containing the lumen 42 and outputs the acquired image. In the present embodiment, the endoscope system 10 has an optical imaging function of emitting light 30 in the colon 28 and imaging reflected light obtained by reflection off an intestinal wall 32 of the colon 28.

[0035] It should be noted that the endoscopy of the large intestine 28 has been described here as an example, but this is only an example and the present disclosure is applicable to an endoscopy of a lumen organ, such as an esophagus, a stomach, a duodenum or a trachea.

[0036] The light source device 20, the control device 22, and the medical support device 24 are installed in a trolley 34. The trolley 34 is equipped with several tables arranged in an up-down direction, and the medical support device 24, the light source device 20, and the control device 22 are installed from a lower table to an upper table. The display device 18 is installed on the uppermost table of the trolley 34.

[0037] The control device 22 controls the entire endoscope system 10. The control device 22 performs various types of processing on the image obtained by imaging the intestinal wall 32 with the endoscope 16. Furthermore, the medical support device 24, under the control of the control device 22, performs AI-type processing or the like on the image that has undergone various types of processing by the control device 22 and outputs various types of information containing a processing result of the AI-type processing or the like.Examples of output destinations for the various types of information include the display device 18, a stationary storage medium (for example, a data storage device provided in the endoscope system 10, a data storage device of a server or the like that is communicatively connected to the endoscope system 10, and the like) and / or a portable storage medium (for example, a memory card, a USB flash drive, and the like).

[0038] The display device 18 shows various types of information (for example, different types of information output by the medical support device 24). Examples of the display device 18 include a liquid crystal display and an EL display. A tablet terminal with a display can be used instead of or in conjunction with the display device 18.

[0039] A screen 35 is displayed on the display device 18. The screen 35 contains several display areas. The several display areas are arranged on the screen 35. In the Fig. In the example shown, a first display area 35A and a second display area 35B are shown as examples of multiple display areas. The first display area 35A is larger than the second display area 35B. The first display area 35A is used as a primary display area, and the second display area 35B is used as a secondary display area. The size relationship between the first display area 35A and the second display area 35B is not limited to this and only needs to be a size relationship that can be accommodated within the screen 35.

[0040] An endoscope video image 39 is displayed in the first display area 35A. The endoscope video image 39 is obtained by performing various types of processing on several images arranged in time series, which are obtained by imaging the interior of the colon 28 of the subject 26 with the endoscope 16. The intestinal wall 32 shown in the endoscope video image 39 contains the lumen 42 as a region of interest (that is, a target area for observation) which the physician 12 is looking at, and the physician 12 can visually perceive the view of the intestinal wall 32 containing the lumen 42 through the endoscope video image 39.

[0041] The image displayed in the first display area 35A is a single frame 40 contained within a video image that contains multiple frames 40 arranged in time sequences. That is, the multiple frames 40 arranged in time sequences are displayed in the first display area 35A at predetermined frame rates (for example, one dozen frames per second or several dozen frames per second). In the present embodiment, the single frame 40 is an example of a “medical image” as described in the present disclosure.

[0042] Examples of the video image displayed in the first display area 35A include a video image in a live view mode. The live view mode is merely an example, and the video image could be a video image, such as a video image in a post-view mode, that is temporarily stored in memory or the like and then displayed. Furthermore, each frame contained in a video image for recording, stored in memory or the like, can be reproduced and displayed as the endoscope video image 39 on screen 35 (for example, in the first display area 35A).

[0043] The second display area 35B is shown in the lower right corner of the screen 35 in a front view. The second display area 35B can be displayed in any position as long as the position is within the screen 35 of the display device 18, but is preferably displayed in a position comparable to the endoscope video image 39. Auxiliary information 44 to assist the physician 12 in a medical determination or the like is displayed in the second display area 35B. The auxiliary information 44 is information to which the physician 12 should refer. Examples of the auxiliary information 44 include various types of information about the patient 26 into whom the endoscope 16 is inserted and / or various types of information obtained by performing medical support processing as described below.

[0044] Fig. Figure 2 is a concept diagram showing an example of a complete configuration of the endoscope system 10. As shown in Fig. As shown in Figure 2, the endoscope 16 comprises an operating part 46 and an insertion part 48. The insertion part 48 is partially curved by the operation of the operating part 46. The insertion part 48 is inserted into the large intestine 28, following the shape of the large intestine 28 (see Figure 2). Fig. 1) in accordance with the operation of the control unit 46 by the physician 12 (see Fig. 1) is curved.

[0045] A camera 52, a lighting device 54, and a treatment tool opening 56 are provided at a distal end section 50 of the insertion part 48. A section of the camera 52 (for example, an imaging optic) and a section of the lighting device 54 (for example, an optical irradiation system) are exposed by a distal end surface 50A of the distal end section 50.

[0046] The camera 52 is mounted in the endoscope 16 and is inserted into a body cavity of the subject 26 to image the observation target area. Examples of the camera 52 include a CMOS camera. However, this is only one example, and the camera 52 can be other types of cameras, such as CCD cameras. In the present embodiment, the camera 52 generates an image showing the view containing the lumen 42 in the colon 28 by imaging the interior of the colon 28 containing the lumen 42. The image generated by the camera 52 is a circular image. For example, the image generated by the camera 52 is processed by the control device 22 into a shape in which an upper end section and a lower end section are masked. Accordingly, as in Fig. As shown in Figure 1, an image in which the upper end edge and the lower end edge are linear and the left side edge and the right side edge are arc-shaped is produced as the single image 40.

[0047] The lighting device 54 includes lighting windows 54A and 54B. The lighting device 54 emits the light 30 (see Fig. 1) via the illumination windows 54A and 54B. Examples of the type of light 30 emitted by the illumination device 54 include visible light (e.g., white light) and invisible light (e.g., near-infrared light). In addition, the illumination device 54 emits special light via the illumination windows 54A and 54B. Examples of the special light include light for BLI and / or light for LCI. The camera 52 images the interior of the colon 28 by using an optical method in a state in which the illumination device 54 illuminates the interior of the colon 28 with the light 30.

[0048] The treatment tool opening 56 is an opening through which a treatment tool 58 protrudes from the distal end part 50. Furthermore, the treatment tool opening 56 is also used as a suction port for aspirating blood, internal contaminants, and the like, and as an ejection port for expelling fluid.

[0049] A treatment tool insertion port 60 is formed on the operating part 46, and the treatment tool 58 is inserted through the treatment tool insertion port 60 into the insertion part 48. The treatment tool 58 passes through the insertion part 48 to protrude outwards from the treatment tool opening 56. In the Fig. In the example shown, a view is depicted in which a biopsy needle, as the treatment tool 58, protrudes from the treatment tool opening 56. Here, the biopsy needle has been described as an example of the treatment tool 58, but this is merely an example, and the treatment tool 58 could be grasping forceps, a papillotomy knife, a snare, a catheter, a guidewire, a cannula, and / or a biopsy needle with a guide sheath.

[0050] The endoscope tube 16 is connected to the light source device 20 and the control device 22 via a universal cable 62. The medical support device 24 and a receiver device 64 are connected to the control device 22. Furthermore, the display device 18 is connected to the medical support device 24. That is, the control device 22 is connected to the display device 18 via the medical support device 24.

[0051] It should be noted that here, since the medical support device 24 is used as an example of an external device for extending the functions of the control device 22, the formal example described is one in which the control device 22 and the display device 18 are indirectly connected to each other via the medical support device 24, but this is merely an example. For instance, the display device 18 could be directly connected to the control device 22.In this case, for example, the functions of the medical support device 24 only need to be provided in the control device 22, or the control device 22 only needs to be provided with a function of instructing a server (not shown) to perform the same processing as the processing (for example, the medical support processing 24 described below), receiving a result of the processing by the server, and using the result.

[0052] The receiving device 64 receives an instruction from the physician 12 and outputs the received instruction as an electrical signal to the control device 22. Examples of the receiving device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and / or a remote control device.

[0053] The control device 22 controls the light source device 20, transmits and receives various signals to and from the camera 52 or transmits and receives various signals to and from the medical support device 24.

[0054] The light source device 20 emits light under the control of the control device 22 to supply the light 30 to the illumination device 54. A light guide is provided in the illumination device 54, and the light 30 supplied by the light source device 20 is emitted from the illumination windows 54A and 54B through the light guide. The control device 22 causes the camera 52 to perform imaging in a state in which the light 30 is emitted from the illumination windows 54A and 54B. The control device 22 generates the multiple individual images 40 arranged in time sequences by processing the external shape of the image obtained by the imaging performed by the camera 52 or by adjusting the image quality or the like.Then the control device 22 outputs the endoscope video image 39, which contains the several generated individual images 40 arranged in a time series, to a predetermined output destination (for example, the medical support device 24).

[0055] The medical support device 24 performs various types of processing on the endoscope video image 39 input by the control device 22 in order to support a medical treatment (here, for example, endoscopy). The medical support device 24 outputs the endoscope video image 39, which has undergone various processing operations, to a predetermined output destination (for example, the display device 18).

[0056] It should be noted that the example described here is one in which the endoscope video image 39 output by the control device 22 is output to the display device 18 via the medical support device 24, but this is merely an example. For instance, an aspect can be used in which the control device 22 and the display device 18 are connected and the endoscope video image 39, which has undergone various types of processing by the medical support device 24, is displayed on the display device 18 via the control device 22.

[0057] Fig. Figure 3 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope system 10. As shown in Fig. As shown in Figure 3, the control device 22 comprises a computer 66, a bus 68, and an external interface 70. The computer 66 comprises a processor 72, a memory 74, and a data storage 76. The processor 72, the memory 74, the data storage 76, and the external interface 70 are connected to the bus 68. The processor 72 controls the entire control device 22. The memory 74 and the data storage 76 are used by the processor 72.

[0058] The external interface 70 transmits and receives various types of information between one or more devices (hereinafter also referred to as "first external devices") that exist outside the control device 22 and the processor 72.

[0059] The camera 52 is one of the first external devices connected to the external interface 70, and the external interface 70 transmits and receives various types of information between the camera 52 and the processor 72. The processor 72 controls the camera 52 via the external interface 70. In addition, the processor 72 generates the endoscope video image 39 (see Fig. 1) by applying different types of processing to the image obtained by imaging the interior of the large intestine 28 (see Fig. 1) is obtained with the camera 52, executes, and captures the generated endoscope video image 39 via the external interface 70.

[0060] The light source device 20 is one of the first external devices connected to the external interface 70, and the external interface 70 transmits and receives various types of information between the light source device 20 and the processor 72. The light source device 20 supplies light to the lighting device 54 under the control of the processor 72. The lighting device 54 emits the light supplied by the light source device 20.

[0061] The receiving device 64 is one of the first external devices connected to the external interface 70, and the processor 72 detects the instruction received by the receiving device 64 via the external interface 70 and performs processing that corresponds to the detected instruction.

[0062] The medical support device 24 comprises a computer 78 and an external interface 80. The computer 78 comprises a processor 82, a memory 84, and a data storage device 86. The processor 82, the memory 84, the data storage device 86, and the external interface 80 are connected by a bus 88. In the present embodiment, the computer 78 is an example of a "computer" according to the present disclosure, and the processor 82 is an example of a "processor" according to the present disclosure.

[0063] It should be noted that the hardware configuration (that is, the processor 82, the memory 84 and the data storage 86) of the Computer 78 is essentially the same as the hardware configuration of the Computer 66, and therefore the description of the hardware configuration of the Computer 78 is omitted here.

[0064] The external interface 80 transmits and receives various types of information between one or more devices (hereinafter also referred to as "second external devices") that exist outside the medical support device 24 and the processor 82.

[0065] As one of the second external devices, the control device 22 is connected to the external interface 80. In the case of the Fig. In the example shown, the external interface 70 of the control device 22 is connected to the external interface 80. The external interface 80 transmits and receives various types of information between the processor 82 of the medical support device 24 and the processor 72 of the control device 22. For example, the processor 82 acquires the endoscope video image 39 (see Figure 3). Fig. 1) from the processor 72 of the control device 22 via the external interfaces 70 and 80 and performs various types of processing on the captured endoscope video image 39. The various types of processing performed by the processor 82 include AI-type processing (for example, processing using a lumen detection model 92 described below).

[0066] As one of the second external devices, the display device 18 is connected to the external interface 80. The processor 82 controls the display device 18 via the external interface 80 so that different types of information (for example, the endoscope video image 39, which has undergone different types of image processing) are displayed on the display device 18.

[0067] Fig. Figure 4 is a block diagram showing an example of the main functions of the processor 82 contained in the medical support device 24, and an example of the information stored in the data memory 86. As shown in Fig. As shown in Figure 4, a medical support program 90 is stored in the data storage device 86. In the present embodiment, the medical support program 90 is an example of a “program” according to the present disclosure.

[0068] Processor 82 reads the medical support program 90 from data memory 86 and executes the read medical support program 90 on memory 84 to perform the medical support processing. The medical support processing is implemented by processor 82, which acts as a detection unit 82A and a control unit 82B in accordance with the medical support program 90 executed on memory 84.

[0069] The lumen detection model 92 is stored in the data memory 86. As described in detail below, the lumen detection model 92 is a trained model used in the AI-type processing and is used by the detection unit 82A. In the present embodiment, the lumen detection model 92 is an example of a “trained model” according to the present disclosure.

[0070] Fig. Figure 5 is a block diagram showing an example of a hardware configuration of an electrical system of an information processing device 100 used to generate the lumen detection model 92. As shown in Fig. As shown in Figure 5, the information processing device 100 comprises a computer 102 and an external interface 104. The computer 102 comprises a processor 106, a memory 108, and a data storage device 110. The processor 106, the memory 108, the data storage device 110, and the external interface 104 are connected by a bus 112.

[0071] It should be noted that the hardware configuration (that is, the processor 106, the memory 108 and the data storage 110) of computer 102 is essentially the same as the hardware configuration of computer 66, and therefore the description of the hardware configuration of computer 102 is omitted here.

[0072] The information processing device 100 comprises a receiving device 116. The receiving device 116 is, for example, a keyboard and / or a mouse and receives an instruction from a user of the information processing device 100, and so on. The receiving device 116 is connected to the bus 112. The processor 106 detects the instruction received from the receiving device 116 and operates in accordance with the detected instruction.

[0073] A display device 118 shows various types of information, including images. Examples of the display device 118 include a liquid crystal display and an EL display. The display device 118 is connected to the bus 112. The processor 106 displays the results obtained by performing various types of processing on the display device 118.

[0074] The external interface 104 transmits and receives various types of information between one or more devices (hereinafter also referred to as "third external devices") that exist outside the information processing device 100 and the processor 106. The medical support device 24 is connected to the external interface 104 as one of the third external devices. Fig. In the example shown in Figure 5, the external interface 80 of the medical support device 24 is connected to the external interface 104. The external interface 104 controls the transmission and reception of various types of information between the processor 82 (see Figure 5). Fig. 3 and Fig. 4) the medical support device 24 and the processor 106 of the information processing device 100. For example, the information processing device 100 generates the lumen detection model 92 and transmits the generated lumen detection model 92 to the medical support device 24 via the external interfaces 80 and 104 in response to a request from the medical support device 24.

[0075] A machine learning processing program 120 is stored in data memory 110. Processor 106 reads the machine learning processing program 120 from data memory 110 and executes the read machine learning processing program 120 on memory 108 to perform machine learning processing. The machine learning processing is implemented by processor 106, which acts as a training data generation unit 106A and a learning execution unit 106B in accordance with the machine learning processing program 120 executed on memory 108.

[0076] A sample image set 122 is stored in data memory 110. As described in detail below, the sample image set 122 is used by the training data generation unit 106A.

[0077] Fig. Figure 6 is a concept diagram showing an example of processing content for the training data generation unit 106A. As shown in Fig. As shown in Figure 6, the information processing device 100 is used by an annotator 124. The annotator 124 is an operator who adds annotations for machine learning to given data (that is, an operator who performs labeling).

[0078] In the Fig. In the example shown, a keyboard 116A and a mouse 116B are shown as examples of the receiving device 116. The annotator 124 issues an instruction to the computer 102 via the keyboard 116A and the mouse 116B.

[0079] The example image set 122 contains several example images 122A, each showing different content. Example image 122A is a pre-defined medical image intended for use in object recognition processing (for example, processing where the recognition unit 82A detects the lumen 42 based on the single image 40 and the lumen recognition model 92). The image pre-defined as the medical image to be used in object recognition processing is an image corresponding to the single image 40. In other words, the image corresponding to the single image 40 can also be described as an image representing the single image 40. In other words, the image representing the single image 40 can also be described as an image showing a sample image of the single image 40.Here is a first example of the image showing the pattern image of single image 40, an image obtained by actually imaging the inside of the colon with the camera. A second example of the image showing the pattern image of single image 40 is a virtually generated image (for example, an image generated by generative AI such as Stable Diffusion or Midjourney).

[0080] The training data generation unit 106A acquires the example image 122A from the example image set 122 in response to the instruction received by the receiver 116. The training data generation unit 106A displays the example image 122A on a screen 118A of the display device 118. When the example image 122A is displayed on the screen 118A, the annotator 124 specifies a lumen correspondence position, which is the position of the lumen shown in the example image 122A, relative to the training data generation unit 106A via the receiver 116. The training data generation unit 106A assigns ground truth data 126 to the example image 122A on the basis of the lumen correspondence position specified via the receiving device 116 in order to generate training data 128.The assignment of the Ground Truth data 126 to the example image 122A is implemented by adding a label to specify the lumen correspondence position as the Ground Truth data 126 to the lumen correspondence position in the example image 122A.

[0081] In this way, the training data generation unit 106A repeatedly performs the processing of mapping the ground truth data 126 to each of the sample images 122A contained in the sample image set 122 in response to the instruction issued by the annotator 124 in order to generate multiple training data 128.

[0082] Fig. Figure 7 is a concept diagram showing an example of a composition of example image 122A. As in Fig. As shown in figure 7, a large intestine 132 is shown in example image 122A. In the Fig. The example shown in Figure 7 shows an intestinal wall 136 in which several folds 134 are formed and a lumen 138 in the example image 122A.

[0083] Example image 122A is divided into several subdivided regions 130A. Eight subdivided regions 130A1 to 130A8 are contained within the several subdivided regions 130A. The subdivided regions 130A1 to 130A8 are regions that extend radially from a center C1 of example image 122A to an outer edge of example image 122A and are arranged along a circumferential direction CD1 (in other words, around the center C1) of example image 122A.

[0084] Fig. 8 and Fig. Figure 9 are concept diagrams showing an example of a procedure in which the training data generation unit 106A maps the ground truth data 126 to the example image 122A to generate the training data 128.

[0085] As in Fig. As shown in Figure 8, in a state where example image 122A is displayed on screen 118A, the annotator 124 specifies a lumen correspondence position 139, which is the position of the lumen 138 shown in example image 122A, relative to the training data generation unit 106A via the receiving device 116. The training data generation unit 106A displays a circular frame 140 superimposed on example image 122A in response to the instruction received from the receiving device 116 and positions the frame 140 at a location surrounding the lumen 138 shown in example image 122A. The frame 140 is a marker that defines the lumen correspondence position 139 in example image 122A. That is, the position of an area surrounded by the single image 140 in the example image 122A is the lumen correspondence position 139.The size and position of frame 140 on screen 118A are freely changed in response to the instruction received from receiver 116. Here, frame 140 has a circular shape, but it can also have a shape other than a circle. The size of frame 140 can be changed in response to the instruction received from receiver 116.

[0086] The annotator 124 issues an acknowledgment instruction, which is an instruction to confirm the lumen correspondence position 139, to the training data generation unit 106A via the receiving device 116 in a state in which the single image 140 is positioned at the location surrounding the lumen 138. As a result, the training data generation unit 106A confirms the lumen correspondence position 139.

[0087] The training data generation unit 106A specifies the subdivided area 130A, which has a maximum overlap area with the single image 140 that defines the lumen correspondence position 139 among the multiple subdivided areas 130A. Then, the training data generation unit 106A generates the training data 128 by combining the ground truth data 126 with the specified subdivided area 130A (in which in Fig. 8 shown example, the subdivided area 130A2) is linked as the label to specify the subdivided area 130A in which the lumen 138 is shown.

[0088] Fig. Figure 8 shows an example of a procedure for generating the training data 128 in a case where the lumen 138 is shown in an area outside the central area in the example image 122A, while Fig. Figure 9 shows an example of a procedure for generating the training data 128 in a case where the lumen 138 is shown in the central area in the example image 122A. As in Fig. As shown in Figure 9, in a case where the lumen 138 is shown in the central area in the example image 122A, the training data generation unit 106A generates the training data 128 by linking the ground truth data 126 with each of the subdivided areas 130A (that is, the subdivided areas 130A1 to 130A8).

[0089] Fig. Figure 10 is a concept diagram showing a view example in which the learning execution unit 106B performs machine learning using the training data 128 to generate the lumen detection model 92. As shown in Fig. Figure 10 shows that in the information processing device 100, the learning execution unit 106B acquires the training data 128 generated by the training data generation unit 106A. The learning execution unit 106B performs the machine learning using the training data 128.

[0090] At the in Fig. In the example shown, the learning execution unit 106B contains a model 142. Examples of model 142 include a neural network. Examples of the neural network include a convolutional neural network. The learning execution unit 106B inputs the example image 122A, which is contained in the training data 128, to the model 142. In a case where the example image 122A is input, the model 142 performs an inference to output an inference result 144. The learning execution unit 106B computes an error 146 between the inference result 144 and the ground-truth data 126, which is contained in the training data 128.

[0091] The learning execution unit 106B calculates several fitting values ​​148 to minimize the error 146. Then, the learning execution unit 106B adjusts several optimization variables in the model 142 using the several fitting values ​​148 to optimize the model 142. For example, the several optimization variables represent several coupling weights and several offset values ​​included in the model 142.

[0092] The learning execution unit 106B repeatedly performs the learning process of inputting the example image 122A into the model 142, calculating the error 146, calculating the multiple fit values ​​148, and adjusting the multiple optimization variables in the model 142 using the multiple data sets 128. That is, the learning execution unit 106B adjusts the multiple optimization variables in the model 142 using the multiple fit values ​​148, which were calculated such that the error 146 is minimized for each of the multiple example images 122A contained in the multiple training data sets 128, in order to optimize the model 142. The lumen detection model 92 is generated by optimizing the model 142 in this way. The lumen detection model 92 is transmitted from the information processing device 100 to the medical support device 24 via the external interfaces 80 and 104 (see Fig. 5) is transmitted and received by the medical support device 24. Then, the lumen detection model 92 is stored in the data memory 86 by the processor 82 of the medical support device 24 (see Fig. 4) The lumen detection model 92 stored in the data memory 86 is used by the detection unit 82A (see Fig. 4).

[0093] In a case where the lumen detection model 92 is actually used, the single image 40 is input into the lumen detection model 92. Then the lumen detection model 92 recognizes the lumen 42 shown in the input single image 40 (see Fig. 1) A result of the recognition is displayed on screen 35. For example, a position where the lumen 42 is shown in frame 40 is visualized by displaying the marker or the like in accordance with one of eight areas (that is, eight areas corresponding to the subdivided areas 130A1 to 130A8) in frame 40.

[0094] Even in a case where the marker or the like is displayed and the area in which the lumen 42 is shown is in a visually specifiable state, it may be difficult for the physician 12 to visually determine where the lumen 42 is shown within the area corresponding to the position where the marker or the like is displayed. For example, the smaller the display size of the lumen 42 is in relation to the display size of the area in which the lumen 42 is shown among the eight areas in the single image 40, the more difficult it is to visually determine where the lumen 42 is shown within the area corresponding to the position where the marker or the like is displayed.

[0095] Therefore, in view of such circumstances, in the present embodiment the medical support processing is carried out by the processor 82 of the medical support device 24.

[0096] Fig. Figure 11 shows an example of processing content from the recognition unit 82A. As in Fig. As shown in Figure 11, an image 150, obtained by imaging the intestinal wall 32 in the large intestine 28, including the lumen 42, with the camera 52, is captured by the recognition unit 82A. The recognition unit 82A generates the single image 40 by performing various types of processing on the image 150. In the Fig. In the example shown in Figure 11, the intestinal wall 32 with the folds 43 and the lumen 42 are shown in the single image 40.

[0097] The detection unit 82A performs lumen detection processing 152 on the single image 40. The lumen detection processing 152 is a process of detecting the lumen 42 shown in the single image 40 using the lumen detection model 92 stored in the data memory 86 (in other words, a process of specifying an existence position of the lumen 42 shown in the single image 40 using the lumen detection model 92). The detection unit 82A captures the single image 40 from the camera 52 and inputs the captured single image 40 to the lumen detection model 92 to cause the lumen detection model 92 to generate confidence-level information 154.

[0098] Fig. Figure 12 shows an example of a composition of the confidence-level information 154 generated by the lumen detection model 92 in a case where the lumen 42 is shown in the single image 40. As in Fig. As shown in Figure 12, the confidence-level information 154 is information contained in a map 156 corresponding to the single image 40. The size and shape of the map 156 are the same as the size and shape of the single image 40. However, this is merely an example, and the outer contour of the map 156 only needs to be in a similar relationship to the outer contour of the single image 40.

[0099] A confidence level 158 (for example, a probability of the existence of lumen 42) is added to map 156. Here, map 156 is shown as an example, but single image 40 can be used instead of map 156. Map 156 contains several subdivided areas 160A, which correspond to the several subdivided areas 130A (see Fig. 7 to 9). Each of the several subdivided areas 160A is an area obtained by subdividing map 156 along a circumferential direction CD2 (in other words, around a center C2 of map 156). In the Fig. In the 12 examples shown, subdivided areas 160A1 to 160A8 are shown as examples of the multiple subdivided areas 160A. The subdivided areas 160A1 to 160A8 are areas obtained by subdividing the map 156 at intervals of a constant angle (for example, at intervals of 45 degrees) along the circumferential direction CD2. In other words, the subdivided areas 160A1 to 160A8 can be described as areas obtained by radially subdividing the map 156 into eight areas from the center C2 of the map 156 to an outer edge of the map 156.

[0100] In the present embodiment, card 156 is an example of an “image corresponding to the medical image” according to the present disclosure. Furthermore, in the present embodiment, confidence level 158 is an example of a “conf i “denz levels” according to the present disclosure. Furthermore, in the present embodiment, the circumferential direction CD2 is an example of a “circumferential direction” according to the present disclosure, and the subdivided areas 160A1 to 160A8 are examples of the “multiple subdivided areas” according to the present disclosure.

[0101] Several center lines CL are provided in map 156. The several center lines CL correspond to the several subdivided areas 160A and are arranged at equal intervals along the circumferential direction CD2. Each of the several center lines CL is a virtual line along a half-angle (for example, 22.5 degrees) of the constant angle described above from the center C2 in each subdivided area 160A. In the Fig. The 12 examples shown are centerlines CL1 to CL8, representing the multiple centerlines CL for the subdivided areas 160A1 to 160A8. Centerlines CL1 to CL8 are arranged around center C2 at 45-degree intervals.

[0102] Fig. Figure 13 shows a view example where several unit direction vectors 162 are added to the map 156. As in Fig. As shown in Figure 13, a direction from the center C2 to the existence position of each of the several subdivided regions 160A is determined by the unit direction vector 162. The unit direction vector 162 is added to each of the several subdivided regions 160A. The unit direction vector 162 is a unit vector that indicates a direction from the center C2 to the existence position of the subdivided region 160 (that is, the existence position of each of the subdivided regions 160A1 to 160A8). In the Fig. In the example shown in Figure 13, a unit direction vector 162 is added to each of the multiple subdivided regions 160A (that is, the subdivided regions 160A1 to 160A8) along the center line CL for each subdivided region 160A. In the present embodiment, the center C2 is an example of a “reference position” according to the present disclosure, and the multiple unit direction vectors 162 are examples of “multiple first vectors” according to the present disclosure.

[0103] Fig. Figure 14 shows a view example where several direction vectors 164 are added to the map 156. As in Fig. As shown in Figure 14, the controller 82B captures the confidence level information 154, including the map 156, to which the multiple unit direction vectors 162 (see Figure 14) are assigned. Fig. 13) were added by the recognition unit 82A. Then the controller 82B generates the multiple direction vectors 164 based on the multiple unit direction vectors 162 and the multiple confidence levels 158 contained in the confidence level information 154.

[0104] The direction vector 164 is a vector whose size is adjusted by adding the confidence level 158 of the subdivided region 160A, to which the unit direction vector 162 is added as a weight to the unit direction vector 162. The size of the direction vector 164 corresponds to the level of the confidence level 158, and the direction vector 164 becomes larger as the confidence level 158 increases.

[0105] Here, a specific example of a method for generating the direction vector 164 is described. For example, in the subdivided area 160A (in which in Fig. 14. Example shown, the subdivided area 160A1), to which “0.3” is added as the confidence level 158, a vector obtained by increasing the size of the unit direction vector 162 by 30%, as the direction vector 164 (in which in Fig. The direction vector 164B (shown in example 14) is generated. Furthermore, for example, in the subdivided area 160A (in which in Fig. 14. Example shown, the subdivided area 160A2), to which “0.7” is added as the confidence level 158, a vector obtained by increasing the size of the unit direction vector 162 by 70%, as the direction vector 164 (in which in Fig. Example 14 shows the direction vector 164A). Furthermore, for example, in the subdivided area 160A (in which in Fig. 14 shown example, the subdivided areas 160A3 to 160A8), to which “0,0” is added as the confidence level 158, the size of the direction vector 164 can be set to “zero” or the unit direction vector 162 can be used as the direction vector 164 as it is.

[0106] It should be noted that the direction vector 164 shown here is merely an example and a vector obtained by simply multiplying the unit direction vector 162 by the confidence level 158 can be used as the direction vector 164.

[0107] In the Fig. In example 14, direction vectors 164A and 164B are shown. Direction vector 164A is a vector obtained by adjusting the size of the unit direction vector 162, which corresponds to the subdivided region 160A2 with a confidence level of 158 (0.7 in the range shown in Figure 14). Fig. The direction vector 164B is obtained by adjusting the size of the unit direction vector 162, which is added to the subdivided region 160A2 with confidence level 158 (0.3 in the example shown in 14). Fig. The magnitude of the direction vector 164B represents a degree of probability of the existence of lumen 42 in the subdivided region 160A to which the unit direction vector 162 of direction vector 164B is added. That is, the larger the direction vector 164B, the higher the probability of the existence of lumen 42 in the subdivided region 160A to which a base unit direction vector 162 of direction vector 164B is added.

[0108] The controller 82B generates a vector sum 166 based on the multiple direction vectors 164. The vector sum 166 is a sum of the multiple direction vectors 164. In the Fig. In the example shown in Figure 14, the vector sum 166 is shown to be a vector sum of the direction vector 164A and the direction vector 164B.

[0109] A direction from center C2 to a lumen existence area 168 in map 156 is determined by the vector sum 166. The lumen existence area 168 refers to an area in which the lumen 42 is shown in the single image 40. The subdivided area 160A is an area in which the position in map 156 is restricted, whereas the lumen existence area 168 is an area in which the position changes depending on the position at which the vector sum 166 is generated, without the position in map 156 being restricted, as in the subdivided area 160A.Furthermore, even if lumen 42 exists within subdivided region 160A, estimating its position within that region is difficult. However, in subdivided region 168, estimating its position is straightforward because lumen 42 exists along the line corresponding to vector sum 166. Therefore, the position of lumen 42 is specified with greater accuracy in subdivided region 168 than in subdivided region 160A. In other words, the lumen existence area 168 can be described as an area in which the position at which the lumen 42 is shown in the single image 40 (that is, the existence position of the lumen 42 in the single image 40) can be specified along the circumferential direction CD2 with a higher resolution than in the several subdivided areas 160A.

[0110] The controller 82B specifies the lumen existence range 168 based on the vector sum 166. For example, the controller 82B specifies as the lumen existence range 168 a range of ±α degrees along the circumferential direction CD2 with a point (for example, an endpoint) other than a starting point of the vector sum 166 as the center. Examples of ±α degrees include ±22.5 degrees. It should be noted that ±22.5 degrees is merely an example and the range can be narrower or wider than ±22.5 degrees. Furthermore, the α degree can be a fixed value or a variable value that is changed in accordance with an instruction received from the receiving device 64 or various conditions (for example, a type of operating mode of the endoscope system 10).

[0111] The controller 82B generates a marker 169 to specify the position of the lumen existence area 168 in the map 156 based on the multiple subdivided areas 160A and the multiple confidence levels 158. The marker 169 is visible information determined based on the vector sum 166, which is generated based on the multiple subdivided areas 160A and the multiple confidence levels 158. In the Fig. In the example shown in Figure 14, the shape of marker 169 is an arc where the endpoint of the vector sum 166 is a midpoint. A midpoint of the arc, which is the shape of marker 169, is the midpoint C2 of map 156. Marker 169 indicates a region from one end to the other end of the lumen existence region 168 in the circumferential direction CD2.

[0112] In the present embodiment, the vector sum 166 is an example of a “second vector” according to the present disclosure. Furthermore, in the present embodiment, the direction vector 164 is an example of a “third vector” according to the present disclosure. Furthermore, in the present embodiment, the marking 169 is an example of a “marking” according to the present disclosure.

[0113] Fig. Figure 15 shows a form example where the single image 40 and the like are displayed on the screen 35 in a case where the lumen 42 is shown in an area other than the central area of ​​the single image 40. As in Fig. As shown in Figure 15, the controller 82B captures the single image 40, which is input by the detection unit 82A into the lumen detection model 92, in order to obtain the confidence level information 154, including the map 156, which is used to generate the mark 169. The controller 82B displays the single image 40 captured by the detection unit 82A in the first display area 35A and displays the mark 169 in the first display area 35A in a manner comparable to the single image 40. For example, the mark 169 is displayed superimposed on the single image 40.

[0114] Furthermore, the controller 82B updates the marker 169 in accordance with a display time of frame 40. For example, the controller 82B generates the marker 169 based on the confidence level information 154 in order to display the marker 169 superimposed on frame 40 each time the confidence level information 154 is received from the recognition unit 82A. In such a case, the marker 169, which is displayed in the first display area 35A, is updated each time frame 40 is displayed.It should be noted that the marker 169, which is displayed in the first display area 35A, can be updated provided that the single image 40 is updated several times and displayed in the first display area 35A (for example, the single image 40 is displayed in the first display area 35A in an area of ​​several sheets up to several hundred sheets that have been predetermined).

[0115] Furthermore, the controller 82B displays visible information 44A in the second display area 35B as part of the auxiliary information 44. Examples of the visible information 44A include text specifying the position of an area corresponding to the lumen existence area 168 in the single image 40, that is, text specifying the position of the marker 169 displayed in the first display area 35A (for example, text indicating an angle that specifies the position of the vector sum 166 in a case where a boundary line between the subdivided area 160A1 and the subdivided area 160A8 is 0 degrees). In the present embodiment, the visible information 44A is an example of "lumen specification information" as disclosed herein.

[0116] Fig. Figure 16 shows a form example where the single image 40 and the like are displayed on the screen 35 in a case where the lumen 42 is shown in the central area of ​​the single image 40 (for example, in a case where the center of the lumen 42 coincides with the center of the single image 40). As in Fig. As shown in Figure 16, in a case where the center of lumen 42 coincides with the center of frame 40, controller 82B displays frame 40 in the first display area 35A and overlays a marker 170 onto frame 40. The marker 170 is a marker (for example, a ring-shaped marker) that surrounds the lumen 42 shown in frame 40. In a case where the direction vectors 164 of all subdivided areas 160A are equivalent, that is, in a case where the vector sum 166 is zero, the marker 170 is generated by controller 82B and overlaid onto frame 40. Furthermore, in this case, the control 82B displays the visible information 44A (for example, text) indicating that the lumen 42 is displayed in the middle of the single image 40 in the second display area 35B.

[0117] It should be noted that although the example described here, in which marker 170 is generated and displayed superimposed on frame 40, is only an example in a case where the vector sum 166 is zero, this is merely an illustration. For example, in a case where the size of the vector sum 166 is less than a threshold value (for example, the size of the unit direction vector 162), marker 170 can be generated and displayed superimposed on frame 40. Furthermore, as an example of marker 170, the marker surrounding the lumen 42 shown in single image 40 has been described here, but this is merely an example, and a marker (for example, a dot located in the center of the lumen 42 shown in single image 40, or an arrow indicating the position of the lumen 42) can be used to specify the position of the lumen 42 shown in single image 40.

[0118] The following describes the operation of the information processing device 100 with reference to Fig. 17 described.

[0119] In the processing for machine learning, which is in Fig. As shown in Figure 17, the training data generation unit 106A first acquires an unprocessed sample image 122A from sample image set 122, which is stored in data memory 110, during step ST10. Here, the unprocessed sample image 122A means the sample image 122A that has not yet been used for machine learning processing. The training data generation unit 106A displays the sample image 122A, which was acquired from sample image set 122, on screen 118A. After the processing of step ST10 has been completed, the machine learning processing proceeds to step ST12.

[0120] In step ST12, the training data generation unit 106A receives the specification of the lumen correspondence position 139. After the processing of step ST12 has been carried out, the processing for machine learning proceeds to step ST14.

[0121] In step ST14, the training data generation unit 106A specifies a positional relationship between the lumen correspondence position 139, received in step ST12, and the multiple subdivided areas 130A. After the processing of step ST14 is completed, the machine learning processing proceeds to step ST16.

[0122] In step ST16, the training data generation unit 106A assigns the ground truth data 126 to the sample image 122A, which was captured in step ST10, in accordance with the positional relationship specified in step ST14. For example, in a case where the lumen correspondence position 139 exists in an area outside the central area of ​​the sample image 122A, the ground truth data 126 is assigned to the subdivided area 130A, which has the greatest overlap with the lumen correspondence position 139, in response to the instruction issued by the annotator 124. Furthermore, for example, in a case where the lumen correspondence position 139 exists in the central area of ​​the example image 122A, the ground truth data 126 are assigned to each of the subdivided areas 130A in response to the instruction issued by the annotator 124.As described above, the training data generation unit 106A maps the ground truth data 126 to the example image 122A to generate the training data 128. The training data 128 generated in this way is stored on a predetermined storage medium (for example, the data storage 110). After the processing of step ST16 has been executed, the machine learning processing proceeds to step ST18.

[0123] In step ST18, the training data generation unit 106A determines whether the unprocessed example image 122A exists or not. If the unprocessed example image 122A exists, a negative determination is made in step ST18, and the machine learning processing proceeds to step ST10. If the unprocessed example image 122A does not exist, a positive determination is made in step ST18, and the machine learning processing proceeds to step ST20.

[0124] In step ST20, the learning execution unit 106B performs machine learning using the multiple training data 128 obtained by repeatedly performing the processing in steps ST10 to ST18 to generate the lumen detection model 92 (see Fig. 10). The lumen detection model 92 is stored in the data memory 86 of the medical support device 24 (see Fig. 4) After the processing of step ST20 has been completed, the machine learning processing ends.

[0125] Below, a process of a part of the endoscope system 10 according to the present disclosure is described with reference to Fig. 18 described. One in Fig. Figure 18, showing the medical support processing flow, is an example of a “medical support process” according to the present disclosure. It should be noted that, for the sake of simplicity, the description below assumes that the lumen detection model 92 is stored in the data memory 86.

[0126] In the medical support processing that takes place in Fig. As shown in Figure 18, in step ST50 the recognition unit 82A captures image 150 from camera 52 and performs various types of processing on the captured image 150 to generate the single image 40. After the processing in step ST50 has been completed, the medical support processing proceeds to step ST52.

[0127] In step ST52, the detection unit 82A performs the lumen detection processing 152 on the single image 40 generated in step ST50, using the lumen detection model 92 stored in the data memory 86, to generate the confidence level information 154. After the processing in step ST52 is completed, the medical support processing proceeds to step ST54.

[0128] In step ST54, the controller 82B generates the multiple direction vectors 164 by adding the confidence level 158 of the subdivided area 160A, to which each unit direction vector 162 is added, as a weight to the unit direction vector 162 of each subdivided area 160A of the map 156, which are contained in the confidence level information 154 generated in step ST52. After the processing in step ST54 is completed, the medical support processing proceeds to step ST56.

[0129] In step ST56, the controller 82B generates the vector sum 166, which is the sum of the multiple direction vectors 164 generated in step ST54. After the processing in step ST56 is completed, the medical support processing proceeds to step ST58.

[0130] In step ST58, the controller 82B specifies the lumen existence range 168 based on the vector sum 166 generated in step ST56. After the processing in step ST58 has been completed, the medical support processing proceeds to step ST60.

[0131] In step ST60, the controller 82B generates the marker 169 to specify the position of the lumen existence area 168, which was specified in step ST58. After the processing in step ST60 is completed, the medical support processing proceeds to step ST62.

[0132] In step ST62, the controller 82B displays the single image 40, which was generated in step ST50, in the first display area 35A. After the processing in step ST62 has been completed, the medical support processing proceeds to step ST64.

[0133] In step ST64, the controller 82B displays the marker 169, which was generated in step ST60, superimposed on the single image 40 displayed in the first display area 35A. After the processing in step ST64 has been completed, the medical support processing proceeds to step ST66.

[0134] In step ST66, the controller 82B determines whether a medical support processing end condition is met. Examples of medical support processing end conditions include a condition that a medical support processing termination instruction is issued to the endoscope system 10 (for example, a condition that the receiving device 64 receives the medical support processing termination instruction).

[0135] If the end condition for medical support processing is not met at step ST66, a negative determination is made, and the medical support processing proceeds to step ST50. If the end condition for medical support processing is met at step ST66, an affirmative determination is made, and the medical support processing ends.

[0136] As described above, in the endoscope system 10, the single image 40, showing the intestinal wall 32 and the lumen 42, is entered into the lumen detection model 92, and thus the confidence level information 154 is generated by the lumen detection model 92. The confidence level information 154 contains the map 156, which is subdivided into several subdivided areas 160A. The confidence level 158, indicating that the lumen 42 exists, is added to each of the several subdivided areas 160A.

[0137] In the endoscope system 10, the lumen existence area 168 is generated based on the multiple subdivided areas 160A and the multiple confidence levels 158. Lumen existence area 168 is an area in which the existence position of the lumen 42 (that is, the position at which the lumen 42 is shown) is specified with higher accuracy than in the subdivided area 160A in the single image 40. Since the lumen existence area 168 is generated based on the multiple subdivided areas 160A and the multiple confidence levels 158, the position of the lumen existence area 168 in the map 156 is not fixed as it is for the subdivided area 160A. Furthermore, the position of the lumen existence area 168 in the map 156 is finely modified along the circumferential direction CD2 depending on the existence position of the lumen 42.This means that the lumen existence area 168 is an area in which the existence position of the lumen 42 is defined with a higher resolution than in the several subdivided areas 160A along the circumferential direction CD2.

[0138] In the endoscope system 10, the marker 169 is generated as information to specify the lumen existence range 168. Then, the single image 40, which is input into the lumen detection model 92 to generate the confidence level information 154, is displayed in the first display area 35A of the screen 35. Furthermore, the marker 169 is superimposed on the single image 40. This means that the marker 169 on the single image 40 is displayed with a higher resolution than in a case where the confidence level 158 of each of the several subdivided areas 160A is simply superimposed on the single image 40, or where the visible information (for example, the marker) indicating the level of the confidence level 158 is simply displayed in the first display area 35A.

[0139] Therefore, the physician 12 can determine the position of the lumen 42 shown in the single image 40 more accurately in the single image 40 by simply visually recognizing the marker 169, which is displayed superimposed on the single image 40, compared with a case in which the confidence level 158 of each of the several subdivided areas 160A is simply displayed superimposed on the single image 40, or the visible information indicating the level of the confidence level 158 is simply displayed in the first display area 35A.

[0140] Furthermore, in the endoscope system 10, marker 169 is displayed in a manner comparable to single image 40. That is, marker 169 is superimposed on single image 40. As a result, the physician 12 can visually determine the positional relationship between single image 40 and marker 169.

[0141] Furthermore, in the endoscope system 10, the direction from the center C2 of map 156 to the existence position of each of the several subdivided areas 160A is determined by the multiple unit direction vectors 162. Additionally, the direction from the center C2 of map 156 to the lumen existence area 168 is determined by the vector sum 166. The vector sum 166 is a sum of the multiple direction vectors 164, obtained by adding the confidence level 158 as the weight to the multiple unit direction vectors 162. The marker 169, which is superimposed on the single image 40, is generated based on the vector sum 166.

[0142] Here, the multiple direction vectors 164 are modified in accordance with the confidence level 158 added to each of the multiple subdivided areas 160A. The vector sum 166 is modified in accordance with the multiple direction vectors 164. It can be said that the vector sum 166 is a vector indicating the direction from the center C2 of the map 156 to the existence position of the lumen 42. It can be said that the marker 169, which is generated on the basis of the vector sum 166, is visible information that represents the existence position of the lumen 42, which is shown in the single image 40, in the single image 40 compared with a case in which the confidence level 158 of each of the several subdivided areas 160A is simply superimposed on the single image 40, or the visible information indicating the height of the confidence level 158 is simply displayed in the first display area 35A, with higher accuracy.

[0143] Therefore, the physician 12 can determine the position of the lumen 42 shown in the single image 40 more accurately in the single image 40 by simply visually recognizing the marker 169, which is displayed superimposed on the single image 40, compared with a case in which the confidence level 158 of each of the several subdivided areas 160A is simply displayed superimposed on the single image 40, or the visible information indicating the level of the confidence level 158 is simply displayed in the first display area 35A.

[0144] Furthermore, in the endoscope system 10, the marker 169 and / or 170, displayed in the first display area 35A, is updated in accordance with the display time of the single image 40. Similarly, the visible information 44A, displayed in the second display area 35B, is also updated in accordance with the display time of the single image 40. Therefore, the physician 12 can visually identify the marker 169 and the visible information 44A (that is, the marker 169 and the visible information 44A for specifying the position of the lumen 42, shown in the single image 40, in the single image 40 displayed in the first display area 35A) in accordance with the contents of the single image 40 displayed in the first display area 35A.

[0145] Furthermore, in the endoscope system 10, the shape of the marker 169 is an arc, and the center of the arc is the center of the single image 40. The marker 169 is not superimposed on the entire single image 40, but only on a portion of it. By superimposing the marker 169 on the single image 40, the physician 12 can accurately determine the position of the lumen 42 shown in the single image 40, while ensuring the visibility of the single image 40 displayed in the first display area 35A.

[0146] It should be noted that in the embodiment described above, the arc is shown as an example of the shape of the marking 169, but this is only an example and the marking may have a different shape.

[0147] As in Fig. As shown in Figure 19 as an example, a marker 169A, having a shape along the outer edge of frame 40, can be generated and displayed in the first display area 35A instead of marker 169. Examples of a method for generating marker 169A include a method in which marker 169 is projected from the side of center C2 onto the outer edge of map 156 to generate the marker with a shape along the outer edge of map 156 as marker 169A. Frame 40, which is input into the lumen detection model 92 to generate the confidence level information 154 including map 156, is displayed in the first display area 35A, and marker 169A is displayed in accordance with the display time of frame 40.

[0148] Since the marker 169 is formed within the map 156, a form example of the procedure for generating the marker 169A has been described here, in which the marker 169 is projected from the side of the center C2 onto the outer edge of the map 156. However, in a case where the marker 169 is formed outside the map 156 (for example, in a case where the marker 169 is formed below an upper end of the map 156 or below a lower end of the map 156), the marker 169 only needs to be projected from the outside of the map 156 onto the outer edge of the map 156 in the direction of the side of the center C2.

[0149] In this way, in a case where the marking with a shape is displayed along the outer edge of the single image 40 in the first display area 35A, the display of the object that visually blocks the single image 40 in the first display area 35A is suppressed, so that it is possible to enable the physician 12 to accurately determine the position of the lumen 42 shown in the single image 40, while ensuring the visibility of the single image 40 displayed in the first display area 35A.

[0150] Furthermore, marker 169A can also be updated in accordance with the display time of single image 40 in the same way as marker 169 is updated in accordance with the display time of single image 40. In this way, the physician 12 can visually recognize marker 169A in accordance with the contents of single image 40 displayed in the first display area 35A.

[0151] In the embodiment described above, an example form was described in which the marker 169 is displayed superimposed on the single image 40, but this is merely an example, and the marker 169 can also be displayed outside the single image 40. In a case where the marker 169 is displayed outside the single image 40, there is no object that visually blocks the single image 40 displayed in the first display area 35A, and thus the visibility of the single image 40 displayed in the first display area 35A can be improved.

[0152] In the embodiment described above, an example form has been described in which the controller 82B generates the arc-shaped marking 169 to specify the area from one end to the other end of the lumen existence area 168 in the circumferential direction CD2 and displays the generated marking 169 in the first display area 35A; however, this is only an example. For instance, as in Fig. As shown in Figure 20, the controller 82B displays the marker 169 in the first display area 35A by displaying at least one marker 171 to specify the area from one end to the other end of the lumen existence area 168 in the circumferential direction CD2 under several hidden markers 171, which are assigned to the card 156 in the first display area 35A. For example, the number of multiple markers 171 (in other words, the number of subdivided parts) can be "40" (= 8 × 5), which is obtained by displaying each of the eight subdivided areas 160A (see Figure 20). Fig. 12 to 14) is divided equally into five parts, or “80” (= 8 × 10), which is obtained by dividing each of the eight subdivided areas 160A equally into ten parts. The number of multiple markings 171 can be a different number than these. As the number of markings 171 increases, the resolution of the marking 169 display increases. That is, the controller 82B can generate and display the marking 169 in greater detail as the number of markings 171 increases.

[0153] In the Fig. The example shown in Figure 20 shows several arc-shaped markings arranged at constant intervals along a circle with a center that coincides with the center C2 of map 156, as examples of the several hidden markings 171.

[0154] In the Fig. In the example shown in 20, the display of the marker 169 in the first display area 35A is implemented by displaying at least one marking 171 to specify the area of ​​the lumen existence area 168 from one end to the other end in the circumferential direction CD2.

[0155] In the Fig. In the example shown in Figure 20, the marker 169 is superimposed on the single image 40, which is displayed in the first display area 35A, but the marker 169 can be displayed outside of the single image 40 depending on the position of the lumen existence area 168. For example, at least one marker 171 is displayed at a position (in which in Fig. 20 example shown above the top of map 156 in a front view and below the bottom of map 156 in a front view) is shown away from map 156, and thus marker 169 is shown outside of single image 40, which is shown in the first display area 35A.

[0156] In the Fig. In the 20 examples shown, the multiple arc-shaped markers arranged at constant intervals along a circle with a center coinciding with center C2 of map 156 were shown as examples of the multiple hidden markers 171, but this is only one example. For instance, the multiple hidden markers 171 could be multiple markers arranged at constant intervals along the outer edge of map 156.

[0157] In the Fig. The example shown in Figure 20 illustrates the multiple arc-shaped markers arranged at constant intervals along a circle whose center coincides with the center C2 of map 156 and whose portion overlaps map 156, but this is merely an example. For instance, multiple arc-shaped markers arranged at constant intervals along a circle whose center coincides with the center C2 of map 156 and which surrounds map 156 can be used. In this case, the marker 169 outside of frame 40 is indicated by displaying at least one marker 171 to specify an area from one end to the other end of the lumen existence area 168 in the circumferential direction CD2.

[0158] As described above, the display of the marker 169 is implemented by displaying at least one marker 171 at a position corresponding to the position of the lumen existence area 168 among the multiple hidden markers 171, so that a processing load required for displaying the marker 169 can be reduced.

[0159] In the Fig. The example shown in section 20 describes a form example where marker 169 is displayed; however, this is merely an example, and, for instance, as in Fig. As shown in Figure 21, an outer contour line 172 can be displayed in the first display area 35A instead of the marker 169. The outer contour line 172 is a line that outlines the outer contour of the lumen existence area 168. In the Fig. In the example shown in Figure 21, the outer contour line 172 is superimposed on the single image 40, which is displayed in the first display area 35A. In a case where the outer contour line 172 is superimposed on the single image 40 in this way, the physician 12 can visually recognize that the lumen 42 is displayed within the outer contour line 172.

[0160] Furthermore, as in the example in Fig. Figure 21 shows that a line segment 173 (for example, a line segment extending from the center C2 along the vector sum 166) is superimposed on the single image 40 along the vector sum 166. In this case, the physician 12 can visually recognize that the lumen 42 is shown on the line segment 173, which is superimposed on the single image 40.

[0161] In the embodiment described above, eight subdivided areas 160A are shown as examples, but the number of subdivided areas 160A can be less than eight or nine or more. Furthermore, the number of subdivided areas 130A must also be determined in accordance with the number of subdivided areas 160A.

[0162] In the embodiment described above, a form example was described in which the single image 40 is displayed on the screen 35 and the marker 169 is displayed in a manner comparable to the single image 40 (for example, the form example in which the marker 169 is displayed superimposed on the single image 40), but this is only an example. For example, the card 156 can be displayed on the screen 35, and the marker 169 can be displayed in a manner comparable to the card 156. An example of the comparable display example is the superimposed display of the marker 169 on the card 156. It should be noted that here the card 156 is an example of an “image corresponding to the medical image” according to the present disclosure.

[0163] In the embodiment described above, an example form has been described in which the visible information 44A is displayed in the second display area 35B; however, this is merely an example. For instance, audible information (such as an electronic tone or a speech tone) to specify the position of the lumen 42 in the single image 40 shown in the single image 40 can be emitted by a loudspeaker (not shown). Furthermore, the information in which the single image 40 and the marker 169 or the like are combined and / or the visible information 44A can be printed on the medium by a printer.Furthermore, the information combining the single image 40 and the marker 169 or the like, the visible information 44A and / or the audible information described above can be stored on a storage medium (for example, a data storage device provided in an external device such as the data storage device 76, the data storage device 86 or the server).

[0164] In the embodiment described above, the embodiment in which the medical support processing is performed by the computer 78 is described, but the present disclosure is not limited thereto, and at least part of the medical support processing can be performed by a device provided outside of the computer 78. An example of such a case is given below with reference to Fig. 22 described.

[0165] Fig. 22 is a concept diagram showing an example of a configuration of an endoscope system 174. In which in Fig. The endoscope system 174 shown in Figure 22 is an example of an “endoscope system” according to the present disclosure. The endoscope system 174 differs from the endoscope system 10 according to the embodiment described above in that an external device 176 is provided.

[0166] For example, the external device 176 is a server and is connected to the computer 78 via a network 178 (for example, a WAN and / or a LAN) so that it can communicate with the computer 78. Although the server is shown here as an example, at least one personal computer or the like can be used instead of the server as the external device 176.

[0167] Examples of the external device 176 include at least one server that transmits data directly or indirectly to and from the computer 78 via the network 178. The external device 176 receives a processing instruction issued by the processor 82 of the computer 78 via the network 178. The external device 176 then performs processing in accordance with the received processing instruction and transmits a processing result to the computer 78 via the network 178. In the computer 78, the processor 82 receives the processing result transmitted by the external device 176 via the network 178 and performs processing using the received processing result.

[0168] Examples of the processing execution instruction include an instruction to cause the external device 176 to perform at least one part of the medical support processing. A first example of the at least one part of the medical support processing (that is, processing to be performed by the external device 176) is the lumen detection processing 152. In this case, the external device 176 performs the lumen detection processing 152 in response to the processing execution instruction issued by the processor 82 via the network 178 and transmits information containing the confidence-level information 154 as a first processing result via the network 178 to the computer 78.In the computer 78, the processor 82 receives the first processing result and performs the same processing as in the embodiment described above, using the received first processing result.

[0169] A second example of at least part of the medical support processing (that is, the processing to be performed by the external device 176) is processing via the controller 82B. In this case, the external device 176 performs the processing via the controller 82B in response to the processing execution instruction issued by the processor 82 via the network 178, and transmits a second processing result (for example, the marking 169 and / or the visible information 44A) via the network 178 to the computer 78. In the computer 78, the processor 82 receives the second processing result and performs the same processing (for example, displaying the information using the display device 18) as in the embodiment described above, using the received second processing result.

[0170] It should be noted that the external device 176 can be implemented through cloud computing. Cloud computing is merely one example, and the external device 176 can also be implemented through network computing, such as fog computing, edge computing, or grid computing.

[0171] In the embodiment described above, the embodiment in which the medical support program 90 is stored in the data storage device 86 was described, but the present disclosure is not limited thereto. For example, the medical support program 90 may be stored on a portable, computer-readable, non-volatile storage medium, such as an SSD or a USB flash drive. The medical support program 90, stored on the non-volatile storage medium, is installed on the computer 78 of the endoscope system 10. The processor 82 performs the medical support processing in accordance with the medical support program 90.

[0172] Furthermore, the medical support program 90 can be stored in a storage device of another computer, server or the like, which is connected to the endoscope system 10 via the network, and the medical support program 90 can be downloaded from the endoscope system 10 and installed on the computer 78 in response to a request.

[0173] It is not necessary to store the entire medical support program 90 in a storage device of another computer or server device connected to the endoscope system 10, or to store the entire medical support program 90 in the data storage 86, and part of the medical support program 90 may be stored.

[0174] The following different processors can be used as hardware resources for performing medical support processing. One example is a CPU, which is a general-purpose processor that executes software, i.e., a program, to perform medical support processing. Another example is a dedicated electrical circuit, which is a processor with a dedicated circuit configuration designed to perform specific processing, such as an FPGA, a PLD, or an ASIC. All processors have memory built into or connected to them, and all processors use this memory to perform medical support processing.

[0175] The hardware resource for performing medical support processing can be configured by one of several processors or by a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, the hardware resource for performing medical support processing can be a single processor.

[0176] A first example of a configuration where the hardware resource is configured by a processor is one where a processor is configured by a combination of one or more CPUs and software and acts as the hardware resource for performing medical support processing. A second example involves a processor that implements all the functions of a system, including multiple hardware resources that perform medical support processing, using an integrated circuit chip (IC chip), such as a system-on-a-chip (SoC). As described above, the medical support processing is implemented by using one or more of these different processors as the hardware resource.

[0177] Furthermore, the hardware structure of the various processors can be, in particular, an electronic circuit in which circuit elements such as semiconductor elements are combined. Moreover, the medical support processing described above is merely an example. Therefore, it is self-evident that, within the scope that does not deviate from the core of this disclosure, unnecessary steps may be deleted, new steps may be added, or the processing order may be changed.

[0178] The content described and shown above is a detailed description of the parts according to this disclosure and is merely an example of this disclosure. For instance, the descriptions of the configurations, functions, operations, and effects are descriptions of examples of the configurations, functions, operations, and effects of the parts according to this disclosure. It is understood, therefore, that with regard to the content described and shown above, unnecessary parts may be deleted, new elements may be added, or substitutions may be made, within a scope that does not deviate from the core of this disclosure.To avoid confusion and to facilitate understanding of the parts according to the present disclosure, the description of general technical knowledge or the like, which does not specifically require description to enable the implementation of the present disclosure, is furthermore omitted from the contents described above and the contents shown above.

[0179] All documents, patent applications and technical standards described in this description are incorporated into this description by reference to the same extent as in a case where it is specifically and individually described that each of the documents, patent applications and technical standards is described by reference. Reference symbol list 10, 174 Endoscope system 12 Doctor 14 employees 16 Endoscope 18, 118 Display device 20 Light source device 22 Control device 24 medical support devices 26 subjects 28 Large intestine 30 lights 32, 132 Intestinal wall 34 cars 35, 118A screen 35A first display area 35B second display area 39 Endoscope video image 40 individual images 42,138 lumens 43, 134 fold 44 Help Information 44A visible information 46 Control unit 48 Insert part 50 distal end section 50A distal end face 52 Camera 54 Lighting device 54A, 54B: Lighting window 56 Treatment tool opening 58 Treatment tool 60 Treatment tool insertion port 62 Universal cables 64, 116 Receiving device 66, 78, 102 Computer Bus 68, 88, 112 70, 80, 104 external interface 72, 82, 106 processor 74, 84, 108 memory 76, 86, 110 data storage 82A Detection Unit 82B Control 90 medical support program 92 lumen detection model 100 Information processing device 106A Training Data Generation Unit 106B Learning Execution Unit 116A keyboard 116B Mouse 120 Processing program for machine learning 122 Example image set 122A Example image 124 Annotator 126 Ground Truth data 128 training data 130A, 130A1 to 130A8, 160A, 160A1 to 160A8 subdivided range 139 lumen correspondence position 140 individual images 142 Model 144 Inference result 146 errors 148 Adjustment value 150 images 152 lumen detection processing 154 Confidence level information 156 map 158 Confidence level 162 Unit direction vector 164, 164A, 164B Direction vector 166 Vector sum 168 lumen existence range 169, 169A, 170 Marking 171 Identification 172 Outer contour line 173 line segment 176 external device 178 Network C1, C2 Middle CD1, CD2 Circumference CL center line CL1 to CL8 center line QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2003-093328A

[0002] JP 2003

[0002] JP 093328A

[0002] WO 2020 / 194472A

[0003]

Claims

[1] Medical support device, comprising: a processor where the processor is configured to: inputs a medical image, generated by imaging the interior of a lumen-containing organ, into a trained model to generate multiple confidence levels corresponding to multiple subdivided regions obtained by subdividing the medical image, or an image corresponding to the medical image, along a circumferential direction, and indicating that the lumen is shown in the multiple subdivided regions; and Lumen specification information for specifying a lumen existence area in which an existence position of the lumen is specified with higher accuracy than in the subdivided areas, in the medical image or the image corresponding to the medical image, based on the multiple subdivided areas and the multiple confidence levels. [2] Medical support device according to claim 1, wherein the lumen existence area is an area in which a position at which the lumen is shown in the medical image can be specified along the circumferential direction with a higher resolution than in the multiple subdivided areas. [3] Medical support device according to claim 1, wherein a direction from a reference position of the medical image or the image corresponding to the medical image to an existence position of each of the several subdivided areas is determined by several first vectors, a direction from the reference position to the lumen existence area is determined by a second vector, the second vector is a sum of at least two third vectors obtained by adding the confidence level as a weight to at least two first vectors among the multiple first vectors, and The lumen specification information is information that is determined based on the second vector. [4] Medical support device according to claim 1, wherein the processor is configured to output the medical image or the image corresponding to the medical image, and The lumen specification information will be updated in accordance with an output time of the medical image or the image corresponding to the medical image. [5] Medical support device according to claim 1, wherein the output of the lumen specification information is implemented by displaying the lumen specification information on a screen. [6] Medical support device according to claim 5, wherein the medical image or the image corresponding to the medical image is displayed on the screen, and The lumen specification information displayed on the screen is updated in accordance with a display time of the medical image or the image corresponding to the medical image. [7] Medical support device according to claim 5, wherein the medical image and / or the image corresponding to the medical image, as well as the lumen specification information, are displayed comparably on the screen. [8] Medical support device according to claim 7, wherein the lumen specification information is superimposed on the medical image and / or the image corresponding to the medical image. [9] Medical support device according to claim 1, wherein a direction from a reference position of the medical image or the image corresponding to the medical image to an existence position of each of the several subdivided areas is determined by several first vectors, a direction from the reference position to the lumen existence area is determined by a second vector, the second vector is a sum of at least two third vectors obtained by adding the confidence level as a weight to at least two first vectors among the multiple first vectors, and The lumen specification information includes a marker to specify an area which is determined as the lumen existence area based on the second vector in the medical image or the image corresponding to the medical image. [10] Medical support device according to claim 9, where one form of marking is an arc, and a center of the arc, a center of the medical image, or the image corresponding to the medical image. [11] Medical support device according to claim 9, wherein a form of marking is a form along an outer edge of the medical image or an outer edge of the image corresponding to the medical image. [12] Medical support device according to claim 9, where several hidden identifiers are assigned to the medical image or the image corresponding to the medical image, the medical image or the image corresponding to the medical image is displayed on a screen, and The marking on the screen is indicated by displaying at least one label that corresponds to a position of the lumen existence area among the multiple labels. [13] Endoscope system, comprising: the medical support device according to any one of claims 1 to 12; and an endoscope the medical image is generated by imaging the interior of the lumen organ, which contains the lumen, with the endoscope. [14] Medical support procedure, comprehensive: Inputting a medical image, generated by mapping the interior of a lumen-containing organ, into a trained model to generate multiple confidence levels corresponding to multiple subdivided regions obtained by subdividing the medical image, or an image corresponding to the medical image, along a circumferential direction, and indicating that the lumen is shown in the multiple subdivided regions; and Output of lumen specification information to specify a lumen existence area in which an existence position of the lumen is specified with higher accuracy than in the subdivided areas, in the medical image or the image corresponding to the medical image, based on the multiple subdivided areas and the multiple confidence levels. [15] Program that causes a computer to execute a process that includes: Inputting a medical image, generated by mapping the interior of a lumen-containing organ, into a trained model to generate multiple confidence levels corresponding to multiple subdivided regions obtained by subdividing the medical image, or an image corresponding to the medical image, along a circumferential direction, and indicating that the lumen is shown in the multiple subdivided regions; and Output of lumen specification information to specify a lumen existence area in which an existence position of the lumen is specified with higher accuracy than in the subdivided areas, in the medical image or the image corresponding to the medical image, based on the multiple subdivided areas and the multiple confidence levels.

Images