MEDICAL SUPPORT DEVICE, ENDOSCOPE SYSTEM, MEDICAL SUPPORT PROCEDURE AND PROGRAM

Abstract

A medical support device, an endoscope system, a medical support procedure, and a program are provided that enable a user or the like to determine the existence position of a lumen shown in multiple medical images arranged in a time series, in each medical image without omission. A processor (82) of a medical support device (24) is configured to: acquire a medical image (40); and selectively output several lumen specification information pieces (170). The medical image (40) is classified into a first medical image and a second medical image. The several lumen specification information pieces (170) comprise first lumen specification information and second lumen specification information. The first lumen specification information is generated based on information obtained from a trained model (92) in a case where the first medical image is input into the trained model (92) and is information capable of specifying a first existence position.The second lumen specification information is generated on the basis of information obtained from a time series model (94) in a case where time series information is input into the time series model (94) and is information that is able to specify a second existence position.

Description

BACKGROUND OF THE INVENTION 1. Field of the invention

[0001] The present disclosure relates to a medical support device, an endoscope system, a medical support procedure and a program. 2. Description of the state of the art

[0002] WO2020 / 194472A discloses a motion support system comprising a multi-operator information processing unit and a presentation information generation unit. In the motion support system disclosed in WO2020 / 194472A, the multi-operator information processing unit calculates multiple operator information sets, specifying multiple operations that differ in time and correspond to multiple operator target scenes—scenes in which the multiple operations that differ in time are required—based on captured images acquired by an imaging unit located in an insertion part. The presentation information generation unit generates presentation information for the insertion part based on the multiple operator information sets calculated by the multi-operator information processing unit.

[0003] Furthermore, the multiple operation information calculation unit calculates the multiple operation information via a trained model trained through machine learning, using as inputs the captured images corresponding to the multiple target operation scenes, which are scenes in which the multiple operations, differing in time, are required.

[0004] Furthermore, the multiple service information calculation unit calculates a probability of the multiple service information and presents information indicating that a certainty of the multiple service information is low in a case where the probability is lower than a threshold set in advance for the probability of the multiple service information. SUMMARY OF THE INVENTION

[0005] An embodiment according to the present disclosure provides a medical support device, an endoscope system, a medical support method and a program that enables a user or the like to determine the existence position of a lumen shown in several medical images arranged in a time series, in each medical image without omission.

[0006] A first aspect according to the present disclosure relates to a medical support device comprising: a processor configured to: acquire a medical image generated by imaging the interior of a luminal organ; and selectively output several lumen specification information sets, which are information capable of specifying the existence position of a lumen shown in the medical image, wherein the medical image is classified into a first medical image and a second medical image, which is acquired later than the first medical image, the several lumen specification information sets containing first lumen specification information and second lumen specification information, the first lumen specification information being based on information obtained from a trained model in a case.in which the first medical image is input into the trained model, information is generated that is capable of specifying a first existence position, which is the existence position of the lumen shown in the first medical image, and the second lumen specification information is generated based on information obtained from a time series model in a case in which time series information, which is information relating to one or more of the first lumen specification information obtained in time series, is input into the time series model, and is capable of specifying a second existence position, which is the existence position of the lumen shown in the second medical image.

[0007] A second aspect according to the present disclosure relates to the medical support device according to the first aspect, wherein the processor is configured to output the second lumen specification information in a case where the second medical image is an image that satisfies a faulty specification trigger condition, triggering the faulty specification of the second existence position by the trained model.

[0008] A third aspect according to the present disclosure relates to the medical support device according to the second aspect, wherein the multiple lumen specification information includes third lumen specification information, the processor is configured to output the third lumen specification information in a case where the second medical image is an image that does not satisfy the faulty specification trigger condition, and the third lumen specification information is generated based on information obtained from the trained model in a case where the second medical image is input into the trained model, and is information capable of specifying a third existence position, which is the existence position of the lumen shown in the second medical image.

[0009] A fourth aspect according to the present disclosure relates to the medical support device according to the first aspect, wherein the multiple lumen specification information includes third lumen specification information, the processor is configured to output the third lumen specification information in a case where the second medical image is an image that does not satisfy a faulty specification trigger condition, which triggers the faulty specification of the second existence position by the trained model, and the third lumen specification information is generated based on information obtained from the trained model in a case where the second medical image is input into the trained model, and is information capable of defining a third existence position, which is the existence position of the lumen shown in the second medical image.to specify in the second medical image.

[0010] A fifth aspect according to the present disclosure relates to the medical support device according to one of the second to fourth aspects, wherein the specification trigger condition includes a first condition in which at least a part of the lumen is not shown in the second medical image, a second condition in which at least a dark section that differs from the lumen is shown in the second medical image, a third condition in which an obstruction that obstructs the lumen is shown in the second medical image, a fourth condition in which an image quality triggers the faulty specification, and / or a fifth condition in which a section of the luminal organ that triggers the faulty specification is shown in the second medical image.

[0011] A sixth aspect according to the present disclosure relates to the medical support device according to one of the second to fifth aspects, wherein the trained model generates a confidence level indicating that the lumen is shown for each of several subdivided areas obtained by subdividing the second medical image or an image corresponding to the second medical image, in a case where the second medical image is input into the trained model, and the specification trigger condition includes a confidence level condition relating to the reliability of the confidence level for each of the several subdivided areas.

[0012] A seventh aspect according to the present disclosure relates to the medical support device according to the sixth aspect, wherein the confidence level condition includes a condition in which the confidence level for each of the several subdivided areas is equal to or greater than a second threshold that is less than a first threshold and less than the first threshold.

[0013] An eighth aspect according to the present disclosure relates to the medical support device according to one of the first to seventh aspects, wherein the first medical image or an image corresponding to the first medical image has several separation areas obtained by separating the first medical image or the image corresponding to the first medical image along a circumferential direction, the trained model generates several confidence levels corresponding to the several separation areas and indicating that the lumen is shown for each of the several separation areas, in a case where the first medical image or the image corresponding to the first medical image is input into the trained model, the first lumen specification information being information capable of determining the first existence position more precisely than the separation areas in the first medical image or the image.to specify that corresponds to the first medical image, and the first lumen specification information is generated based on the multiple separation ranges and the multiple confidence levels.

[0014] A ninth aspect according to the present disclosure relates to the medical support device according to the eighth aspect, wherein the first lumen specification information is information that is able to specify the first existence position with a higher resolution than the multiple separation areas along the circumferential direction.

[0015] A tenth aspect according to the present disclosure relates to the medical support device according to the eighth or ninth aspect, wherein a direction from a reference position of the first medical image or the image corresponding to the first medical image to an existence position of each of the multiple separation regions is determined by multiple first vectors, a direction from the reference position to a lumen existence region containing the first existence position is determined by a second vector, the second vector being a sum of at least two third vectors obtained by adding the confidence level as a weight to at least two first vectors among the multiple first vectors, and the first lumen specification information being information determined on the basis of the second vector.

[0016] An eleventh aspect according to the present disclosure relates to the medical support device according to one of the first to tenth aspects, wherein the second medical image, or an image corresponding to the second medical image, has a central region of the second medical image and several separation regions obtained by radially separating the surroundings of the central region in the second medical image, the trained model having a confidence level indicating that the lumen for the central region and each of the several separation regions is shown, in a case where the second medical image is input into the trained model, and in a case where the confidence level for each of the central region and the several separation regions is equal to or greater than a third threshold, or the confidence level for each of the central region and the several separation regions is less than a fourth threshold.If the value is smaller than the third threshold, information based on a processing result obtained by the trained model will be preferentially output over information based on a processing result obtained by the time series model.

[0017] A twelfth aspect according to the present disclosure relates to the medical support device according to one of the first to eleventh aspects, wherein the trained model generates a confidence level indicating that the lumen is shown in the first medical image, in a case where the first medical image is input into the trained model, and in a case where the confidence level is equal to or less than a reference value, continues to output the lumen specification information that was output before generating the confidence level that is equal to or less than the reference value.

[0018] A thirteenth aspect according to the present disclosure relates to the medical support device according to the twelfth aspect, wherein, in a case where a state in which the confidence level generated by the trained model is equal to or less than the reference value persists for a predetermined period each time the first medical image is input into the trained model, an output level of the lumen specification information output before the start of the predetermined period gradually decreases during the predetermined period.

[0019] A fourteenth aspect according to the present disclosure relates to the medical support device according to one of the first to thirteenth aspects, wherein the output of the lumen specification information is implemented by displaying the lumen specification information on a screen.

[0020] A fifteenth aspect according to the present disclosure relates to the medical support device according to the fourteenth aspect, wherein the medical image and / or an image corresponding to the medical image is displayed on the screen and the lumen specification information is displayed on the screen in a state comparable to the medical image or the image corresponding to the medical image displayed on the screen.

[0021] A sixteenth aspect according to the present disclosure relates to the medical support device according to the fifteenth aspect, wherein the lumen specification information is displayed superimposed on the medical image and / or the image corresponding to the medical image.

[0022] A seventeenth aspect according to the present disclosure relates to the medical support device according to the fifteenth or sixteenth aspect, wherein the lumen specification information displayed on the screen is updated in accordance with the display of the medical image and / or the image corresponding to the medical image.

[0023] An eighteenth aspect according to the present disclosure relates to an endoscope system comprising: the medical support device according to one of the first to seventeenth aspects; and an endoscope, wherein the medical image is produced by imaging the interior of the luminal organ with the endoscope.

[0024] A nineteenth aspect according to the present disclosure relates to a medical support procedure that includes: capturing a medical image generated by imaging the interior of a luminal organ; and selectively outputting multiple lumen specification information, which is information capable of specifying the existential position of a lumen shown in the medical image, wherein the medical image is classified into a first medical image and a second medical image, which is obtained later than the first medical image, the multiple lumen specification information comprising first lumen specification information and second lumen specification information, the first lumen specification information being based on information obtained from a trained model in a case.in which the first medical image is input into the trained model, information is generated that is capable of specifying a first existence position, which is the existence position of the lumen shown in the first medical image, and the second lumen specification information is generated based on information obtained from a time series model in a case in which time series information, which is information relating to one or more of the first lumen specification information obtained in time series, is input into the time series model, and is capable of specifying a second existence position, which is the existence position of the lumen shown in the second medical image.

[0025] A twentieth aspect according to the present disclosure relates to a program that causes a computer to perform medical support processing, which includes: acquiring a medical image produced by imaging the interior of a luminal organ; and selectively outputting multiple lumen specification information pieces, which are information capable of specifying the existential position of a lumen shown in the medical image, wherein the medical image is classified into a first medical image and a second medical image, which is acquired later than the first medical image, the multiple lumen specification information pieces containing first lumen specification information and second lumen specification information, the first lumen specification information being based on information obtained from a trained model in a case.in which the first medical image is input into the trained model, information is generated that is capable of specifying a first existence position, which is the existence position of the lumen shown in the first medical image, and the second lumen specification information is generated based on information obtained from a time series model in a case in which time series information, which is information relating to one or more of the first lumen specification information obtained in time series, is input into the time series model, and is capable of specifying a second existence position, which is the existence position of the lumen shown in the second medical image. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a concept diagram showing an example of a view where an endoscope system is used by a doctor. Fig. Figure 2 is a concept diagram showing an example of a complete configuration of the endoscope system. Fig. Figure 3 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope system. Fig. Figure 4 is a block diagram showing an example of the main functions of a processor intended for use in a medical support device and an example of information stored in a data storage device. Fig. Figure 5 is a block diagram showing an example of a hardware configuration of an electrical system of an information processing device. Fig. Figure 6 is a concept diagram showing an example of a view where training data is generated by the information processing device. Fig. Figure 7 is a concept diagram showing an example of a sample image. Fig. Figure 8 is a concept diagram that shows an example of the training data generated in a case where a lumen is located in an area outside a central region of the Fig. The example image shown in 7 shows. Fig. Figure 9 is a concept diagram that shows an example of the training data generated in a case where the lumen is in the middle of the Fig. The example image shown in 7 shows. Fig. Figure 10 is a concept diagram showing an example of processing content at the information processing device in a case where a lumen detection model is generated by training a model through machine learning using the training data. Fig. Figure 11 is a block diagram showing an example of the hardware configuration of the electrical system of the information processing device. Fig. Figure 12 is a concept diagram showing an example of processing content in the information processing device in a case where a time series model is constructed based on a time series data set. Fig. Figure 13 is a concept diagram showing an example of processing content in lumen detection processing performed by a detection unit. Fig. Figure 14 is a concept diagram showing an example of confidence-level information generated by the lumen detection model in a case where the lumen is shown in a single image. Fig. Figure 15 is a concept diagram showing an example of a relationship between multiple separation regions obtained by radially separating a map contained in the confidence level information and multiple direction unit vectors added to the multiple separation regions. Fig. Figure 16 is a concept diagram showing an example of processing content in a case where a lumen existence area is specified by a control of the medical support device and a marker capable of specifying the lumen existence area is generated. Fig. Figure 17 is a concept diagram showing an example of processing content using an Nth frame. Fig. Figure 18 is a concept diagram showing a view example in which, in a case where the lumen is shown in an area outside a central area of ​​the single image, the single image is displayed in a first display area of ​​a screen, a marker is able to specify a position of the lumen existence area in which an existence position of the lumen shown in the single image is specified, in which the single image is displayed superimposed on the single image, and visible information is displayed in a second display area of ​​the screen as part of auxiliary information. Fig. Figure 19 is a concept diagram showing a view example where, in a case where the lumen is shown in the central area of ​​the single image, the single image is displayed in the first display area of ​​the screen, the marker which is able to specify the position of the lumen existence area in which the existence position of the lumen shown in the single image is specified, in which the single image is displayed superimposed on the single image, and the visible information is displayed in the second display area of ​​the screen as part of the auxiliary information. Fig. Figure 20 is a concept diagram showing an example of processing content in the control in a case where each of the confidence levels added to all the separation areas of the map, which are included in the confidence level information, is equal to or less than a reference value. Fig. Figure 21 is a concept diagram showing a form example where the display of the marker continues in a case where the confidence level transitions to being equal to or less than the reference value. Fig. Figure 22 is a concept diagram showing a view example where the display intensity of the marker gradually decreases in a case where a state in which the confidence level is equal to or less than the reference value persists for a predetermined period. Fig. Figure 23 is a concept diagram showing an example of processing content in Lumen prediction processing performed by the detection unit. Fig. Figure 24 is a concept diagram showing a view example where the single image is displayed in the first display area of ​​the screen and the marker is generated based on state information obtained by performing the Lumen prediction processing and is displayed overlaid on the single image. Fig. Figure 25 is a concept diagram showing an example of processing content in the recognition unit in a case where a condition within the reliability range is met for the confidence level information obtained by feeding an (N+1)th single image into the lumen recognition model. Fig. Figure 26 is a flowchart showing an example of a processing flow for machine learning. Fig. Figure 27 is a flowchart showing an example of a model construction processing flow. Fig. 28A is a flowchart showing an example of a medical support processing flow. Fig. 28B is a continuation of the one in Fig. Flowchart shown in 28A. Fig. 29 is a modification example of the one in Fig. 28A and Fig. Flowchart shown in 28B. Fig. Figure 30 is a concept diagram showing an example of processing content in the control in a case where the marking is generated in units of the separation areas without using a vector sum. Fig. Figure 31 is a concept diagram showing a view example where, in a case where the separation area to which a maximum confidence level is added among all the confidence levels added to all the separation areas of the map is changed, the marker is displayed at a position corresponding to the separation area after the change, and the display of the marker before the change continues. Fig. Figure 32 is a concept diagram showing a modification example of the example image. Fig. Figure 33 is a concept diagram that shows an example of training data generated in a case where the lumen in a region outside a central region of the Fig. The example image shown in 32 illustrates this. Fig. Figure 34 is a concept diagram that shows an example of training data generated in a case where the lumen is in the midrange of the Fig. The example image shown in 32 illustrates this. Fig. Figure 35 is a concept diagram showing an example of processing content in the information processing device in a case where the lumen detection model is generated by training a model through machine learning using training data. Fig. Figure 36 is a concept diagram showing an example of processing content in the information processing device in a case where the time series model is based on a time series data set that differs from the one in Fig. The time series data set shown in section 12 is constructed. Fig. 37 is a concept diagram that shows a modification example of the in Fig. The 17 processing contents shown are displayed. Fig. 38 is a concept diagram that shows a modification example of the one in Fig. The 20 processing contents shown are displayed. Fig. 39 is a concept diagram that shows a modification example of the one in Fig. The 23 processing contents shown are displayed. Fig. 40 is a concept diagram that shows a modification example of the in Fig. The 25 displayed processing contents are shown. Fig. Figure 41 is a concept diagram showing a form example where the marker is displayed along an outer edge of the single image shown on the screen. Fig. Figure 42 is a concept diagram showing a form example in which the marking is displayed superimposed on the single image by displaying at least one label corresponding to the position of the lumen existence area under several hidden labels associated with the map and / or the single image. Fig. Figure 43 is a concept diagram showing a form example in which an outer contour line, which is a line outlining an outer contour of the lumen existence area, is displayed superimposed on the single image. Fig. Figure 44 is a concept diagram illustrating an example of a series of processing operations in which a processor contained within a computer issues a processing execution request to an external device via a network, the external device performs processing in response to the processing execution request, and the processor contained within the computer receives a processing result from the external device. DESCRIPTION OF PREFERRED EXECUTION FORMS

[0026] Examples of embodiments of a medical support device, an endoscope system, a medical support method, and a program according to the present disclosure are described below with reference to the accompanying drawings. It should be noted that the present disclosure also applies to a program and a computer program product.

[0027] First, the terms used in the following description will be described.

[0028] CPU (central processing unit) is an abbreviation for "central processing unit". GPU (graphics processing unit) is an abbreviation for "graphics processing unit". GPGPU (general-purpose computing on graphics processing units) is an abbreviation for "general-purpose computing on graphics processing units". APU (accelerated processing unit) is an abbreviation for "accelerated processing unit". TPU is an abbreviation for "Tensor Processing Unit". RAM (random-access memory) is an abbreviation for "random access memory". ASIC (application-specific integrated circuit) is an abbreviation for "application-specific integrated circuit". PLD (programmable logic device) is an abbreviation for "programmable logic circuit". FPGA is an abbreviation for "field-programmable gate array". SoC is an abbreviation for "system-on-a-chip". SSD (solid-state drive) is an abbreviation for "solid-state drive". USB is an abbreviation for "Universal Serial Bus".EL is an abbreviation for "electroluminescence". CMOS (complementary metal-oxide semiconductor) is an abbreviation for "complementary metal-oxide semiconductor". CCD (charge-coupled device) is an abbreviation for "charge-coupled device". AI is an abbreviation for "artificial intelligence". WLI is an abbreviation for "white light imaging". BLI is an abbreviation for "blue light imaging". LCI is an abbreviation for "linked color imaging". NBI is an abbreviation for "narrow-band imaging". CT is an abbreviation for "computed tomography". MRI is an abbreviation for "magnetic resonance imaging". I / F is an abbreviation for "interface". LAN (local area network) is an abbreviation for "local area network". WAN (wide area network) is an abbreviation for "wide area network". 5G is an abbreviation for "5th generation mobile communications system".

[0029] Hereinafter, a processor with a reference symbol (hereinafter simply referred to as a "processor") can be a computing device or a combination of several computing devices. Furthermore, the processor can be a type of computing device or a combination of several types of computing devices. Examples of computing devices include a CPU, a GPU, a GPGPU, an APU, and a TPU.

[0030] Below, a memory with a reference symbol is a memory such as RAM that temporarily stores information and is used by the processor as working memory.

[0031] Below, a data storage device, indicated by a reference symbol, is one or more non-volatile storage devices that store various programs, parameters, and the like. Examples of non-volatile storage devices include flash memory, magnetic disks, and magnetic tape. Another example of data storage is cloud storage.

[0032] In the following embodiment, an external interface, identified by a reference numeral, controls the transmission and reception of various types of information between multiple interconnected devices. Examples of the external interface include a USB interface. A communication interface, containing a communication processor, an antenna, and the like, can be applied to the external interface. The communication interface controls communication between multiple computers. Examples of a communication standard applied to the communication interface include a wireless communication standard, such as 5G, Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0033] In the following embodiment, “A and / or B” is synonymous with “at least one of A and B”. That is, “A and / or B” may possibly mean only A, only B, or a combination of A and B. In the present description, the same concept as “A and / or B” is applied to a case in which the connection of three or more matters is expressed by “and / or”.

[0034] Fig. Figure 1 is a concept diagram showing an example of a view using an endoscope system 10. As shown in Fig. As shown in Figure 1, the endoscope system 10 is used by a physician 12 during an endoscopy and the like. A staff member 14, such as a nurse, assists with the endoscopy.

[0035] The endoscope system 10 is communicatively connected to a communication device (not shown), and information received by the endoscope system 10 is transmitted to the communication device. Examples of the communication device include a server, a personal computer, and / or a tablet terminal that manage various types of information, such as electronic health records. The communication device receives the information transmitted by the endoscope system 10 and performs processing using the received information (for example, processing and storing the information in the electronic health record, or the like).

[0036] The endoscope system 10 comprises an endoscope 16, a display device 18, a light source device 20, a control device 22, and a medical support device 24. In the present embodiment, the endoscope system 10 is an example of an “endoscope system” according to the present disclosure, the endoscope 16 is an example of an “endoscope” according to the present disclosure, and the medical support device 24 is an example of a “medical support device” according to the present disclosure.

[0037] The endoscope system 10 is a modality for performing a medical examination on a large intestine 28, which is a luminal organ contained in the body of an examinee 26 (for example, a patient), by using the endoscope 16. In the present embodiment, the large intestine 28 is an object to be observed by the physician 12.

[0038] The endoscope 16 is used by the physician 12 and inserted into the body of the subject 26. In the present embodiment, the endoscope 16 is inserted into the large intestine 28 of the subject 26. In the present embodiment, the large intestine 28 is an example of a "luminal organ" as defined in the present disclosure.

[0039] The endoscope system 10 forms an interior of the colon 28 including a lumen 42 by using the endoscope 16, which is inserted into the colon 28 of the subject 26, and performs various medical treatments on the colon 28 as required.

[0040] The colon 28 has a lumen 42. The endoscope 16 is inserted into the lumen 42. The position of the lumen 42 in the colon 28 can be medically specified based on a shape pattern of multiple folds 43 (for example, the shape, orientation, and the like of the multiple folds 43), which are characteristic regions in the colon 28. In the present embodiment, as described in detail below, the position of the lumen 42 is recognized by AI trained by machine learning using various types of information, such as the shape pattern of the multiple folds 43, and the result of the recognition is provided to the physician 12 as visually identifiable information. In the present embodiment, the lumen 42 is an example of a “lumen” as defined in the present disclosure.

[0041] The endoscope system 10 acquires an image showing a view containing the lumen 42 in the colon 28 by imaging the interior of the colon 28 containing the lumen 42 and outputs the acquired image. In the present embodiment, the endoscope system 10 has an optical imaging function of emitting light 30 in the colon 28 and imaging reflected light obtained by reflection off an intestinal wall 32 of the colon 28.

[0042] Here, the endoscopy of the large intestine 28 was described as an example, but this is only an example and the present disclosure is applicable to an endoscopy of a luminal organ, such as an esophagus, a stomach, a duodenum or a trachea.

[0043] The light source device 20, the control device 22, and the medical support device 24 are installed in a trolley 34. The trolley 34 is equipped with several tables arranged in an up-down direction, and the medical support device 24, the light source device 20, and the control device 22 are installed from a lower table to an upper table. Furthermore, the display device 18 is installed on the uppermost table of the trolley 34.

[0044] The control device 22 controls the entire endoscope system 10. The control device 22 performs various types of processing on the image obtained by imaging the intestinal wall 32 with the endoscope 16. Furthermore, the medical support device 24, under the control of the control device 22, performs AI-based processing or the like on the image that has undergone various types of processing by the control device 22 and outputs various types of information containing a processing result of the AI-based processing or the like.Examples of output destinations for the various types of information include the display device 18, a stationary storage medium (for example, a data storage device mounted on the endoscope system 10, a data storage device of a server or the like that communicatively connected to the endoscope system 10, and the like) and / or a portable storage medium (for example, a memory card, a USB flash drive, and the like).

[0045] The display device 18 shows various types of information (for example, different types of information output by the medical support device 24). Examples of the display device 18 include a liquid crystal display and an EL display. A tablet terminal equipped with a display can be used instead of or in conjunction with the display device 18.

[0046] The display device 18 displays a screen 35. The screen 35 contains several display areas. The several display areas are arranged on the screen 35. In the Fig. In the example shown, a first display area 35A and a second display area 35B are shown as examples of multiple display areas. The first display area 35A is larger than the second display area 35B. The first display area 35A is used as a primary display area, and the second display area 35B is used as a secondary display area. The size relationship between the first display area 35A and the second display area 35B is not limited to this and only needs to be a size relationship that can be accommodated within the screen 35.

[0047] An endoscopy video image 39 is displayed in the first display area 35A. The endoscopy video image 39 is obtained by performing various types of processing on several images arranged in time series, which are obtained by imaging the interior of the colon 28 of the subject 26 with the endoscope 16. The intestinal wall 32 shown in the endoscopy video image 39 contains the lumen 42 as a region of interest (that is, a target area for observation) which the physician 12 is looking at, and the physician 12 can visually perceive the view of the intestinal wall 32 containing the lumen 42 through the endoscopy video image 39.

[0048] The image displayed in the first display area 35A is a single frame 40 contained within a video image that contains multiple frames 40 arranged in a time series. That is, the multiple frames 40 arranged in a time series are displayed in the first display area 35A at predetermined frame rates (for example, one dozen frames per second or several dozen frames per second). The single frame 40 in the present embodiment is an example of a "medical image" as described in the present disclosure.

[0049] Examples of the video image displayed in the first display area 35A include a video image in a live view mode. The live view mode is merely an example, and the video image could be a video image, such as a video image in a post-view mode, that is temporarily stored in memory or the like and then displayed. Furthermore, each frame contained in a video image for recording, stored in memory or the like, can be reproduced and displayed as the endoscopy video image 39 on screen 35 (for example, in the first display area 35A).

[0050] The second display area 35B is shown in the lower right corner of the screen 35 in a front view. The second display area 35B can be displayed in any position as long as the position is within the screen 35 of the display device 18, but is preferably displayed in a position comparable to the endoscopy video image 39. Auxiliary information 44 to assist the physician 12 in a medical determination or the like is displayed in the second display area 35B. The auxiliary information 44 is information to which the physician 12 is to refer. Examples of the auxiliary information 44 include various types of information about the patient 26 into whom the endoscope 16 is inserted and / or various types of information obtained by performing medical support processing, which will be described later.

[0051] Fig. Figure 2 is a concept diagram showing an example of a complete configuration of the endoscope system 10. As shown in Fig. As shown in Figure 2, the endoscope 16 comprises an operating part 46 and an insertion part 48. The insertion part 48 is partially curved by the operation of the operating part 46. The insertion part 48 is inserted into the large intestine 28, following the shape of the large intestine 28 (see Figure 2). Fig. 1) in accordance with the operation of the control unit 46 performed by the physician 12 (see Fig. 1) is curved.

[0052] A camera 52, a lighting device 54, and a treatment tool opening 56 are provided at a distal end section 50 of the insertion part 48. A section of the camera 52 (for example, an imaging optic) and a section of the lighting device 54 (for example, an optical irradiation system) are exposed by a distal end surface 50A of the distal end section 50.

[0053] The camera 52 is mounted in the endoscope 16 and is inserted into a body cavity (here, for example, the lumen 42) of the subject 26 to image the observation target area. Examples of the camera 52 include a CMOS camera. However, this is only an example, and the camera 52 can be other types of cameras, such as CCD cameras. In the present embodiment, the camera 52 generates an image showing the view contained by the lumen 42 in the colon 28 by imaging the interior of the colon 28 containing the lumen 42. The image generated by the camera 52 is a circular image. For example, the image generated by the camera 52 is processed by the control device 22 into a shape in which an upper end section and a lower end section are masked. Accordingly, as in Fig. As shown in Figure 1, an image in which the upper end edge and the lower end edge are linear and the left side edge and the right side edge are arc-shaped is produced as the single image 40.

[0054] The illumination device 54 includes illumination windows 54A and 54B. The illumination windows 54A and 54B are provided on the distal end surface 50A. The illumination device 54 emits the light 30 (see Fig. 1) through the illumination windows 54A and 54B. Examples of the type of light 30 emitted by the illumination device 54 include light for WLI (for example, white light), light for LCI (for example, light obtained by combining red, green, and blue light), light for BLI (for example, blue light), and / or light for NBI (for example, light obtained by combining blue and green light). The camera 52 images the interior of the colon 28 using an optical method in a state in which the illumination device 54 illuminates the interior of the colon 28 with the light 30 (see Fig. 1) irradiated.

[0055] The treatment tool opening 56 is an opening to allow a treatment tool 58 to protrude from the distal end section 50. Furthermore, the treatment tool opening 56 also serves as a suction port for aspirating blood, internal contaminants, and the like, and as an ejection port for expelling fluid.

[0056] A treatment tool insertion port 60 is formed on the operating part 46, and the treatment tool 58 is inserted through the treatment tool insertion port 60 into the insertion part 48. The treatment tool 58 passes through the insertion part 48 to protrude outwards from the treatment tool opening 56. In the Fig. In the example shown, a view is depicted in which a biopsy needle, as the treatment tool 58, protrudes through the treatment tool opening 56. Here, the biopsy needle has been described as an example of the treatment tool 58, but this is merely an example, and the treatment tool 58 could be grasping forceps, a papillotomy knife, a snare, a catheter, a guidewire, a cannula, and / or a biopsy needle with a guide sheath.

[0057] The endoscope tube 16 is connected to the light source device 20 and the control device 22 via a universal cable 62. The medical support device 24 and a receiver device 64 are connected to the control device 22. Furthermore, the display device 18 is connected to the medical support device 24. That is, the control device 22 is connected to the display device 18 via the medical support device 24.

[0058] Since the medical support device 24 is used here as an example of an external device for extending the functions of the control device 22, the form example described shows the control device 22 and the display device 18 being indirectly connected to each other via the medical support device 24, but this is only an example. For instance, the display device 18 could be directly connected to the control device 22.In this case, for example, the functions of the medical support device 24 only need to be implemented in the control device 22, or the control device 22 only needs to have a function of instructing a server (not shown) to perform the same processing as the processing (for example, the medical support processing 24, which is described later), receiving a processing result by the server, and using the processing result.

[0059] The receiving device 64 receives an instruction from the physician 12 and outputs the received instruction as an electrical signal to the control device 22. Examples of the receiving device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and / or a remote control device.

[0060] The control device 22 controls the light source device 20, transmits and receives various signals to and from the camera 52 or transmits and receives various signals to and from the medical support device 24.

[0061] The light source device 20 emits light under the control of the control device 22 to supply the light 30 to the illumination device 54. A light guide is provided in the illumination device 54, and the light 30 supplied by the light source device 20 is emitted through the light guide from the illumination windows 54A and 54B. The control device 22 causes the camera 52 to perform imaging in a state in which the light 30 is emitted from the illumination windows 54A and 54B. The control device 22 generates the multiple individual images 40 arranged in time sequences by processing the external shape of the image obtained by the imaging performed by the camera 52 or by adjusting the image quality or the like.The control device 22 outputs the endoscopy video image 39, which contains the several generated individual images 40 arranged in a time series, to a predetermined output destination (for example, the medical support device 24).

[0062] The medical support device 24 performs various types of processing on the endoscopy video image 39 input by the control device 22 in order to support a medical treatment (here, for example, endoscopy). The medical support device 24 outputs the endoscopy video image 39, which has undergone various types of processing, to a predetermined output destination (for example, the display device 18).

[0063] Here, the method in which the endoscopy video image 39 output by the control device 22 is output to the display device 18 via the medical support device 24 has been described, but this is only an example. For instance, an aspect can be used in which the control device 22 and the display device 18 are connected to each other and the endoscopy video image 39, which has undergone various types of processing by the medical support device 24, is displayed on the display device 18 via the control device 22.

[0064] Fig. Figure 3 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope system 10. As shown in Fig. As shown in Figure 3, the control device 22 comprises a computer 66, a bus 68, and an external interface 70. The computer 66 comprises a processor 72, a memory 74, and a data storage 76. The processor 72, the memory 74, the data storage 76, and the external interface 70 are connected to the bus 68. The processor 72 controls the entire control device 22. The memory 74 and the data storage 76 are used by the processor 72.

[0065] The external interface 70 transmits and receives various types of information between one or more devices (hereinafter also referred to as "first external devices") that exist outside the control device 22 and the processor 72.

[0066] The camera 52 is one of the first external devices connected to the external interface 70, and the external interface 70 transmits and receives various types of information between the camera 52 and the processor 72. The processor 72 controls the camera 52 via the external interface 70. In addition, the processor 72 captures an image by imaging the interior of the large intestine 28 (see Fig. 1) is generated with the camera 52, via the external interface 70 and performs various processing operations on the captured image to create the endoscopy video image 39 (see Fig. 1) to generate.

[0067] The light source device 20 is one of the first external devices connected to the external interface 70, and the external interface 70 transmits and receives various types of information between the light source device 20 and the processor 72. The light source device 20 supplies the light 30 to the lighting device 54 under the control of the processor 72. The lighting device 54 emits the light 30 supplied by the light source device 20.

[0068] The receiving device 64 is one of the first external devices connected to the external interface 70, and the processor 72 detects the instruction received by the receiving device 64 via the external interface 70 and performs the processing that corresponds to the detected instruction.

[0069] The medical support device 24 comprises a computer 78 and an external interface 80. The computer 78 comprises a processor 82, a memory 84, and a data storage device 86. The processor 82, the memory 84, the data storage device 86, and the external interface 80 are connected by a bus 88. In the present embodiment, the computer 78 is an example of a "computer" according to the present disclosure, and the processor 82 is an example of a "processor" according to the present disclosure.

[0070] It should be noted that the hardware configuration (that is, the processor 82, the memory 84 and the data storage 86) of the Computer 78 is essentially the same as the hardware configuration of the Computer 66, and therefore the description of the hardware configuration of the Computer 78 is omitted here.

[0071] The external interface 80 transmits and receives various types of information between one or more devices (hereinafter also referred to as "second external devices") that exist outside the medical support device 24 and the processor 82.

[0072] As one of the second external devices, the control device 22 is connected to the external interface 80. In the case of the Fig. In the example shown, the external interface 70 of the control device 22 is connected to the external interface 80. The external interface 80 transmits and receives various types of information between the processor 82 of the medical support device 24 and the processor 72 of the control device 22. For example, the processor 82 acquires the endoscopy video image 39 (see Figure 3). Fig. 1) from the processor 72 of the control device 22 via the external interfaces 70 and 80 and performs various types of processing on the captured endoscopy video image 39. The various types of processing performed by the processor 82 include AI-based processing (for example, processing using a lumen detection model 92 described later).

[0073] As one of the second external devices, the display device 18 is connected to the external interface 80. The processor 82 controls the display device 18 via the external interface 80 so that different types of information (for example, the endoscopy video image 39, which has undergone different types of image processing) are displayed on the display device 18.

[0074] Fig. Figure 4 is a block diagram showing an example of the main functions of the processor 82 contained in the medical support device 24, and an example of the information stored in the data memory 86. As shown in Fig. As shown in Figure 4, a medical support program 90 is stored in the data storage device 86. The medical support program 90 in the present embodiment is an example of a “program” according to the present disclosure.

[0075] The processor 82 reads the medical support program 90 from the data memory 86 and executes the read medical support program 90 on the memory 84 to perform medical support processing. The medical support processing is implemented by the processor 82, which acts as a detection unit 82A and a control unit 82B in accordance with the medical support program 90 executed on the memory 84.

[0076] The lumen detection model 92 and a time series model 94 are stored in data memory 86. As will be described in detail later, the lumen detection model 92 is a machine learning model and is used by the detection unit 82A. Examples of the machine learning model include a neural network. The time series model 94 is a probabilistic model and is also used by the detection unit 82A. Examples of the probabilistic model include a Markov model.

[0077] A first example of a Markov model is a Markov model based on a Markov chain. A Markov chain is a probabilistic model that possesses a Markov property where a current state influences a subsequent state, and which predicts the subsequent state based on a probability of state transition. Examples of Markov chains include a simple Markov chain and an Nth-order Markov chain. Examples of simple Markov chains include a discrete-time Markov chain and a continuous-time Markov chain. A second example of a Markov model is a Markov model based on a hidden Markov model.

[0078] The Markov model is merely an example, and instead of the Markov model, a time series model incorporating a deep learning model (for example, a deep learning model based on a convolutional neural network, a recurrent neural network, a transformer, or the like) or a time series analysis mechanism (for example, a time series unit based on an autoregressive model, a moving average model, an autoregressive moving average model, an integrated autoregressive moving average model, a conditional variance model, or the like) can be used.

[0079] Furthermore, the time series model 94 can be a time series model that incorporates two or more of the Markov model, the deep learning model, or the time series analysis mechanism.

[0080] In the present embodiment, the lumen detection model 92 is an example of a “trained model” according to the present disclosure, and the time series model 94 is an example of a “time series model” according to the present disclosure.

[0081] Fig. Figure 5 is a block diagram showing an example of a hardware configuration of an electrical system of an information processing device 100, which is used to generate the lumen detection model 92 and the time series model 94. As shown in Fig. As shown in Figure 5, the information processing device 100 comprises a computer 102 and an external interface 104. The computer 102 comprises a processor 106, a memory 108, and a data storage device 110. The processor 106, the memory 108, the data storage device 110, and the external interface 104 are connected by a bus 112.

[0082] It should be noted that the hardware configuration (that is, the processor 106, the memory 108 and the data storage 110) of computer 102 is essentially the same as the hardware configuration of computer 66, and therefore the description of the hardware configuration of computer 102 is omitted here.

[0083] The information processing device 100 comprises a receiving device 116. The receiving device 116 is, for example, a keyboard and / or a mouse and receives an instruction from a user of the information processing device 100, and so on. The receiving device 116 is connected to the bus 112. The processor 106 detects the instruction received from the receiving device 116 and operates in accordance with the detected instruction.

[0084] A display device 118 shows various types of information, including images. Examples of the display device 118 include a liquid crystal display and an EL display. The display device 118 is connected to the bus 112. The processor 106 displays the results obtained by performing various types of processing on the display device 118.

[0085] The external interface 104 transmits and receives various types of information between one or more devices (hereinafter also referred to as "third external devices") that exist outside the information processing device 100 and the processor 106. The medical support device 24 is connected to the external interface 104 as one of the third external devices. Fig. In the example shown in Figure 5, the external interface 80 of the medical support device 24 is connected to the external interface 104. The external interface 104 controls the transmission and reception of various types of information between the processor 82 (see Figure 5). Fig. 3 and Fig. 4) the medical support device 24 and the processor 106 of the information processing device 100. For example, the information processing device 100 generates the lumen detection model 92 and the time series model 94 and transmits the generated lumen detection model 92 and the time series model 94 to the medical support device 24 via the external interfaces 80 and 104 in response to a request from the medical support device 24.

[0086] A machine learning processing program 120 is stored in data memory 110. Processor 106 reads the machine learning processing program 120 from data memory 110 and executes the read program on memory 108 to perform machine learning processing. The machine learning processing is implemented by processor 106, which acts as a training data generation unit 106A and a learning execution unit 106B in accordance with the machine learning processing program 120 executed on memory 108.

[0087] A sample image set 122 is stored in data memory 110. As described in detail below, the sample image set 122 is used by the training data generation unit 106A.

[0088] Fig. Figure 6 is a concept diagram showing an example of processing content at the training data generation unit 106A. As shown in Fig. As shown in Figure 6, the information processing device 100 is used by an annotator 124. The annotator 124 is an operator who adds annotations for machine learning to given data (that is, an operator who performs labeling).

[0089] In the Fig. In the example shown, a keyboard 116A and a mouse 116B are shown as examples of the receiving device 116. The annotator 124 issues an instruction to the computer 102 via the keyboard 116A and the mouse 116B.

[0090] The example image set 122 contains several example images 122A, each showing different content. Example image 122A is a pre-defined medical image intended for use in object recognition processing (for example, processing where the recognition unit 82A detects the lumen 42 based on the single image 40 and the lumen recognition model 92). The image pre-defined as the medical image to be used in object recognition processing is an image corresponding to the single image 40. In other words, the image corresponding to the single image 40 can also be described as an image representing the single image 40. In other words, the image representing the single image 40 can also be described as an image showing a sample image of the single image 40.Here is a first example of the image showing the pattern image of single image 40, an image obtained by actually imaging the inside of the colon with the camera. A second example of the image showing the pattern image of single image 40 is a virtually generated image (for example, an image generated by generative AI such as Stable Diffusion or Midjourney).

[0091] The training data generation unit 106A acquires the example image 122A from the example image set 122 in response to the instruction received by the receiver 116. The training data generation unit 106A displays the example image 122A on a screen 118A of the display device 118. When the example image 122A is displayed on the screen 118A, the annotator 124 specifies a lumen correspondence position, which is the position of the lumen shown in the example image 122A, relative to the training data generation unit 106A via the receiver 116. The training data generation unit 106A assigns ground truth data 126 to the example image 122A on the basis of the lumen correspondence position specified via the receiving device 116 in order to generate training data 128.The mapping of the Ground Truth data 126 to the example image 122A is implemented by adding a label that is able to specify the lumen correspondence position as the Ground Truth data 126 to the lumen correspondence position in the example image 122A.

[0092] In this way, the training data generation unit 106A repeatedly performs the processing of mapping the ground truth data 126 to each of the sample images 122A contained in the sample image set 122 in response to the instruction issued by the annotator 124 in order to generate multiple training data 128.

[0093] Fig. Figure 7 is a concept diagram showing an example of a composition of example image 122A. As in Fig. As shown in figure 7, a large intestine 132 is shown in example image 122A. In the Fig. The example shown in Figure 7 shows an intestinal wall 136 in which several folds 134 are formed and a lumen 138 in the example image 122A.

[0094] Example image 122A is subdivided into several separation regions 130A. Eight separation regions 130A1 to 130A8 are contained within the multiple separation regions 130A. The separation regions 130A1 to 130A8 are regions that extend radially from a center C1 of example image 122A to an outer edge of example image 122A and are arranged along a circumferential direction CD1 (in other words, around the center C1) of example image 122A.

[0095] Fig. 8 and Fig. Figure 9 are concept diagrams showing an example of a procedure in which the training data generation unit 106A maps the ground truth data 126 to the example image 122A to generate the training data 128.

[0096] As in Fig. As shown in Figure 8, in a state where example image 122A is displayed on screen 118A, the annotator 124 specifies a lumen correspondence position 139, which is the position of the lumen 138 shown in example image 122A, relative to the training data generation unit 106A via the receiving device 116. The training data generation unit 106A displays a circular frame 140 superimposed on example image 122A in response to the instruction received from the receiving device 116 and positions the frame 140 at a location surrounding the lumen 138 shown in example image 122A. The frame 140 is a marker that defines the lumen correspondence position 139 in example image 122A. That is, the position of an area surrounded by the single image 140 in the example image 122A is the lumen correspondence position 139.The size and position of frame 140 on screen 118A are freely changed in response to the instruction received from receiver 116. Here, frame 140 has a circular shape, but it can be a shape other than a circle. The size of frame 140 can be changed in response to the instruction received from receiver 116.

[0097] The annotator 124 issues an acknowledgment instruction, which is an instruction to confirm the lumen correspondence position 139, to the training data generation unit 106A via the receiving device 116 in a state in which the single image 140 is positioned at the location surrounding the lumen 138. As a result, the training data generation unit 106A confirms the lumen correspondence position 139.

[0098] The training data generation unit 106A specifies the separation area 130A, which has a maximum overlap area with the single image 140 that defines the lumen correspondence position 139 among the multiple separation areas 130A. The training data generation unit 106A then generates the training data 128 by combining the ground truth data 126 with the specified separation area 130A (in which in Fig. 8 shown example, the separation area 130A2) is linked as the label that is able to specify the separation area 130A in which the lumen 138 is shown.

[0099] Fig. Figure 8 shows an example of a procedure for generating the training data 128 in a case where the lumen 138 is shown in an area outside the central area in the example image 122A, while Fig. Figure 9 shows an example of a procedure for generating the training data 128 in a case where the lumen 138 is shown in the central region in the example image 122A. As in Fig. As shown in Figure 9, in a case where the lumen 138 is shown in the central area in the example image 122A, the training data generation unit 106A generates the training data 128 by linking the ground truth data 126 with each of the separation areas 130A (that is, the separation areas 130A1 to 130A8).

[0100] Fig. Figure 10 is a concept diagram showing an example of a view in which the learning execution unit 106B performs machine learning using the training data 128 to generate the lumen detection model 92. As shown in Fig. Figure 10 shows that in the information processing device 100, the learning execution unit 106B acquires the training data 128 generated by the training data generation unit 106A. The learning execution unit 106B performs the machine learning using the training data 128.

[0101] At the in Fig. In the example shown, the learning execution unit 106B contains a model 142. Examples of model 142 include a neural network. Examples of the neural network include a convolutional neural network. The learning execution unit 106B inputs the example image 122A, which is contained in the training data 128, to the model 142. In a case where the example image 122A is input, the model 142 performs an inference to output an inference result 144. The learning execution unit 106B computes an error 146 between the inference result 144 and the ground-truth data 126, which is contained in the training data 128.

[0102] The learning execution unit 106B calculates several fitting values ​​148 to minimize the error 146. Then, the learning execution unit 106B adjusts several optimization variables in the model 142 using the several fitting values ​​148 to optimize the model 142. For example, the several optimization variables represent several coupling weights and several offset values ​​included in the model 142.

[0103] The learning execution unit 106B repeatedly performs the learning process of inputting the example image 122A into the model 142, calculating the error 146, calculating the multiple fit values ​​148, and adjusting the multiple optimization variables in the model 142 using the multiple data sets 128. That is, the learning execution unit 106B adjusts the multiple optimization variables in the model 142 using the multiple fit values ​​148, which were calculated such that the error 146 is minimized for each of the multiple example images 122A contained in the multiple training data sets 128, in order to optimize the model 142. The lumen detection model 92 is generated by optimizing the model 142 in this way. The lumen detection model 92 is transmitted from the information processing device 100 to the medical support device 24 via the external interfaces 80 and 104 (see Fig. 5) is transmitted and received by the medical support device 24. Then, the lumen detection model 92 is stored in the data memory 86 by the processor 82 of the medical support device 24 (see Fig. 4) The lumen detection model 92 stored in the data memory 86 is used by the detection unit 82A (see Fig. 4).

[0104] As in Fig. As shown in Figure 11 as an example, a model design processing program 150 is stored in the data memory 110 of the information processing device 100. The processor 106 reads the model design processing program 150 from the data memory 110 and executes the read model design processing program 150 on the memory 108 to perform model design processing. The model design processing is implemented by the processor 106, which operates as a model design unit 106C in accordance with the model design processing program 150 executed by the processor 106 on the memory 108.

[0105] A time series data set 152A is stored in the data storage unit 110. As will be described in detail later, the time series data set 152A is used by the model construction unit 106C.

[0106] Fig. Figure 12 is a concept diagram showing an example of processing content where the Markov model is constructed based on the Markov chain as the time series model 94 by the model construction unit 106C. As in Fig. As shown in Figure 12, the time-series dataset 152A contains multiple time-series images 152A1 arranged in a time series. These multiple time-series images 152A1 are multiple images corresponding to the multiple single images 40 arranged in a time series and contained within the endoscopy video image 39 (that is, multiple images representing the multiple single images 40 arranged in a time series). A first example of the multiple time-series images 152A1 is multiple images obtained in a time series by actually imaging the interior of the colon with the camera. A second example of the multiple time-series images 152A1 is multiple images (for example, images generated by generative AI, such as stable diffusion or midjourney) that were virtually created as multiple images showing samples of the multiple single images 40 arranged in a time series.

[0107] Similar to example image 122A, the interior of the large intestine 132 is shown in time series image 152A1. Furthermore, time series image 152A1 is divided into several separation regions 130B using the same procedure as example image 122A. In the Fig. The 12 examples shown are the multiple separation areas 130B, specifically separation areas 130B1 to 130B8. The geometric properties of separation areas 130B1 to 130B8 in the time series image 152A1 correspond to the geometric properties of separation areas 130A1 to 130A8 in the example image 122A. Furthermore, using the same procedure as in the case of assigning ground truth data 126 to each example image 122A, ground truth data 152A2, corresponding to ground truth data 126, is also assigned to each time series image 152A1.

[0108] The model construction unit 106C acquires several ground truth data 152A2 in time series, which are assigned to the several time series images 152A1 arranged in time series, and counts a transition count 154 between the respective states in which the lumen 138 is shown in the separation area 130A in the several time series images 152A1 arranged in time series, from the several ground truth data 152A2 arranged in time series. Each state in which the lumen 138 is shown in the separation area 130A is specified from the ground truth data 152A2 linked to the several time series images 152A1.

[0109] Examples of the state in which the lumen 138 is shown in the separation range 130A comprise eight states, that is, states one through eight. The first state represents a state in which the lumen 138 is shown in separation range 130A1. The second state represents a state in which the lumen 138 is shown in separation range 130A2. The third state represents a state in which the lumen 138 is shown in separation range 130A3. The fourth state represents a state in which the lumen 138 is shown in separation range 130A4. The fifth state represents a state in which the lumen 138 is shown in separation range 130A5. The sixth state represents a state in which the lumen 138 is shown in separation range 130A6. The seventh state represents a state in which the lumen 138 is shown in separation range 130A7. The eighth state signifies a state in which the lumen 138 is shown in the separation region 130A8.The transition number 154 means the number of transitions among the first to eighth states (in other words, the number of times the separation area 130A, in which the lumen 138 is shown, transitions from one separation area 130A to another separation area 130A among the eight separation areas 130A1 to 130A8).

[0110] The model construction unit 106C counts the transition count 154 by pairing a state in an Nth time series image 152A1 with a state in an (N+1)th time series image 152A1 and sums the transition count 154 among the first to eighth states. That is, the model construction unit 106C sums the number of times a transition occurs from a specific state to the next state among the first to eighth states as the transition count 154.

[0111] The model construction unit 106C calculates a transition probability 156 based on the transition number 154. The transition probability 156 represents the probability of a transition from one state among the first eight states to another state among the first eight states. For example, in a case where N and M are both natural numbers from 1 to 8, the transition probability 156 from an Nth state to an Mth state is a value obtained by dividing the transition number 154 from the Nth state to the Mth state by the total transition number from the Nth state.

[0112] The model construction unit 106C constructs the time series model 94 by creating a transition probability matrix based on the transition probability 156 and setting an initial state. In a case where there are eight states, that is, the first through eighth states, the transition probability matrix created based on the transition probability 156 is an 8 × 8 matrix. The transition probability matrix is ​​a core component of the time series model 94. The initial state refers to a starting state (for example, a state selected from the first through eighth states by applying a rule basis to the multiple time series images 152A1) to initiate the prediction performed by the time series model 94. Although eight states, that is, the first through eighth states, are shown here, these are merely examples, and there may be fewer than eight states or nine or more.For example, in a case of seven states, the matrix used as the transition probability matrix is ​​a 7 × 7 matrix, and in a case of 40 states, the matrix used as the transition probability matrix is ​​a 40 × 40 matrix.

[0113] The time series model 94, constructed by the model construction unit 106C, is accessed by the information processing device 100 via the external interfaces 80 and 104 (see Fig. 11) is transmitted to and received by the medical support device 24. In the medical support device 24, the time series model 94 is stored in the data memory 86 by the processor 82 (see Fig. 4) The time series model 94, which is stored in the data memory 86, is used by the recognition unit 82A (see Fig. 4) used.

[0114] Fig. Figure 13 shows an example of processing content in the recognition unit 82A. As in Fig. As shown in Figure 13, an image 151, obtained by imaging the intestinal wall 32 in the large intestine 28, including the lumen 42, with the camera 52, is captured by the recognition unit 82A. The recognition unit 82A generates the single image 40 by performing various types of processing on the image 151. In the Fig. In the example shown in Figure 13, the intestinal wall 32 with the folds 43 and the lumen 42 are shown in the single image 40.

[0115] The detection unit 82A performs lumen detection processing 153 on the single image 40. The lumen detection processing 153 is a process of detecting the lumen 42 shown in the single image 40 using the lumen detection model 92 stored in the data memory 86 (in other words, a process of specifying the existence position of the lumen 42 shown in the single image 40 using the lumen detection model 92). The detection unit 82A captures the single image 40 from the camera 52 and inputs the captured single image 40 into the lumen detection model 92 to cause the lumen detection model 92 to generate confidence-level information 157. The confidence-level information 157 in the present embodiment is an example of “information obtained from the trained model” according to the present disclosure.

[0116] Fig. Figure 14 shows an example of a composition of the confidence-level information 157 generated by the lumen detection model 92 in a case where the lumen 42 is shown in the single image 40. As in Fig. As shown in Figure 14, the confidence-level information 157 is information contained in a map 157A corresponding to the single image 40. The size and shape of map 157A are the same as the size and shape of the single image 40. However, this is merely an example, and the outer contour of map 157A only needs to be in a similar relationship to the outer contour of the single image 40.

[0117] A confidence level 158 (for example, a probability of the existence of lumen 42) is added to map 157A. Here, map 157A is shown as an example, but single image 40 can be used instead of map 157A. Map 157A contains several separation areas 160A, which correspond to the several separation areas 130A (see Fig. 7, Fig. 8 to Fig. 9) Each of the multiple separation areas 160A is an area obtained by separating the map 157A along a circumferential direction CD2 (in other words, around a center point C2 of the map 157A). In the case of the Fig. The 14 examples shown are examples of the multiple separation areas 160A, specifically separation areas 160A1 to 160A8. These separation areas are obtained by separating the map 157A at intervals of a constant angle (for example, at 45-degree intervals) along the circumferential direction CD2. In other words, separation areas 160A1 to 160A8 can be described as areas obtained by separating the map 157A into eight areas radially from the center C2 of the map 157A to an outer edge of the map 157A. The area obtained by the subdivision can also be referred to as a subdivided area.

[0118] In the present embodiment, card 157A is an example of an “image corresponding to the first medical image” and an “image corresponding to the second medical image” according to the present disclosure. Furthermore, confidence level 158 in the present embodiment is an example of a “confidence level” and a “weight” according to the present disclosure. Additionally, separation areas 160A1 to 160A8 in the present embodiment are examples of “multiple separation areas” and “multiple subdivided areas” according to the present disclosure.

[0119] Several centerlines CL are provided in map 157A. The several centerlines CL correspond to the several separation zones 160A and are arranged at equal intervals along the circumferential direction CD2. Each of the several centerlines CL is a virtual line along a half-angle (for example, 22.5 degrees) of the constant angle described above from the center C2 in each separation zone 160A. In the Fig. The 14 examples shown are center lines CL1 to CL8, representing the multiple center lines CL for the separation zones 160A1 to 160A8. Center lines CL1 to CL8 are arranged around center C2 at 45-degree intervals.

[0120] Fig. Figure 15 shows a view example where several unit direction vectors 162 are added to map 157A. As in Fig. As shown in Figure 15, a direction from the center C2 to the existence position of each of the multiple separation regions 160A is determined by the unit direction vector 162. The unit direction vector 162 is added to each of the multiple separation regions 160A. The unit direction vector 162 is a unit vector that indicates a direction from the center C2 to the existence position of the separation region 160 (that is, the existence position of each of the separation regions 160A1 to 160A8). In the Fig. In the example shown in Figure 15, a unit direction vector 162 is added to each of the multiple separation regions 160A (that is, separation regions 160A1 to 160A8) along the center line CL for each separation region 160A. The midpoint C2 in the present embodiment is an example of a “reference position” according to the present disclosure. Furthermore, the multiple unit direction vectors 162 in the present embodiment are examples of “multiple first vectors” according to the present disclosure.

[0121] Fig. Figure 16 shows a view example where several direction vectors 164 are added to map 157A. As in Fig. 16 shown, in a case where a condition described later is met within the reliability range, the controller 82B captures the confidence level information 157, which the map 157A, to which the several unit direction vectors 162 (see Fig. 15) are added, included, by the detection unit 82A.

[0122] The controller 82B generates the multiple direction vectors 164 based on the multiple unit direction vectors 162 and the multiple confidence levels 158 contained in the confidence level information 157 acquired by the detection unit 82A. The direction vector 164 is a vector whose size is adjusted by adding the confidence level 158 of the separation range 160A, to which the unit direction vector 162 is added, as a weight to the unit direction vector 162. The size of the direction vector 164 corresponds to the level of the confidence level 158, and the direction vector 164 becomes larger as the confidence level 158 increases. The multiple direction vectors 164 in the present embodiment are examples of a “multiple third vector” as defined in the present disclosure.

[0123] Here, a specific example of a method for generating the direction vector 164 is described. For example, in the separation area 160A (in which in Fig. 16 shown in the example, the separation region 160A1), to which “0.3” is added as the confidence level 158, a vector obtained by increasing the size of the unit direction vector 162 by 30%, as the direction vector 164 (in which in Fig. The direction vector 164B (as shown in example 16) is generated. Furthermore, for example, in the separation area 160A (where in Fig. 16 shown example, the separation region 160A2), to which “0.7” is added as the confidence level 158, a vector obtained by increasing the size of the unit direction vector 162 by 70%, as the direction vector 164 (in which in Fig. Example 16 shows the direction vector 164A). Furthermore, for example, in the separation area 160A (in which in Fig. 16 shown example, the separation ranges 160A3 to 160A8), to which “0,0” is added as the confidence level 158, the size of the direction vector 164 can be set to “zero” or the unit direction vector 162 can be used as the direction vector 164 as it is.

[0124] The direction vector 164 shown here is merely an example and a vector that can be used as the direction vector 164 by simply multiplying the unit direction vector 162 by the confidence level 158.

[0125] In the Fig. In example 16, direction vectors 164A and 164B are shown. Direction vector 164A is a vector obtained by adjusting the size of the unit direction vector 162, which corresponds to the separation region 160A2 with confidence level 158 (0.7 in the Fig. The direction vector 164B is obtained by adjusting the size of the unit direction vector 162, which is added to the separation region 160A1 with confidence level 158 (0.3 in the example shown in 16). Fig. The magnitude of the direction vector 164 represents a degree of probability of the existence of lumen 42 in the separation region 160A to which the unit direction vector 162 of direction vector 164 is added. That is, the larger the direction vector 164A, the higher the probability of the existence of lumen 42 in the separation region 160A to which a base unit direction vector 162 of direction vector 164 is added.

[0126] The controller 82B generates a vector sum 166 based on the multiple direction vectors 164. The vector sum 166 is a sum of the multiple direction vectors 164. In the Fig. In the example shown in Figure 16, the vector sum 166 is shown to be a vector sum of the direction vector 164A and the direction vector 164B.

[0127] A direction from center C2 to a lumen existence area 168 in map 157A is determined by the vector sum 166. The lumen existence area 168 refers to an area where the lumen 42 is shown in the single image 40. The separation area 160A is an area where the position in map 157A is restricted, whereas the lumen existence area 168 is an area where the position changes depending on the position at which the vector sum 166 is generated, without the position in map 157A being restricted, as in the separation area 160A. Furthermore, in the separation region 160A, even in a case where the lumen 42 exists in the separation region 160A, it is difficult to estimate the existence position of the lumen 42 in the separation region 160A, however, in the lumen existence region 168, since the lumen 42 exists on the line along the vector sum 166, it is easy to estimate the existence position of the lumen 42.Therefore, in lumen existence region 168, the existence position of lumen 42 is specified more precisely than in a case where the existence position of lumen 42 is specified in separation region 160A. In other words, lumen existence region 168 can be described as a region in which the position at which lumen 42 is shown in frame 40 (that is, the existence position of lumen 42 in frame 40) can be specified along the circumferential direction CD2 with a higher resolution than in the multiple separation regions 160A.

[0128] The controller 82B specifies the lumen existence range 168 based on the vector sum 166. For example, the controller 82B specifies as the lumen existence range 168 a range of ±α degrees along the circumferential direction CD2 with a point (for example, an endpoint) other than a starting point of the vector sum 166 as the center. Examples of ±α degrees include ±22.5 degrees. It should be noted that ±22.5 degrees is merely an example and the range may be narrower or wider than ±22.5 degrees. Furthermore, the α degree may be a fixed value or a variable value that is changed in accordance with an instruction received from the receiving device 64 or various conditions (for example, a type of operating mode of the endoscope system 10).

[0129] The controller 82B generates a marker 170 which is able to specify the existence position of the lumen existence area 168 in the map 157A, based on the multiple separation areas 160A and the multiple confidence levels 158.

[0130] The “existence position of the lumen existence area 168 in the map 157A” in the present embodiment is an example of a “first existence position” according to the present disclosure. Furthermore, the vector sum 166 in the present embodiment is an example of a “second vector” according to the present disclosure. In addition, the lumen existence area 168 in the present embodiment is an example of a “lumen existence area” according to the present disclosure.

[0131] The markings 170 represent visible information determined on the basis of the vector sum 166, which is generated based on the multiple separation ranges 160A and the multiple confidence levels 158. In the Fig. In the example shown in Figure 16, the shape of marker 170 is an arc where the endpoint of the vector sum 166 is a midpoint. A midpoint of the arc, which is the shape of marker 170, is the midpoint C2 of map 157A. Marker 170 indicates an area from one end to the other end of the lumen existence area 168 in the circumferential direction CD2. The marker 170 at the point shown in Fig. Example 16 shown is an example of “lumen specification information” and “first lumen specification information” according to the present disclosure.

[0132] Fig. Figure 17 shows an example of processing content using an Nth frame 40 in a case where N is a natural number. In the case described in Fig. In the example shown in Figure 17, the Nth image 40 is an example of a “first medical image” according to the present disclosure. As in Fig. As shown in Figure 17, the detection unit 82A determines whether the condition is within the reliability range, which is applied to the confidence level 158 added to each of the multiple separation ranges 160A of the map 157A contained in the confidence level information 157 output by the lumen detection model 92, by examining the single image 40 (where in Fig. In the example shown in Figure 17, the Nth single image (40) is fed into the lumen detection model 92. The condition "within the reliability range" refers to a condition where each of the multiple confidence levels 158 (here, as an example, all the confidence levels 158) added to the multiple separation ranges 160A of the map 157A, contained in the confidence level information 157 output by the lumen detection model 92 by feeding the single image 40 into the lumen detection model 92, lies within the reliability range.

[0133] The reliability range refers to a range of confidence level 158 that is reliable in determining whether or not Lumen 42 exists. This range is derived in advance through testing with an actual machine and / or computer simulation, or the like. The reliability range is defined by a first threshold TH1 and a second threshold TH2. Both TH1 and TH2 are pre-determined thresholds for confidence level 158. TH2 is a value less than TH1. The reliability range encompasses the range equal to or greater than TH1 and the range less than TH2.Here, the first threshold TH1 is an example of a “first threshold” and a “third threshold” according to the present disclosure, and the second threshold TH2 is an example of a “second threshold” and a “fourth threshold” according to the present disclosure.

[0134] In the present embodiment, confidence level 158 is a reliable value in the range equal to or greater than the first threshold TH1 when determining that the lumen 42 is shown in the separation range 160A, and confidence level 158 is a reliable value in the range less than the second threshold TH2 when determining that the lumen 42 is not shown in the separation range 160A. Conversely, confidence level 158 is set as a value that is not reliable in the range equal to or greater than the second threshold TH2 and less than the first threshold TH1 when determining that the lumen 42 is shown in the separation range 160A.Here, the “value which is not reliable in a case of determining that the lumen 42 is shown in the separation area 160A” can also be described as a value which triggers the faulty specification (in other words, faulty detection) of the existence position of the lumen 42 shown in the single image 40 in the single image 40 via the lumen detection model 92.

[0135] In a case where the condition is within the reliability range, the detection unit 82A generates state information 174 based on the confidence level information 157. Here, the state information 174 generated on the basis of the confidence level information 157 is an example of "time series information relating to one or more specification information about first lumens obtained in time series" according to the present disclosure.

[0136] The state information 174 represents information capable of specifying a state in which the lumen 42 is shown in the separation area 160A (in other words, a state in which the lumen 42 is highly likely to be shown in the image area corresponding to separation area 160A within the entire image area of ​​the single frame 40). Examples of the state in which the lumen 42 is shown in separation area 160A comprise eight states, namely states nine through sixteen. Therefore, the state information 174 can also be described as information capable of specifying which of the ninth through sixteenth states corresponds to the state in which the lumen 42 is shown in separation area 160A. State nine represents a state in which the lumen 42 is shown in separation area 160A1. The tenth state signifies a state in which the lumen 42 is shown in the separation range 160A2.The eleventh state represents a state in which the lumen 42 is shown in the separation area 160A3. The twelfth state represents a state in which the lumen 42 is shown in the separation area 160A4. The thirteenth state represents a state in which the lumen 42 is shown in the separation area 160A5. The fourteenth state represents a state in which the lumen 42 is shown in the separation area 160A6. The fifteenth state represents a state in which the lumen 42 is shown in the separation area 160A7. The sixteenth state represents a state in which the lumen 42 is shown in the separation area 160A8. It should be noted that in the... Fig. 14, Fig. 15 to Fig. 16 examples shown, since the tenth state is shown as a state in which the lumen 42 is shown in the separation area 160A, the detection unit 82A generates information that is able to specify the tenth state as the state information 174.

[0137] Furthermore, in a case where the condition is within the reliability range, the controller 82B generates a marker 170 (see Fig. 16) based on the confidence level information 157 and shows the marker 170 in the first display area 35A (see Fig. 18 and Fig. 19) The marker 170, which is generated on the basis of the confidence-level information 157, is information capable of specifying the existence position of the lumen 42 shown in the Nth frame 40. Here, the existence position of the lumen 42 shown in the Nth frame 40 is an example of a “first existence position” according to the present disclosure.

[0138] Fig. Figure 18 shows a form example where different types of information, such as the single image 40 and the marker 170, are displayed on the screen 35 in a case where the lumen 42 is shown in an area other than the central area of ​​the single image 40. As in Fig. As shown in 18, the controller 82B displays single image 40 (where in Fig. In the example shown in Figure 18, the Nth frame (40) that was input into the lumen detection model 92 to obtain the confidence level information 157, which contains the map 157A used to generate the mark 170, is displayed in the first display area 35A. Then, the controller 82B displays the mark 170 in the first display area 35A in a state comparable to frame 40. For example, the mark 170 is displayed superimposed on frame 40.

[0139] Furthermore, the controller 82B updates the marker 170 in accordance with the display of frame 40. For example, each time the detection unit 82A determines that the confidence level 158 is within the reliability range (that is, each time the condition is met within the reliability range), the controller 82B generates the marker 170 based on the confidence level information 157 and displays the marker 170 superimposed on frame 40. The marker 170, which is displayed in the first display area 35A, is updated each time frame 40 is displayed.It should be noted that the marker 170, which is displayed in the first display area 35A, can be updated provided that the single image 40 is updated several times and displayed in the first display area 35A (for example, the single image 40 is displayed in the first display area 35A in a range of several single images up to several hundred single images that have been predetermined).

[0140] Furthermore, the controller 82B displays visible information 44A in the second display area 35B as part of the auxiliary information 44. Examples of the visible information 44A include text capable of specifying the position of an area corresponding to the lumen existence area 168 in the single image 40; that is, text capable of specifying the position of the marker 170 displayed in the first display area 35A (for example, text representing an angle indicating the position of the vector sum 166 in a case where a boundary line between the separation area 160A1 and the separation area 160A8 is 0 degrees). In addition, the following can be displayed in the Fig. In example 18, the visible information 44A includes the map 157A or an image based on the map 157A (for example, an image obtained by processing the map 157A), and the marker 170, which is superimposed on the map 157A or the image based on the map 157A. The visible information 44A of the in Fig. The 18 examples shown are an example of “lumen specification information” and “first lumen specification information” according to the present disclosure.

[0141] Fig. Figure 19 shows a form example where the single image 40 and the like are displayed on the screen 35 in a case where the lumen 42 is shown in the central area of ​​the single image 40 (for example, in a case where the center of the lumen 42 coincides with the center of the single image 40). As in Fig. Figure 19 shows that in a case where the center of lumen 42 coincides with the center of frame 40, the controller 82B displays frame 40 (where in Fig. In the example shown in Figure 19, the Nth frame (40) is displayed in the first display area 35A, and the marker 170 is superimposed on the frame 40. The marker 170 is a marker (for example, a ring-shaped marker) that surrounds the lumen 42 shown in the frame 40. In a case where the direction vectors 164 of all separation areas 160A are equivalent, that is, in a case where the vector sum 166 is zero, the marker 170 is generated by the controller 82B and displayed superimposed on the frame 40. Furthermore, in this case, the controller 82B displays information (for example, text) as the visible information 44A indicating that the lumen 42 is displayed in the center of the frame 40 in the second display area 35B. In addition, in the example shown in Figure 19, the controller 82B can display information (for example, text) indicating that the lumen 42 is displayed in the center of the frame 40 in the second display area 35B. Fig. In example 19, the visible information 44A includes the map 157A or the image based on the map 157A (for example, the image obtained by processing the map 157A), and the marker 170 (for example, a ring-shaped marker) that is superimposed on the central area of ​​the map 157A or the image based on the map 157A. The visible information 44A of the in Fig. The 19 examples shown are an example of “lumen specification information” and “first lumen specification information” according to the present disclosure.

[0142] Although the example described here, in which marker 170 is generated and superimposed on frame 40, is described in a case where the vector sum 166 is zero, this is merely an example. For instance, in a case where the size of the vector sum 166 is less than a threshold value (for example, the size of the unit direction vector 162), marker 170 can be generated and superimposed on frame 40. Furthermore, as an example of marker 170, the marker surrounding the lumen 42 shown in single image 40 has been described here, but this is merely an example, and a marker (for example, a dot located in the center of the lumen 42 shown in single image 40, or an arrow indicating the position of the lumen 42) that is able to specify the position of the lumen 42 shown in single image 40 can be used.

[0143] Fig. Figure 20 shows an example of processing content in a case where each of the confidence levels 158 added to the multiple separation areas 160A (here, as an example, all the separation areas 160A) of the map 157A, which is contained in the confidence level information 157 output by the lumen detection model 92 by inputting the Nth frame 40 into the lumen detection model 92, is equal to or less than a reference value. As in Fig. As shown in Figure 20, in a case where each of the confidence levels 158 added to the multiple separation ranges 160A (here, as an example, all the separation ranges 160A) of the map 157A, which is contained in the confidence level information 157 output by the lumen detection model 92 by inputting the Nth frame 40 into the lumen detection model 92, is equal to or less than a reference value, the detection unit 82A continues to display the previous marker 170. In the present embodiment, “0.0” is used as the reference value. Here, “0.0” is an example of a “reference value” according to the present disclosure. It should be noted that “0.0” is merely an example and a value (for example, 0.05) greater than “0.0” can be used as the reference value instead of “0.0”. It is preferred that the value used as the reference value is a value that is smaller than the second threshold TH2.

[0144] The continuation of the display of the previous marker 170 means the continuation of the display of marker 170, which preceded the one in Fig. 18 or Fig. Mark 170 was displayed as shown in 19. Examples of the previous mark 170 include the mark 170 generated by controller 82B in a case where the same processing as in Fig. 17. Example shown on an (N-1)th single image 40, as in Fig. 21 shown, is carried out, or the marker 170, which is generated by the controller 82B in a case where the same processing as in Fig. 24 or Fig. The example shown in Figure 25 is performed on the (N-1)th frame 40. Furthermore, in a case where the (N-1)th frame 40 does not exist, for example, the default marker 170 or the marker 170 generated based on the initial state set in a case of constructing the time series model 94 is superimposed on frame 40.

[0145] Furthermore, the 82B controller, for example, performs the following actions, as shown in Fig. 22 shown, in a case in which a state in which each of the confidence levels 158 added to the multiple separation areas 160A (here as an example all the separation areas 160A) of the map 157A, which is contained in the confidence level information 157 output by the lumen detection model 92 by inputting the N-th frame 40 into the lumen detection model 92, is equal to or less than a reference value, continues for the (N+1)-th and subsequent frames 40, controls the continuation of the display of the marker 170, which is superimposed on the N-th frame 40.

[0146] Furthermore, for example, in a case where a condition in which each of the confidence levels 158 added to the multiple separation areas 160A (here, as an example, all the separation areas 160A) of the map 157A, which is contained in the confidence level information 157 output by the lumen detection model 92 by inputting the Nth frame 40 into the lumen detection model 92, is equal to or less than a reference value, persists for a predetermined period, the controller 82B performs the stepwise reduction of a display intensity (in other words, a display level) of the marker 170, which is displayed before the start of the predetermined period, during the predetermined period. Examples of a specific procedure for stepwise reduction of the display intensity include a procedure of stepwise increasing the transparency of the marker 170 and finally hiding the marker 170.

[0147] Examples of predetermined time periods include a period from the start of the display of the Nth frame 40 to the end of the display of several frames (for example, four frames) 40. Here, the period from the start of the display of the Nth frame 40 to the end of the display of several frames 40 is shown as the predetermined period, but this is only an example, and the predetermined period could be a period from the start of the display of the Nth frame 40 to the end of the display of several ten frames 40 or several hundred frames 40.

[0148] Fig. Figure 23 shows an example of processing content using an (N+1)-th frame 40 (that is, the frame 40 obtained one frame after the N-th frame 40) in a case where N is a natural number. In the case described in Fig. In the example shown in 23, the (N+1)th image 40 is an example of a “second medical image” according to the present disclosure. In a case where the Nth image 40 of the in Fig. 16, Fig. 17, Fig. 18, Fig. 19, Fig. 20, Fig. 21 to Fig. In the 22 examples shown, the (N-1)th frame 40 is treated as the (N+1)th frame 40, which is in Fig. 23 is shown as the Nth frame 40 of the in Fig. 16, Fig. 17, Fig. 18, Fig. 19, Fig. 20, Fig. 21 to Fig. The 22 examples shown are discussed.

[0149] As in Fig. As shown in Figure 23, the detection unit 82A determines whether an out-of-reliability condition is met or not, whereby the out-of-reliability condition is applied to each of the confidence levels 158 added to the multiple separation ranges 160A (here, as an example, all separation ranges 160A) of the map 157A contained in the confidence level information 157 output by the lumen detection model 92, by applying the single image 40 (at which in Fig. In the example shown in Figure 23, the (N+1)th single image (40) is entered into the lumen detection model 92. The condition "outside the reliability range" means a condition where each of the confidence levels 158 added to the multiple separation ranges 160A (here, as an example, all the separation ranges 160A) of the map 157A, which are contained in the confidence level information 157 output by the lumen detection model 92 by entering the single image 40 into the lumen detection model 92, is not within the reliability range.In other words, the out-of-reliability condition can also be described as a condition where each of the confidence levels 158 added to the multiple separation ranges 160A (here, as an example, all the separation ranges 160A) of the map 157A, which are contained in the confidence level information 157 output by the lumen detection model 92 by inputting the single image 40 into the lumen detection model 92, lies outside the reliability range. In other words, the out-of-reliability condition can also be described as a condition where the in-reliability condition is not satisfied.

[0150] The out-of-reliability condition is a condition relating to the reliability of each of the confidence levels 158 for the multiple separation ranges 160A (here, by way of example, all separation ranges 160A). In the present embodiment, examples of the out-of-reliability condition include a condition where each of the confidence levels 158 for the multiple separation ranges 160A (here, by way of example, all separation ranges 160A) is equal to or greater than the second threshold TH2 and less than the first threshold TH1. The out-of-reliability condition in the present embodiment is an example of a "failed specification trigger condition" and a "confidence level condition" according to the present disclosure.

[0151] In a case where the detection unit 82A determines that the condition is outside the reliability range, the detection unit 82A performs lumen prediction processing 177. The lumen prediction processing 177 is processing using the time series model 94, in which the existence position of the lumen 42, shown in the (N+1)th frame 40, is predicted in the (N+1)th frame 40.

[0152] In the Lumen prediction processing 177, the latest state information 174 (that is, information capable of specifying a current state among the ninth to sixteenth states), obtained in a case where the frame 40 obtained before the (N+1)th frame is set as a processing target, is fed into the time series model 94. A first example of the latest state information 174 is the state information 174 obtained by executing the in Fig. The processing shown in section 17 is generated. A second example of the latest state information 174 is the state information 174 generated by executing the process shown in Fig. The processing shown in Figure 23 is generated on the Nth frame 40. The time series model 94 outputs a transition probability 176 in response to the input of the latest state information 174. Although the embodiment described here is one in which the time series model 94 outputs the transition probability 176 in response to the input of a single piece of state information 174, this is merely an example. For instance, in a case where the Markov model based on the Nth-order Markov chain is used instead of the time series model 94, the Markov model can output the transition probability 176 in response to the input of multiple pieces of state information 174 that were previously obtained. The transition probability 176 output by the time series model 94 in the present embodiment is an example of “information obtained from the time series model” according to the present disclosure.

[0153] The transition probability 176 output by the time series model 94 is a probability of transition from a state (for example, the tenth state in the in Fig. 14, Fig. 15 to Fig. (16 examples shown), which is specified from the state information 174 input into the time series model 94, to a next state. In the Lumen prediction processing 177, the state information 174 is generated based on the transition probability 176 output by the time series model 94. The time series model 94 outputs several transition probabilities 176. In the Lumen prediction processing 177, information that is able to specify a state corresponding to a maximum transition probability 176 among the several transition probabilities 176 output by the time series model 94 is generated as the state information 174.Here, the state specified from the generated state information 174 is a next state (that is, a state in which the lumen 42 is shown in the separation area 160A used for the (N+1)th frame 40) that transitions from the state specified from the state information 174 input into the time series model 94. The next state that transitions from the state specified from the state information 174 input into the time series model 94 is one of the ninth to sixteenth states.

[0154] Fig. Figure 24 shows an example of an aspect where the state specified from the state information 174 generated by the detection unit 82A through the execution of the lumen prediction processing 177 is visualized on the screen 35. As in Fig. As shown in 24, the controller 82B displays single image 40 (where in Fig. In example 24, the (N+1)th frame 40) in the first display area 35A is processed by the same method as in the example shown. Fig. The example shown in Figure 18 illustrates this. Furthermore, in a case where the condition is outside the reliability range, the controller 82B acquires the state information 174 generated by executing the Lumen prediction processing 177 from the detection unit 82A. The controller 82B then generates the marker 170 based on the state information 174 acquired by the detection unit 82A. The marker 170, generated based on the state information 174, is a marker capable of specifying the state identified by the state information 174 (that is, any state from the ninth to the sixteenth states).In other words, the marker 170, which is generated on the basis of the state information 174, can also be described as information capable of specifying the existence position of the lumen 42 shown in the (N+1)th frame 40. Here, the existence position of the lumen 42 shown in the (N+1)th frame 40 is an example of a “second existence position” according to the present disclosure.

[0155] The controller 82B displays marker 170, generated based on the state information 174 obtained from the time series model 94, in the first display area 35A in a state comparable to single frame 40, provided the condition is outside the reliability range. For example, marker 170 is superimposed on single frame 40. Furthermore, marker 170 is displayed at the position of the separation area 160A corresponding to the state specified from the state information 174. For example, if the state specified from the state information 174 is the tenth state, marker 170 is displayed at a position corresponding to separation area 160A2. Consequently, the existence position of the lumen 42, which is shown in the (N+1)th frame 40, is visually specified in the (N+1)th frame 40 by the display position of the marker 170.

[0156] Furthermore, the controller 82B updates the marker 170 in accordance with the display of frame 40. For example, the controller 82B generates the marker 170 based on the state information 174 and displays the marker 170 superimposed on frame 40 each time the state information 174 is obtained by executing the lumen prediction processing 177. In a case where the marker 170 is generated based on the state information 174, the visible information 44A in the second display area 35B is updated by the same procedure as in the Fig. The example shown in 18 is displayed. Fig. The example shown in Figure 24 is the mark 170, which is generated on the basis of the state information 174 and is displayed in the first display area 35A, an example of “lumen specification information” and “second lumen specification information” according to the present disclosure.

[0157] Fig. Figure 25 shows an example of processing content in a case where the condition is within the reliability range (that is, in a case where each of the confidence levels 158 added to the separation ranges 160A of the map 157A contained in the confidence level information 157 output by the lumen detection model 92 is within the reliability range). As in Fig. Figure 25 shows that, in a case where the condition is within the reliability range, the detection unit 82A and the controller 82B perform the same processing as in the case shown in Fig. 17. In this case, the controller 82B generates the marker 170 (see 17). Fig. 16) based on the confidence level information 157 output by the lumen detection model 92, by inputting the (N+1)th frame 40 into the lumen detection model 92. Then the controller 82B displays the marker 170 in the first display area 35A by the same procedure as in Fig. The example shown in Figure 18 illustrates this. The marker 170, which is generated on the basis of the confidence level information 157 output by the lumen detection model 92 by inputting the (N+1)th frame 40 into the lumen detection model 92, is information capable of specifying the existence position of the lumen 42 shown in the (N+1)th frame 40.

[0158] The marker 170, which is generated in the present embodiment by inputting the (N+1)th frame 40 into the lumen detection model 92 based on the confidence level information 157 output by the lumen detection model 92, is an example of “lumen specification information” and “third lumen specification information” according to the present disclosure. Furthermore, the existence position of the lumen 42 shown in the (N+1)th frame 40 is an example of a “third existence position” according to the present disclosure.

[0159] The following is an example of a processing flow performed by the information processing device 100 for machine learning, with reference to Fig. 26 described.

[0160] In the processing for machine learning, which is in Fig. As shown in Figure 26, the training data generation unit 106A first captures an unprocessed example image 122A from the example image set 122 at step ST10 (see Figure 26). Fig. 5 and Fig. 6), which is stored in data storage 110. Here, the unprocessed sample image 122A means the sample image 122A that has not yet been used for machine learning processing. The training data generation unit 106A displays the sample image 122A, which was captured from the sample image set 122, on the screen 118A (see Fig. 6, Fig. 8 and Fig. 9) After the processing of step ST10 has been completed, the machine learning processing moves on to step ST12.

[0161] In step ST12, the training data generation unit 106A receives an instruction for the lumen correspondence position 139 (see Fig. 8 and Fig. 9) After the processing of step ST12 has been completed, the machine learning processing moves on to step ST14.

[0162] In step ST14, the training data generation unit 106A specifies a positional relationship between the lumen correspondence position 139, received in step ST12, and the multiple separation areas 130A (see Fig. 8 and Fig. 9) After the processing of step ST14 has been completed, the machine learning processing moves on to step ST16.

[0163] In step ST16, the training data generation unit 106A assigns the ground truth data 126 to the example image 122A, which was captured in step ST10, in accordance with the position relationship specified in step ST14 (see Fig. 8 and Fig. 9) For example, if the lumen correspondence position 139 exists in an area outside the center of the sample image 122A, the ground truth data 126 is assigned to the separation area 130A that has the greatest overlap with the lumen correspondence position 139 in response to the instruction issued by the annotator 124. Furthermore, if the lumen correspondence position 139 exists in the center of the sample image 122A, the ground truth data 126 is assigned to each of the separation areas 130A in response to the instruction issued by the annotator 124. In this way, the training data generation unit 106A assigns the ground truth data 126 to the sample image 122A to generate the training data 128. The training data 128 generated in this way are stored on a predetermined storage medium (for example, the data storage device 110).After the processing of step ST16 has been completed, the machine learning processing moves on to step ST18.

[0164] In step ST18, the training data generation unit 106A determines whether or not there is an unprocessed example image 122A. If an unprocessed example image 122A exists, step ST18 returns a negative result, and the machine learning processing proceeds to step ST10. If no unprocessed example image 122A exists, step ST18 returns a positive result, and the machine learning processing proceeds to step ST20.

[0165] In step ST20, the learning execution unit 106B performs machine learning using the multiple training data 128 obtained by repeatedly performing the processing in steps ST10 to ST18 to generate the lumen detection model 92 (see Fig. 10). The lumen detection model 92 is stored in the data memory 86 of the medical support device 24 (see Fig. 4) After the processing of step ST20 has been completed, the machine learning processing ends.

[0166] The following is an example of a model construction processing flow performed by the information processing device 100, with reference to Fig. 27 described.

[0167] At the in Fig. In the model construction processing shown in Figure 27, the model construction unit 106C first acquires, at step ST50, the several ground truth data 152A2, which are assigned to several time series images 152A1, which are arranged in time series, from the time series data set 152A stored in the data memory 110 (see Figure 27). Fig. 12) in time series. After the processing of step ST50 has been executed, the model construction processing moves on to step ST52.

[0168] In step ST52, the model construction unit 106C counts the transition number 154 between the respective states in which the lumen 138 is shown in the separation area 130A, in the multiple time series images 152A1, which are arranged in time series, from the multiple ground truth data 152A2, which are arranged in time series (see Fig. 12). After the processing of step ST52 has been completed, the model design processing proceeds to step ST54.

[0169] In step ST54, the model construction unit 10C calculates the transition probability 156 based on the transition number 154 (see Fig. 12) After the processing of step ST54 is executed, the model construction processing proceeds to step ST56.

[0170] In step ST56, the model construction unit 106C constructs the time series model 94 by creating the transition probability matrix based on the transition probability 156 and setting the initial state (see Fig. 12). After the processing of step ST56 is executed, the model design processing ends.

[0171] The following is an example of a flow of medical support processing performed by the endoscope system 10, with reference to Fig. 28A and Fig. 28B described. The in Fig. 28A and Fig. The medical support processing flow shown in Figure 28B is an example of a “medical support process” according to the present disclosure. For the sake of simplicity, the description below is given assuming that the lumen detection model 92 generated by performing the machine learning processing and the time series model 94 constructed by performing the model construction processing are stored in the data memory 86, and that the lumen detection model 92 and the time series model 94 are retrieved from the data memory 86 and used by the detection unit 82A.

[0172] In the medical support processing that takes place in Fig. As shown in Figure 28A, in step 100 the recognition unit 82A first captures the image 151 from the camera 52 and performs various types of processing on the captured image 151 in order to generate the single image 40 (see Figure 28A). Fig. 13) After processing at step ST100 has been completed, the medical support processing proceeds to step ST102.

[0173] In step ST102, the detection unit 82A performs the lumen detection processing 153 via the lumen detection model 92 to cause the lumen detection model 92 to generate the confidence level information 157 (see Fig. 13, Fig. 17, Fig. 23 and Fig. 25). After the processing at step ST102 has been completed, the medical support processing proceeds to step ST104.

[0174] In step ST104, the detection unit 82A determines whether the condition is within the reliability range or not. That is, the detection unit 82A determines whether each of the confidence levels 158 contained in the confidence level information 157 generated in step ST102 lies within the reliability range or not (see Fig. 17, Fig. 23 and Fig. 25). In step ST104, if the condition is not within the reliability range (that is, if the condition is met outside the reliability range), a negative determination is made, and the medical support processing proceeds to step ST126, which is in Fig. As shown in 28B, in step ST104, if the condition is within the reliability range, a positive determination is made, and the medical support processing proceeds to step ST106.

[0175] In step ST106, the detection unit 82A generates the state information 174 based on the confidence level information 157 generated by the lumen detection model 92 (see Fig. 17 and Fig. 25). Here, examples of the generated state information include 174 pieces of information capable of specifying a state in which the lumen 42 is shown in the separation range 160A, to which the maximum confidence level 158 is added among all the confidence levels 158 that are added to all the separation ranges 160A of the map 157A contained in the confidence level information 157. After the processing at step ST106 has been performed, the medical support processing proceeds to step ST108.

[0176] In step ST108, the controller 82B generates the multiple direction vectors 164 by adding the confidence level 158 of the separation area 160A, to which each unit direction vector 162 is added, as a weight to the unit direction vector 162 of each separation area 160A of the map 157A, which are contained in the confidence level information 157 generated in step 102 (see Fig. 16) After processing at step ST108 has been completed, medical support processing proceeds to step ST110.

[0177] In step ST110, the controller 82B generates the vector sum 166, which is a sum of the several direction vectors 164 that were generated in step ST108 (see Fig. 16) After the processing at step ST110 has been completed, the medical support processing moves on to step ST112.

[0178] In step ST112, the controller 82B specifies the lumen existence range 168 based on the vector sum 166 generated in step ST110. After the processing in step ST112 has been completed, the medical support processing proceeds to step ST114.

[0179] In step ST114, the controller 82B generates the marker 170, which is able to specify the position of the lumen existence area 168 specified in step ST112 (see Fig. 16). After the processing at step ST114 has been completed, the medical support processing proceeds to step ST116.

[0180] In step ST116, the controller 82B displays the single image 40 generated in step ST100 in the first display area 35A (see Fig. 18 and Fig. 19). After the processing at step ST116 has been completed, the medical support processing proceeds to step ST118.

[0181] In step ST118, the controller 82B displays the marker 170, which was generated in step ST114, superimposed on the single image 40, which is displayed in the first display area 35A (see Fig. 18 and Fig. 19) After processing at step ST118 has been completed, the medical support processing proceeds to step ST142, which is in Fig. 28B is shown, over.

[0182] At in Fig. In step ST126, shown on page 28B, controller 82B displays the single image 40 generated in step ST100 in the first display area 35A. After the processing in step ST126 has been completed, the medical support processing proceeds to step ST128.

[0183] In step ST128, the controller 82B determines whether each of the confidence levels 158 added to the separation areas 160A of the card 157A, which are contained in the confidence level information 157 generated in step ST102, is equal to or less than the reference value or not (see Fig. 20). In step ST128, if each of the confidence levels 158 added to the separation areas 160A of map 157A, which are contained in the confidence level information 157 generated in step ST102, is equal to or less than the reference value, a positive determination is made, and the medical support processing proceeds to step ST130. In step ST128, if each of the confidence levels 158 added to the separation areas of map 157A, which are contained in the confidence level information 157 generated in step ST102, is not equal to or less than the reference value, a negative determination is made, and the medical support processing proceeds to step ST132.

[0184] At step ST130, the controller 82B continues the superimposed display of the previous marker 170 (see Fig. 21 and Fig. 22). For example, controller 82B continues displaying marker 170, which is superimposed on the preceding frame 40. After processing at step ST130 is completed, the medical support processing proceeds to step ST142.

[0185] In step ST132, the detection unit 82A performs the lumen prediction processing 177 using the latest state information 174. That is, the detection unit 82A inputs the latest state information 174 into the time series model 94 to cause the time series model 94 to output the transition probability 176 (see Fig. 23). Here, examples of the latest status information 174 include the status information 174 (see Fig. 17), which are generated in a case where the in Fig. 17 processing shown on the preceding single image 40 is carried out, or the status information 174 (see Fig. 23), which are generated in a case where the in Fig. The lumen prediction processing shown in 23 is performed on the preceding frame 40. In a case where the Fig. The state information 174 generated by the processing shown in Figure 17 on the preceding frame 40 refers to the state information 174 generated by executing the processing of step ST106 in a case where the preceding frame 40 is set as the processing target. Furthermore, the state information 174 generated in a case where the processing shown in Figure 17 is performed on the preceding frame 40 refers to the state information 174 generated by executing the processing of step ST106 in a case where the preceding frame 40 is set as the processing target. Fig. The lumen prediction processing 177 shown in Figure 23 is performed on the preceding frame 40. The processing is then applied to the state information 174 generated by executing the processing of step ST134 in a case where the preceding frame 40 is set as the processing target. After the processing at step ST132 has been executed, the medical support processing proceeds to step ST134.

[0186] In step ST134, the detection unit 82A generates the state information 174 based on the transition probability 176 output by the time series model 94 in step ST132 (see Fig. 23). After the processing at step ST134 has been completed, the medical support processing proceeds to step ST136.

[0187] In step ST136, the controller 82B generates the marker 170 based on the state information 174 generated in step ST134 (see Fig. 24). After processing at step ST136 has been completed, medical support processing moves to step ST138.

[0188] In step ST138, the controller 82B displays the single image 40 generated in step ST100 in the first display area 35A (see Fig. 24). After processing at step ST138 has been completed, medical support processing proceeds to step ST140.

[0189] In step ST140, the controller 82B displays the marker 170, which was generated in step ST136, superimposed on the single image 40, which is displayed in the first display area 35A (see Fig. 24). After processing at step ST140 has been completed, the medical support processing moves to step ST142.

[0190] In step ST142, the controller 82B determines whether a medical support processing end condition is met. Examples of medical support processing end conditions include a condition where an instruction to terminate medical support processing is issued to the endoscope system 10 (for example, a condition where the receiving device 64 receives the instruction to terminate medical support processing).

[0191] In step 142, if the end condition for medical support processing is not met, a negative determination is made, and the medical support processing proceeds to step ST100, which is in Fig. As shown in Figure 28A, in a case where the end condition for medical support processing is met at step ST100, an affirmative determination is made, and the medical support processing ends.

[0192] As described above, in the endoscope system 10, the information capable of specifying the existence position of the lumen 42 shown in the single image 40 is the marker 170 (hereinafter also referred to as a “first marker” without reference), which was generated on the basis of the confidence level information 157 obtained by the lumen detection processing 153 using the lumen detection model 92, and the marker 170 (hereinafter also referred to as a “second marker” without reference), which was generated on the basis of the state information 174 obtained by the lumen prediction processing 177 using the time series model 94, which is selectively displayed in the first display area 35A.Here, the first marker is generated based on the confidence level information 157 obtained from the lumen detection model 92 by inputting the Nth frame 40 into the lumen detection model 92 and is a marker capable of specifying the existence position of the lumen 42 shown in the Nth frame 40 in the Nth frame 40. Furthermore, the second marker is generated on the basis of the transition probability 176, which is obtained from the time series model 94 by inputting information (for example, the state information 174, which was generated on the basis of the confidence level information 157 used to generate the first marker) relating to the first marker into the time series model 94, and is a marker that is able to specify the existence position of the lumen 42, which is shown in the (N+1)th frame 40, in the (N+1)th frame 40.Therefore, with the endoscope system 10, the physician 12 or the like can determine the existence position of the lumen 42, which is shown in each of the several individual images 40 arranged in time series (for example, the Nth individual image 40 and the (N+1)th individual image 40), in each individual image 40 without omission.

[0193] Furthermore, in the endoscope system 10, the marker 170 is displayed in the first display area 35A in a state comparable to the single image 40 displayed in the first display area 35A. For example, the marker 170 is superimposed on the single image 40 displayed in the first display area 35A. As a result, the physician 12 or the like can visually determine the positional relationship between a section in the colon 28 observed by the single image 40 and the lumen 42.

[0194] Furthermore, in the endoscope system 10, the marker 170, which is displayed in the first display area 35A, is updated in accordance with the display of the single image 40. As a result, it is possible to ensure the consistency between the content of the single image 40 displayed in the first display area 35A and the marker 170, and it is possible for the physician 12 or the like to visually recognize the marker 170 in accordance with the content shown in the single image 40 displayed in the first display area 35A.

[0195] Furthermore, in the endoscope system 10, if the single image 40 displayed in the first display area 35A is an image that meets the condition outside the reliability range, the second marker is displayed in the first display area 35A. Therefore, even if the single image 40 displayed in the first display area 35A is an image that meets the condition outside the reliability range, the physician 12 or the like can visually determine the position of the lumens 42 shown in the single image 40.

[0196] Furthermore, in the endoscope system 10, if the single image 40 displayed in the first display area 35A is an image that does not meet the condition outside the reliability range (i.e., an image that meets the condition within the reliability range), a third marker is generated, and the third marker is superimposed on the single image 40 displayed in the first display area 35A. The third marker is generated based on the confidence level information 157 obtained from the lumen detection model 92 by inputting the (N+1)th single image 40 into the lumen detection model 92. The confidence level information 157 contains the map 157A, which is subdivided into the multiple separation areas 160A. The confidence level 158, indicating that the lumen 42 exists, is added to each of the multiple separation areas 160A.Furthermore, the third marker is a marker 170, which is capable of specifying the existence position of the lumen 42 located in the (N+1)th frame 40. Therefore, even in a case where the frame 40 displayed in the first display area 35A is a frame that does not satisfy the condition outside the reliability range (that is, a frame that satisfies the condition within the reliability range), the physician 12 or the like can visually determine the existence position of the lumen 42 shown in the frame 40.

[0197] Furthermore, in the case of the endoscope system 10, if all confidence levels 158 contained in the confidence level information 157 output by the lumen detection model 92 by inputting the Nth frame 40 into the lumen detection model 92 are equal to or less than the reference value, the display of the marker 170, which is displayed in the first display area 35A before the confidence level 158 that is equal to or less than the reference value, is continued.Furthermore, in the case of the endoscope system 10, if a condition in which all confidence levels 158 contained in the confidence level information 157 output by the lumen detection model 92 by inputting the Nth frame 40 into the lumen detection model 92 are equal to or less than the reference value persists for a predetermined period (for example, several frames or a time required to display several frames), the display level of the marker 170, which is displayed in the first display area 35A before the start of the predetermined period, gradually decreases during the predetermined period.In this way it is possible to reduce the visual discomfort caused to the doctor 12 or the like due to switching between displaying and hiding marker 170 (for example, flickering due to switching between displaying and hiding marker 170), compared to a case where the display and hiding of marker 170 are frequently switched.

[0198] Furthermore, in the endoscope system 10, the single image 40, showing the intestinal wall 32 and the lumen 42, is entered into the lumen detection model 92, so that the confidence level information 157 is generated by the lumen detection model 92. The confidence level information 157 contains the map 157A, which is subdivided into several separation regions 160A. The confidence level 158, indicating that the lumen 42 exists, is added to each of the several separation regions 160A.

[0199] In the endoscope system 10, the lumen existence region 168 is generated based on the multiple separation regions 160A and the multiple confidence levels 158. Lumen existence region 168 is a region in which the existence position of the lumen 42 (that is, the position at which the lumen 42 is shown) is specified more precisely than in the separation region 160A in the single image 40. Since the lumen existence region 168 is generated based on the multiple separation regions 160A and the multiple confidence levels 158, the position of the lumen existence region 168 in map 157A is not fixed as it is for the separation region 160A. Furthermore, the position of the lumen existence region 168 in map 157A is finely modified along the circumferential direction CD2 depending on the existence position of the lumen 42.This means that the lumen existence region 168 is a region in which the existence position of the lumen 42 is defined with a higher resolution than in the multiple separation regions 160A along the circumferential direction CD2.

[0200] In the endoscope system 10, the marker 170 is used as information capable of specifying the lumen existence range 168. Then, the single image 40, which is input into the lumen detection model 92 to generate the confidence level information 157, is displayed in the first display area 35A of the screen 35. Furthermore, the marker 170 is superimposed on the single image 40. This means that the marker 170 on the single image 40 is displayed with a higher resolution than in a case where the confidence level 158 of each of the multiple separation ranges 160A is simply superimposed on the single image 40, or where the visible information (for example, the marker) indicating the level of the confidence level 158 is simply displayed in the first display area 35A.

[0201] Therefore, the physician 12 can determine the position of the lumen 42 shown in the single image 40 more accurately in the single image 40 by simply visually recognizing the marker 170, which is displayed superimposed on the single image 40, compared with a case in which the confidence level 158 of each of the several separation areas 160A is simply displayed superimposed on the single image 40, or the visible information indicating the level of the confidence level 158 is simply displayed in the first display area 35A.

[0202] Furthermore, in the endoscope system 10, the direction from the center C2 of map 157A to the existence position of each of the multiple separation areas 160A is determined by the multiple unit direction vectors 162. Additionally, the direction from the center C2 of map 157A to the lumen existence area 168 is determined by the vector sum 166. The vector sum 166 is a sum of the multiple direction vectors 164, obtained by adding the confidence level 158 as the weight to the multiple unit direction vectors 162. The marker 170, which is superimposed on the single image 40, is generated based on the vector sum 166.

[0203] Here, the multiple direction vectors 164 are modified in accordance with the confidence level 158 added to each of the multiple separation regions 160A. The vector sum 166 is modified in accordance with the multiple direction vectors 164. It can be said that the vector sum 166 is a vector indicating the direction from the center C2 of map 157A to the existence position of lumen 42. It can be said that the marker 170, which is generated on the basis of the vector sum 166, is visible information that more accurately represents the existence position of the lumen 42, which is shown in the single image 40, compared with a case in which the confidence level 158 of each of the several separation areas 160A is simply superimposed on the single image 40, or the visible information indicating the height of the confidence level 158 is simply displayed in the first display area 35A.

[0204] Therefore, the physician 12 can determine the position of the lumen 42 shown in the single image 40 more accurately in the single image 40 by simply visually recognizing the marker 170, which is displayed superimposed on the single image 40, compared with a case in which the confidence level 158 of each of the several separation areas 160A is simply displayed superimposed on the single image 40, or the visible information indicating the level of the confidence level 158 is simply displayed in the first display area 35A.

[0205] In the embodiment described above, the form example was described in which the lumen existence range 168 is generated on the basis of the multiple separation ranges 160A and the multiple confidence levels 158 (see Fig. 16), but the present disclosure is not limited thereto. For example, by the same procedure as in a case where the lumen existence range 168 is generated on the basis of the multiple separation ranges 160A and the multiple confidence levels 158, the lumen existence range 168 can be generated on the basis of the multiple separation ranges 160A, which are based on the (N+1)th frame 40 (see Fig. 23) are applied, and the several transition probabilities 176 (see Fig. 23), which are output by the time series model 94. In this case, too, the marker 170 can be generated using the same procedure as in Fig. The example shown in 16 is generated, and the generated marker 170 can be displayed in the first display area 35A using the same procedure as in the example shown. Fig. The 18 examples shown will be displayed.

[0206] Although the embodiment described above uses the form example in which step ST104, which is in Fig. As shown in Figure 28A, it is determined whether the condition is satisfied within the reliability range or not. This is merely an example, and it is sufficient to determine whether the faulty specification trigger condition, which is a condition that triggers the faulty specification (in other words, faulty detection) via the lumen detection model 92, is satisfied or not. Here, the faulty specification refers to the faulty specification of the existence position of the lumen 42 shown in frame 40 (for example, the (N+1)th frame 40) that is input into the lumen detection model 92.

[0207] The faulty specification trigger condition is a condition that contains at least one of the first through fifth conditions. The first condition is a condition where at least part of the lumen 42 is not shown in frame 40 (for example, the (N+1)th frame 40). The second condition is a condition where at least one dark section, distinct from the lumen 42, is shown in frame 40 (for example, the (N+1)th frame 40). The "dark section" here refers, for example, to a section of darkness at a level where, by inputting frame 40 into the lumen recognition model 92, it is incorrectly identified as the lumen 42. The third condition is a condition in which an obstruction that obstructs the lumen 42 is shown in the single frame 40 (for example, the (N+1)th single frame 40).A first example of an obstruction is a treatment instrument (for example, a grasping forceps, a papillotomy knife, a snare, a catheter, a guidewire, a cannula, a biopsy needle with a guide tube, and / or a clamp). A second example of an obstruction is an adhering substance to a lens of the camera 52. Examples of adhering substances include water droplets, residues, liquid drugs, and / or blood. The fourth condition is one in which image quality triggers the faulty specification. The image quality that triggers the faulty specification refers, for example, to blurriness, blurring, noise, and / or brightness at a level where it is incorrectly identified by the lumen detection model 92 as the lumen 42 when the lumen detection model 92 inputs the single frame 40.The fifth condition is a condition in which a section in the large intestine 28 (for example, a section with a curved part) that triggers the faulty specification is shown in frame 40 (for example, the (N+1)th frame 40).

[0208] In a case where the condition containing at least one of the first to fifth conditions is used instead of the condition within the reliability range, an example is given, as in Fig. Figure 29 shows that processing of step ST104A is performed by processor 82 instead of processing of step ST104, which is included in the medical support processing. Step ST104A, which is in the Fig. The medical support processing shown in Figure 29 includes the recognition unit 82A, which determines whether at least one of the first five conditions is met. In step ST104A, if none of the first five conditions are met, a negative determination is made, and the medical support processing proceeds to step ST106 (see Figure 29). Fig. 28A). In step ST104A, if at least one of the first to fifth conditions is met, a positive determination is made and the medical support processing proceeds to step ST126 (see Fig. 28B). Therefore, in a case where the condition containing at least one of the first to fifth conditions is met, the existence position of lumen 42 is predicted by the lumen prediction processor 177 using the time series model 94, without specifying the existence position of lumen 42 by the lumen detection processor 153 using the lumen detection model 92. Consequently, it is possible to prevent the information (for example, the marker 170) based on the result (for example, the lumen existence area 168) that was incorrectly specified by the lumen detection processor 153 using the lumen detection model 92 from being visually recognized by the physician 12 or the like.

[0209] Although the embodiment described above represents a form in which the marking 170 is generated based on the vector sum 166, the present disclosure is realized even in a case where the marking 170 is not generated based on the vector sum 166. For example, as in Fig. Figure 30 shows that the marker 170 (for example, an arc-shaped marker with its center point located on the center line CL) is generated in the separation area 160A, to which the maximum confidence level 158 is added among all confidence levels 158 added to all separation areas 160A of the map 157A, and the generated marker 170 can be displayed in the first display area 35A by the same procedure as in the embodiment described above.

[0210] Furthermore, as in Fig. Figure 31 shows as an example a case in which the separation area 160A, to which the maximum confidence level 158 is added among all confidence levels 158 added to all separation areas 160A of map 157A, is changed (for example, in a case in which separation area 160A2 is changed to separation area 160A3), the marker 170 is placed at a position corresponding to separation area 160A after the change (where in Fig. In the example shown in Figure 31, the separating area 160A3) is displayed, and the display of marker 170 before the change (for example, marker 170 displayed at the position corresponding to separating area 160A2) can continue. Furthermore, as in the embodiment described above, the display level of marker 170 before the change can gradually decrease during the predetermined period.

[0211] In the embodiment described above, example image 122A was used, but this is only an example, and, for instance, as shown in Fig. As shown in Figure 32, example image 122B is used instead of example image 122A.

[0212] The interior of the large intestine 132 is shown in example image 122B. In the Fig. The example shown in Figure 32 shows an intestinal wall 136 in which the several folds 134 are formed and a lumen 138 in the example image 122B.

[0213] Example image 122B is divided into several separation regions 130C. These separation regions 130C comprise a central region 130C0 and eight radial regions 130C1 to 130C8. The central region 130C0 is a circular area with a center point that coincides with a center point C3 of example image 122B. The radial regions 130C1 to 130C8 extend radially from the central region 130C0 to an outer edge of example image 122B and are arranged along a circumferential direction CD3 of example image 122B (in other words, around the center C3 of example image 122B).

[0214] Fig. Figure 33 is a concept diagram illustrating an example of a method for generating training data 128A by mapping ground truth data 126A to the example image 122B using the training data generation unit 106A. As shown in Fig. Figure 33 shows that, in a state where example image 122B is displayed on screen 118A, the annotator 124 specifies a lumen correspondence position 139A, which is the position of the lumen 138 shown in example image 122B, relative to the training data generation unit 106A via the receiving device 116. The training data generation unit 106A displays a circular frame 140A superimposed on example image 122B in response to the instruction received from the receiving device 116 and positions the frame 140A at a location surrounding the lumen 138 shown in example image 122B. The single image 140A is a marker that defines the lumen correspondence position 139A in the example image 122B. That is, the position of an area surrounded by the single image 140A in the example image 122B is the lumen correspondence position 139A.The size and position of frame 140A are freely changed on screen 118A in response to the instruction received from receiver 116. Here, frame 140A has a circular shape, but it can be a shape other than circular. The size of frame 140A can be changed in response to the instruction received from receiver 116.

[0215] The annotator 124 issues an acknowledgment instruction, which is an instruction to acknowledge the lumen correspondence position 139A, to the training data generation unit 106A via the receiving device 116 in a state where the single image 140A is positioned at the location surrounding the lumen 138. As a result, the training data generation unit 106A acknowledges the lumen correspondence position 139A.

[0216] The training data generation unit 106A specifies the separation area 130C with a largest area that overlaps with the single image 140A, which defines the lumen correspondence position 139A, from the multiple separation areas 130C. Then the training data generation unit 106A generates the training data 128A by using the ground truth data 126A with the separation area 130C, in which the lumens 138 are shown, as the annotation that is able to identify the separation area 130C (where in Fig. The example shown in Figure 33 is to specify the radial area 130C2), which was specified from the several separation areas 130C.

[0217] At the in Fig. The example shown in 33 depicts the view where the ground truth data 126 is assigned to the radial area 130C2, but this is merely an example. For instance, as shown in Fig. Figure 34 shows a case in which the separation area 130C with the largest area overlapping with the single image 140A is separation area 130C, the central area is 130C0, the training data generation unit 106A generates the training data 128A by assigning the ground truth data 126A to the central area 130C0.

[0218] Fig. Figure 35 is a concept diagram showing an example of a view in which the learning execution unit 106B performs machine learning using the training data 128A to generate the lumen detection model 92. As shown in Fig. As shown in Figure 35, in the information processing device 100, the learning execution unit 106B acquires the training data 128A generated by the training data generation unit 106A. The learning execution unit 106B then generates a lumen detection model 92A by performing machine learning using the training data 128A, using the same procedure as in a case where machine learning is performed using the training data 128 in the embodiment described above. The lumen detection model 92A is transmitted by the information processing device 100 to the medical support device 24 via the external interfaces 80 and 104 (see Figure 35). Fig. 5) is transmitted and received by the medical support device 24. Then, the lumen detection model 92A is stored in the data memory 86 by the processor 82 of the medical support device 24. The lumen detection model 92A stored in the data memory 86 is used by the detection unit 82A.

[0219] Fig. Figure 36 is a concept diagram showing an example of processing content where the Markov model is constructed based on the Markov chain as the time series model 94A by the model construction unit 106C. As in Fig. As shown in Figure 36, a time-series data set 152B contains several time-series images 152B1 arranged in a time series. As with the several time-series images 152A1 described in the embodiment described above, the several time-series images 152B1 are several images corresponding to the several single images 40 arranged in a time series and contained in the endoscopy video image 39 (that is, the several images representing the several single images 40 arranged in a time series).

[0220] As in example image 122B, the interior of the large intestine 132 is shown in time series image 152B1. Furthermore, time series image 152B1 is divided into several separation regions 130D using the same procedure as in example image 122B. In the Fig. The 36 examples shown are the multiple separation areas 130D, separation areas 130D0 to 130D8. The geometric properties of all separation areas 130D in the time series image 152A1 correspond to the geometric properties of all separation areas 130C in the example image 122B. Furthermore, using the same procedure as in a case where the ground truth data 126A is assigned to each example image 122B, the ground truth data 152B2 corresponding to the ground truth data 126A is also assigned to each time series image 152B1.

[0221] The model construction unit 106C acquires several ground truth data 152B2 in time series, which are assigned to the several time series images 152B1 arranged in time series, and counts a transition count 154A between the respective states in which the lumen 138 is shown in the separation area 130D in the several time series images 152B1 arranged in time series, from the several ground truth data 152B2 arranged in time series. Each state in which the lumen 138 is shown in the separation area 130D is specified from the ground truth data 152B2 linked to the several time series images 152B1.

[0222] Examples of the state in which the lumen 138 is shown in the 130D separation range comprise nine states, namely the seventeenth through twenty-fifth states. The seventeenth state represents a state in which the lumen 138 is shown in the 130D0 separation range. The eighteenth state represents a state in which the lumen 138 is shown in the 130D1 separation range. The nineteenth state represents a state in which the lumen 138 is shown in the 130D2 separation range. The twentieth state represents a state in which the lumen 138 is shown in the 130D3 separation range. The twenty-first state represents a state in which the lumen 138 is shown in the 130D4 separation range. The twenty-second state represents a state in which the lumen 138 is shown in the 130D5 separation range. The twenty-third state signifies a state in which the lumen 138 is shown in the separation range 130D6.The twenty-fourth state represents a state in which the lumen 138 is shown in the separation area 130D7. The twenty-fifth state represents a state in which the lumen 138 is shown in the separation area 130D8.

[0223] The transition number 154A means the number of times of transition among the seventeenth to twenty-fifth states (in other words, the number of times the separation region 130D, in which the lumen 138 is shown, transitions from one separation region 130D to another separation region 130D among the nine separation regions 130D0 to 130D8).

[0224] The model construction unit 106C counts the transition count 154A by pairing a state in an Nth time series image 152B1 with a state in an (N+1)th time series image 152B1 and sums the transition count 154A among the seventeenth to twenty-fifth states. That is, the model construction unit 106C sums the number of times a transition occurs from a given state among the seventeenth to twenty-fifth states to the next state as the transition count 154A.

[0225] The model construction unit 106C calculates a transition probability 156A based on the transition number 154A. The transition probability 156A represents the probability of a transition from one state among the seventeenth to twenty-fifth states to another state among the seventeenth to twenty-fifth states. For example, in a case where N and M are natural numbers from 17 to 25, the transition probability 156A from an Nth state to an Mth state is a value obtained by dividing the transition number 154A from the Nth state to the Mth state by the total transition number from the Nth state.

[0226] The model construction unit 106C constructs the time series model 94A by creating a transition probability matrix based on the transition probability 156A and setting an initial state. In a case where there are nine states, that is, the seventeenth to twenty-fifth states, the transition probability matrix created based on the transition probability 156A is a 9 × 9 matrix. The transition probability matrix is ​​a core component of the time series model 94A. The initial state refers to a starting state (for example, a state selected from the seventeenth to twenty-fifth states based on the multiple time series images 152B1) to begin the prediction via the time series model 94A.Although the nine states described here, that is, the seventeenth to twenty-fifth states, these are merely examples, and there may be fewer than nine states or more than nine states.

[0227] The time series model 94A, constructed by the model construction unit 106C, is accessed by the information processing device 100 via the external interfaces 80 and 104 (see Fig. 11) is transmitted to and received by the medical support device 24. In the medical support device 24, the time series model 94A is stored in the data memory 86 by the processor 82. The time series model 94A, stored in the data memory 86, is used by the recognition unit 82A.

[0228] Fig. 37, Fig. 38, Fig. 39 to Fig. 40 show an example of processing content that corresponds to the in Fig. 17, Fig. 20, Fig. 23 and Fig. This corresponds to the 25 examples shown. As in Fig. 37, Fig. 38, Fig. 39 to Fig. As shown in Figure 40, the single image 40 contains separation regions 40A0 to 40A8, which correspond to separation regions 130D0 to 130D8. The geometric properties of the separation regions 40A0 to 40A8 applied to the single image 40 are the same as the geometric properties of the separation regions 130D0 to 130D8 applied to the time series image 152B1.

[0229] As in Fig. As shown in Figure 37, in a case where the Nth frame 40 is input into the lumen detection model 92A, confidence level information 157 is output by the lumen detection model 92A. As contained in the confidence level information 157 described in the embodiment described above, the confidence level information 157 includes a map corresponding to map 157A described in the embodiment described above. The map contained in the confidence level information 157 has nine cutoff regions corresponding to cutoff regions 40A0 to 40A8, and the confidence level 158 is added to each cutoff region of the map contained in the confidence level information 157 by the same procedure as in the embodiment described above. In the case described in Figure 37, the confidence level 158 is output by the lumen detection model 92A. Fig. In the example shown in Figure 37, the controller 82B, in a case where the condition is within the reliability range for all confidence levels 158 added to each of the separation areas of the map contained in the confidence level information 157, generates the marker 170 based on the confidence level information 157 output by the lumen detection model 92A by inputting the Nth frame 40 into the lumen detection model 92A and displays the generated marker 170 in the first display area 35A by the same procedure as in the embodiment described above.

[0230] Furthermore, in a case where the condition is within the reliability range for all confidence levels 158 added to each of the separation areas of the map contained in the confidence level information 157, the controller 82B generates the state information 174A corresponding to the state information 174 described in the embodiment described above, based on the confidence level information 157, using the same procedure as in the embodiment described above.

[0231] The state information 174A is information capable of specifying a state in which the lumen 42 is shown in the separation area of ​​the map contained in the confidence level information 157. Examples of the state in which the lumen 42 is shown in the map area contained in the confidence level information 157 comprise nine states, that is, the twenty-sixth to the thirty-fourth states. Therefore, the state information 174A can also be described as information capable of specifying which of the twenty-sixth to the thirty-third states corresponds to the state in which the lumen 42 is shown in the separation area of ​​the map contained in the confidence level information 157.

[0232] The twenty-sixth through thirty-fourth states are states corresponding to the seventeenth through twenty-fifth states described above. The twenty-sixth state signifies a state in which the lumen 42 is shown in the separation area corresponding to separation area 40A0 among the nine separation areas of the map contained in Confidence Level Information 157. The twenty-seventh state signifies a state in which the lumen 42 is shown in the separation area corresponding to separation area 40A1 among the nine separation areas of the map contained in Confidence Level Information 157. The twenty-eighth state signifies a state in which the lumen 42 is shown in the separation area corresponding to separation area 40A2 among the nine separation areas of the map contained in Confidence Level Information 157.The twenty-ninth state signifies a state in which the lumen 42 is shown in the separation area corresponding to separation area 40A3 of the nine separation areas of the map contained in the confidence level information 157. The thirtieth state signifies a state in which the lumen 42 is shown in the separation area corresponding to separation area 40A4 of the nine separation areas of the map contained in the confidence level information 157. The thirty-first state signifies a state in which the lumen 42 is shown in the separation area corresponding to separation area 40A5 among the nine separation areas of the map contained in the confidence level information 157. The thirty-second state signifies a state in which the lumen 42 is shown in the separation area corresponding to separation area 40A6 among the nine separation areas of the map contained in the confidence level information 157.The thirty-third state signifies a state in which the lumen 42 is shown in the separation area corresponding to separation area 40A7 among the nine separation areas of the map contained in the confidence level information 157. The thirty-fourth state signifies a state in which the lumen 42 is shown in the separation area corresponding to separation area 40A8 among the nine separation areas of the map contained in the confidence level information 157.

[0233] As in Fig. As shown in Figure 38, in a case where all confidence levels 158 contained in the confidence level information 157 output by the lumen detection model 92A by inputting the Nth frame 40 into the lumen detection model 92A are equal to or less than the reference value, the controller 82B continues to display the previous marker 170 using the same procedure as in the embodiment described above.

[0234] As in Fig. As shown in Figure 39, in a case where the condition is outside the reliability range for all confidence levels 158 contained in the confidence level information 157 output by the lumen detection model 92A by inputting the (N+1)th frame 40 into the lumen detection model 92A, the detection unit 82A performs the lumen prediction processing 177A using the time series model 94A with the same procedure as in the lumen prediction processing 177 described above. In the lumen prediction processing 177A, the latest state information 174A is inputted into the time series model 94A. That is, the detection unit 82A inputs the latest state information 174A into the time series model 94A to cause the time series model 94A to output a transition probability 176A. Examples of the latest status information 174A include the status information 174A (see Fig. 37), which are generated in a case where the in Fig. 37 processing shown on the preceding single image 40 is carried out, or the status information 174A (see Fig. 39), which are generated in a case where the in Fig. Figure 39 shows the lumen prediction processing 177A being performed on the preceding frame 40. The time series model 94A outputs the transition probability 176A, which corresponds to the transition probability 176 described in the embodiment described above, in response to the input of the latest state information 174. Although the embodiment described here is one in which the time series model 94A outputs the transition probability 176A in response to the input of a single piece of state information 174A, this is merely an example. For instance, in a case where the Markov model based on the Nth-order Markov chain is used instead of the time series model 94A, the Markov model can output the transition probability 176A in response to the input of multiple pieces of state information 174A that were previously obtained.

[0235] The transition probability 176A output by the time series model 94A is a probability of transition from the state specified from the state information 174A input into the time series model 94A to the next state. In the lumen prediction processing 177A, the state information 174A is generated based on the transition probability 176A output by the time series model 94A. Here, the state specified from the generated state information 174A is a next state (that is, a state in which the lumen 42 is shown in one of the separation ranges 40A0 to 40A8 used for the (N+1)th frame 40) that transitions from the state specified from the state information 174A input into the time series model 94A.The next state to transition from the state specified in the state information 174A entered into the time series model 94A is one of the twenty-sixth to the thirty-fourth states.

[0236] As in Fig. As shown in Figure 40, in a case where the condition is within the reliability range for each of the confidence levels 158 added to the separation ranges of the confidence level information 157 output by the lumen detection model 92A by inputting the (N+1)th frame 40 into the lumen detection model 92A, the detection unit 82A and the controller 82B perform the same processing as in the case shown in Figure 40. Fig. The example shown in Figure 37 illustrates this. In this case, the controller 82B generates the marker 170 based on the confidence-level information 157A output by the lumen detection model 92A by inputting the (N+1)th frame 40 into the lumen detection model 92A. The controller 82B then displays the marker 170 in the first display area 35A using the same procedure as in the embodiment described above. The marker 170, generated based on the confidence-level information 157 output by the lumen detection model 92A by inputting the (N+1)th frame 40 into the lumen detection model 92A, is information capable of specifying the existence position of the lumen 42 shown in the (N+1)th frame 40.

[0237] In this way, in a case where the condition is within the reliability range for each of the confidence levels 158 added to the separation areas of the map, which are output by the lumen detection model 92A by inputting the (N+1)th frame 40 into the lumen detection model 92A, the controller 82B preferentially displays on the screen 35 the information based on the processing result obtained by the lumen detection model 92A (i.e., the mark 170 generated on the basis of the confidence level information 157) over the information based on the processing result obtained by the time series model 94A (i.e., the mark 170 generated on the basis of the state information 174A generated by the lumen prediction processing 177A).As a result, the physician 12 or the like can accurately determine the existence position of the lumen 42 shown in the (N+1)th single image 40.

[0238] In the embodiment described above, the arc was described as the shape of the marking 170, but this is merely an example, and the marking with a different shape, such as a toroidal marking or a polygonal marking, can be used.

[0239] As in Fig. As shown in Figure 41 as an example, a marker 170A, having a shape along the outer edge of the single image 40, can be generated and displayed in the first display area 35A instead of the marker 170. Examples of the method for generating the marker 170A include a method in which the marker 170 is generated from the side of center C2 (see Figure 41). Fig. 16) on the outer edge of map 157A (see Fig. 16) is projected to generate the marker 170A with a shape along the outer edge of the map 157A. The single image 40, which is input into the lumen detection model 92 to generate the confidence level information 157 including the map 157A, is displayed in the first display area 35A, and the marker 170A is displayed in accordance with the display time of the single image 40.

[0240] Since the marker 170 is formed within the map 157A, a form example has been described as an example of the procedure for generating the marker 170, in which the marker 170 is projected from the side of the center C2 onto the outer edge of the map 157A. However, in a case where the marker 170 is formed outside the map 157A (for example, in a case where the marker 170 is formed below an upper end of the map 157A or below a lower end of the map 157A), the marker 170 only needs to be projected from the outside of the map 157A onto the outer edge of the map 157A in the direction of the side of the center C2.

[0241] As described above, in a case where the marking with a shape is displayed along the outer edge of the single image 40 in the first display area 35A, the display of the object that visually obstructs the single image 40 in the first display area 35A is suppressed, so that it is possible to enable the physician 12 to accurately determine the position of the lumen 42 shown in the single image 40, while ensuring the visibility of the single image 40 displayed in the first display area 35A.

[0242] Furthermore, the marker 170A can also be updated in accordance with the display time of single image 40 by the same procedure as the marker 170 is updated in accordance with the display time of single image 40. In this way, the physician 12 can visually recognize the marker 170A in accordance with the contents of single image 40 displayed in the first display area 35A.

[0243] In the embodiment described above, an example form was described in which the marker 170 is displayed superimposed on the single image 40, but this is merely an example, and the marker 170 can also be displayed outside the single image 40. In a case where the marker 170 is displayed outside the single image 40, there is no object that visually obstructs the single image 40 displayed in the first display area 35A, and thus the visibility of the single image 40 displayed in the first display area 35A can be improved.

[0244] In the embodiment described above, a form example was described in which the control 82B defines the arc-shaped marking 170, which is capable of defining the area from one end to the other end of the lumen existence area 168 (see Fig. 16) in the circumferential direction CD2 (see Fig. 16) to specify, generates and displays the generated mark 170 in the first display area 35A; however, this is only an example. For example, as in Fig. 42 shown, the controller 82B displays the mark 170 in the first display area 35A by displaying at least one marking 171 in the first display area 35A, which is capable of extending the area from one end to the other end of the lumen existence area 168 (see Fig. 16) to specify in the circumferential direction CD2 under several hidden markings 171, which are assigned to the card 157A. For example, the number of multiple markings 171 (in other words, the number of subdivided parts) can be “40” (= 8 × 5), which is obtained by each of the eight separation areas 160A (see Fig. 14, Fig. 15 to Fig. 16) is divided equally into five parts, or “80” (= 8 × 10), which is obtained by dividing each of the eight separation areas 160A equally into ten parts. The number of multiple markings 171 can be a different number than these. As the number of markings 171 increases, the resolution of the marking 170 display increases. That is, the controller 82B can generate and display the marking 170 in greater detail as the number of markings 171 increases.

[0245] In the Fig. The example shown in Figure 42 shows several arc-shaped markings arranged at constant intervals along a circle with a center that coincides with the center C2 of map 157A, as examples of the several hidden markings 171.

[0246] In the Fig. In the example shown in Figure 42, the display of the marker 170 in the first display area 35A is achieved by displaying at least one marking 171, which is capable of covering the area from one end to the other end of the lumen existence area 168 (see Figure 42). Fig. 16) to specify in the circumferential direction CD2, implemented (see Fig. 16).

[0247] In the Fig. In the example shown in Figure 42, the marker 170 is superimposed on the single image 40, which is displayed in the first display area 35A, but the marker 170 can be displayed outside of the single image 40 depending on the position of the lumen existence area 168 (see Figure 42). Fig. 16). For example, at least one marking 171 is placed at a position (in which in Fig. 20 example shown above the top of map 157A in a front view and below the bottom of map 157A in a front view) is shown away from map 157A, and thus marker 170 is shown outside of single image 40, which is shown in the first display area 35A.

[0248] In the Fig. In the example shown in Figure 42, the multiple arc-shaped markers arranged at constant intervals along a circle with a center coinciding with center C2 of map 157A were shown as examples of the multiple hidden markers 171, but this is only one example. For instance, the multiple hidden markers 171 could be multiple markers arranged at constant intervals along the outer edge of map 157A.

[0249] In the Fig. The example shown in Figure 42 illustrates the multiple arc-shaped markers arranged at constant intervals along a circle with a center coinciding with center C2 of map 157A, a portion of which overlaps map 157A. However, this is merely an example. For instance, multiple arc-shaped markers arranged at constant intervals along a circle with a center coinciding with center C2 of map 157A, surrounding map 157A, can be used. In this case, the marker 170 outside of single image 40 is indicated by at least one marker 171 capable of extending from one end to the other of the lumen existence area 168 (see Figure 42). Fig. 16) in the circumferential direction CD2 (see Fig. 16) to specify, shown.

[0250] As described above, the display of the marker 170 is achieved by displaying at least one marking 171 at a position corresponding to the position of the lumen existence area 168 (see Fig. 16), implemented among the several hidden markers 171, so that the processing load required to display marker 170 can be reduced.

[0251] In the Fig. In example 42, a form example was described in which the marker 170 is displayed; however, this is merely an example, and, for instance, as in Fig. As shown in Figure 43, the outer contour line 172 can be displayed in the first display area 35A instead of the marker 170. The outer contour line 172 is a line that outlines an outer contour of the lumen existence area 168 (see Figure 43). Fig. 16). In the Fig. In the example shown in Figure 43, an outer contour line 172 is superimposed on the single image 40, which is displayed in the first display area 35A. In a case where the outer contour line 172 is superimposed on the single image 40 in this way, the physician 12 can visually recognize that the lumen 42 is displayed within the outer contour line 172.

[0252] Furthermore, as in the example in Fig. As shown in Figure 43, a line segment 173 (for example, a line segment extending from the center C2 along the vector sum 166) is superimposed on the single image 40 along the vector sum 166. In this case, the physician 12 can visually recognize that the lumen 42 is shown on the line segment 173, which is superimposed on the single image 40.

[0253] In the embodiment described above, eight separation zones 160A are shown as examples, but the number of separation zones 160A can be less than eight or nine or more. Furthermore, the number of separation zones 130A must also be determined in accordance with the number of separation zones 160A.

[0254] In the embodiment described above, a form example was described in which the single image 40 is displayed on the screen 35 and the marker 170 is displayed in a state comparable to the single image 40 (for example, the form example in which the marker 170 is displayed superimposed on the single image 40), but this is only an example. For example, the map 157A can be displayed on the screen 35, and the marker 170 can be displayed in a state comparable to the map 157A. An example of the comparable display example is superimposed display of the marker 170 on the map 157A.

[0255] In the embodiment described above, an example form has been described in which the visible information 44A is displayed in the second display area 35B; however, this is merely an example. For instance, audible information (such as an electronic tone or a speech tone) capable of specifying the position of the lumen 42 in the single image 40 shown in the single image 40 can be emitted by a loudspeaker (not shown). Furthermore, the information, in which the single image 40 and the marker 170 or the like are combined, and / or the visible information 44A can be printed on a medium by a printer.Furthermore, the information combining the single image 40 and the marker 170 or the like, the visible information 44A and / or the audible information described above can be stored on a storage medium (for example, a data storage device provided in an external device such as the data storage device 76, the data storage device 86 or the server).

[0256] In the embodiment described above, the endoscopy image was described as the single image 40, but this is merely an example, and the present disclosure is realized even in a case where a medical image, such as an MRI image, a CT image, an X-ray image, or three-dimensional volume data (for example, three-dimensional volume data generated on the basis of multiple layer images, such as multiple MRI images or multiple CT images), is used instead of the single image 40.

[0257] Although the embodiment described above describes an example in which the single image 40, in which the lumen 42 is shown, is input into the lumen detection model 92 or 92A, the present disclosure is not limited thereto. For example, even in a case in which the single image 40, in which the lumen 42 is not shown, is input into the lumen detection model 92 or 92A, as long as the single image 40, which satisfies the specification trigger condition, is input into the lumen detection model 92 or 92A, the erroneous specification (in other words, the erroneous detection) of the lumen 42 is suppressed, thus establishing the present disclosure.

[0258] In the embodiment described above, the embodiment in which the medical support processing is performed by the computer 78 is described, but the present disclosure is not limited thereto, and at least part of the medical support processing can be performed by a device provided outside of the computer 78. An example of such a case is given below with reference to Fig. 44 described.

[0259] Fig. Figure 44 is a concept diagram showing an example configuration of an endoscope system 200. The endoscope system 200 is an example of an “endoscope system” according to the present disclosure. The endoscope system 200 differs from the endoscope system 10 according to the embodiment described above in that an external device 202 is provided.

[0260] The external device 202 is communicatively connected to the computer 78 via a network 204 (for example, a WAN and / or a LAN).

[0261] Examples of the external device 202 include at least one server that transmits data directly or indirectly to and from computer 78 via network 204. The external device 202 receives a processing instruction issued by the processor 82 of computer 78 via network 204. The external device 202 then performs processing in accordance with the received processing instruction and transmits a processing result to computer 78 via network 204. In computer 78, the processor 82 receives the processing result transmitted by the external device 202 via network 204 and performs processing using the received processing result.

[0262] Examples of the processing execution instruction include an instruction to cause the external device 202 to perform at least one part of the medical support processing. A first example of the at least one part of the medical support processing (that is, processing to be performed by the external device 202) is the lumen detection processing 153. In this case, the external device 202 performs the lumen detection processing 153 in response to the processing execution instruction issued by the processor 82 via the network 204 and transmits information containing the confidence-level information 157 to the computer 78 via the network 204. In the computer 78, the processor 82 receives the confidence-level information 157 and performs the same processing as in the embodiment described above, using the received confidence-level information 157.

[0263] A second example of at least part of the medical support processing (that is, processing to be performed by the external device 202) is the lumen prediction processing 177. In this case, the external device 202 performs the lumen prediction processing 177 in response to the processing execution instruction issued by the processor 82 via the network 204 and transmits the transition probability 176 and / or the state information 174 via the network 204 to the computer 78. In the computer 78, the processor 82 receives the transition probability 176 and / or the state information 174 and performs the same processing as in the embodiment described above, using the received transition probability 176 and / or state information 174.

[0264] A third example of at least one part of the medical support processing (that is, the processing to be performed on the external device 202) is processing performed by the controller 82B (for example, processing related to display). In this case, the external device 202 performs the processing through the controller 82B in response to the processing execution instruction issued by the processor 82 via the network 204 and transmits the processing result (for example, the single image 40 and / or the marker 170) via the network 204 to the computer 78. In the computer 78, the processor 82 receives the processing result and performs the same processing (for example, displaying using the display device 18) as in the embodiment described above, using the received processing result.

[0265] A fourth example of at least part of the medical support processing (that is, the processing to be performed by the external device 202) includes various types of determination processing (for example, determining whether the condition is within the reliability range or not, determining whether the confidence level 158 is equal to or less than the reference value or not, and / or determining whether the condition is outside the reliability range or not).In this case, the external device 202 performs the processing, which relates to various determinations, in accordance with the processing execution instruction issued by the processor 82 via the network 204 and transmits the processing result (for example, the determination result of whether the condition is within the reliability range or not, the determination result of whether the confidence level 158 is equal to or less than the reference value or not, and / or the determination result of whether the condition is outside the reliability range or not) via the network 204 to the computer 78. In the computer 78, the processor 82 receives the processing result and performs the same processing as in the embodiment described above, using the received processing result.

[0266] The external device 202 can be implemented using cloud computing. Cloud computing is just one example, and the external device 202 can also be implemented using network computing, such as fog computing, edge computing, or grid computing.

[0267] In the embodiment described above, the embodiment in which the medical support program 90 is stored in the data storage device 86 was described, but the present disclosure is not limited thereto. For example, the medical support program 90 may be stored in a portable, non-volatile, computer-readable storage medium, such as an SSD or a USB flash drive. The medical support program 90, stored on the non-volatile storage medium, is installed on the computer 78 of the endoscope system 10. The processor 82 performs the medical support processing in accordance with the medical support program 90.

[0268] Furthermore, the medical support program 90 can be stored in a storage device of another computer, server or the like, which is connected to the endoscope system 10 via the network, and the medical support program 90 can be downloaded from the endoscope system 10 and installed on the computer 78 in response to a request.

[0269] It is not necessary to store the entire medical support program 90 in a storage device of another computer or server device connected to the endoscope system 10, or to store the entire medical support program 90 in the data storage 86, and part of the medical support program 90 may be stored.

[0270] The following different processors can be used as hardware resources for performing medical support processing. One example is a CPU, which is a general-purpose processor that executes software, i.e., a program, to perform medical support processing. Another example is a dedicated electrical circuit, which is a processor with a dedicated circuit configuration designed to perform specific processing, such as an FPGA, a PLD, or an ASIC. All processors have memory built into or connected to them, and all processors use this memory to perform medical support processing.

[0271] The hardware resource for performing medical support processing can be configured by one of several processors or by a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, the hardware resource for performing medical support processing can be a single processor.

[0272] A first example of a configuration where the hardware resource is configured by a processor is one where a processor is configured by a combination of one or more CPUs and software, and this processor acts as the hardware resource for performing medical support processing. A second example involves using a processor that implements all the functions of a system, including multiple hardware resources that perform medical support processing with an integrated circuit (IC), such as a system-on-a-chip (SoC). In this way, medical support processing is implemented by using one or more of the different processors as the hardware resource.

[0273] Furthermore, the hardware structure of the various processors can be, in particular, an electronic circuit in which circuit elements such as semiconductor elements are combined. The medical support processing described above is merely an example. Therefore, it is self-evident that, within the scope that does not deviate from the core of this disclosure, unnecessary steps may be deleted, new steps may be added, or the processing order may be changed.

[0274] The content described and shown above is a detailed description of the parts according to this disclosure and is merely an example of this disclosure. For instance, the descriptions of the configurations, functions, operations, and effects are descriptions of examples of the configurations, functions, operations, and effects of the parts according to this disclosure. It is therefore understood that, with regard to the content described and shown above, unnecessary parts may be deleted, new elements may be added, or substitutions may be made, within a scope that does not deviate from the core of this disclosure.To avoid confusion and to facilitate understanding of the parts according to the present disclosure, the description of general technical knowledge or the like, which does not specifically require the description to enable the implementation of the present disclosure, is omitted from the contents described above and the contents shown above.

[0275] All documents, patent applications and technical standards described in this description are incorporated into this description by reference to the same extent as in a case where it is specifically and individually described that each of the documents, patent applications and technical standards is described by reference.

[0276] With regard to the embodiment described above, the following additional remarks are further disclosed. Supplementary Note 1

[0277] A medical support device comprising: a processor configured to: acquire a medical image produced by imaging the interior of a luminal organ; and selectively output multiple lumen specification information sets, which are information capable of specifying the existence position of a lumen shown in the medical image, classifying the medical image into a first medical image and a second medical image acquired later than the first medical image, the multiple lumen specification information sets comprising first lumen specification information and second lumen specification information, the first lumen specification information being based on information obtained from a trained model in a case where the first medical image is input into the trained model.are generated and are information capable of specifying a first existence position, which is the existence position of the lumen shown in the first medical image, and the second lumen specification information based on information obtained from a time series model in a case where time series information relating to one or more of the first lumen specification information obtained in time series is input into the time series model, and are information capable of specifying a second existence position, which is the existence position of the lumen shown in the second medical image, the first medical image or an image corresponding to the first medical image, several separation areas,which are obtained by separating the first medical image or the image corresponding to the first medical image along a circumferential direction, the trained model has multiple confidence levels corresponding to the multiple separation regions and indicating that the lumen is shown for each of the multiple separation regions, in a case where the first medical image or the image corresponding to the first medical image is input into the trained model, the first lumen specification information is information that is able to specify the first existence position more precisely than the separation regions in the first medical image or the image corresponding to the first medical image, the trained model generates the first lumen specification information based on the multiple separation regions and the multiple confidence levels, and the first lumen specification information is a marker that is able toto specify an area that is determined as the lumen existence area on the basis of a second vector in the first medical image or the image corresponding to the first medical image. Supplementary Note 2

[0278] The medical support device according to supplementary note 1, wherein one form of marking is an arc and one center of the arc is a center of the medical image or of the image corresponding to the medical image. Supplementary Note 3

[0279] The medical support device according to supplementary note 2, wherein a form of marking is a form along an outer edge of the medical image or an outer edge of the image corresponding to the medical image. Supplementary note 4

[0280] The medical support device according to one of the supplementary notes 1 to 3, wherein several hidden markings are associated with the medical image or the image corresponding to the medical image, the medical image or the image corresponding to the medical image is displayed on a screen, and the marking on the screen is displayed by showing at least one marking corresponding to a position of the lumen existence area among the several markings. Supplementary note 5

[0281] A medical support device comprising: a processor configured to: acquire a medical image generated by imaging the interior of a luminal organ; and selectively output multiple lumen specification information pieces, which are information capable of specifying the existence position of a lumen shown in the medical image, classifying the medical image into a first medical image and a second medical image acquired later than the first medical image, the multiple lumen specification information pieces comprising first lumen specification information and second lumen specification information, the first lumen specification information being based on information obtained from a trained model in a case where the first medical image is input into the trained model.are generated and are information capable of specifying a first existence position, which is the existence position of the lumen shown in the first medical image, in the first medical image, the second lumen specification information is generated based on information obtained from a time series model in a case where time series information, which is information relating to one or more of the first lumen specification information obtained in time series, is input into the time series model, and are information capable of specifying a second existence position, which is the existence position of the lumen shown in the second medical image, in the second medical image, the second medical image or an image corresponding to the second medical image, several separation areas,which are obtained by separating the second medical image or the image corresponding to the second medical image along a circumferential direction, the information obtained from the time series model includes multiple transition probabilities, the second lumen specification information is information capable of specifying the second existence position more precisely than the separation regions in the second medical image or the image corresponding to the second medical image, and the second lumen specification information is generated based on the multiple separation regions and the multiple transition probabilities. Reference symbol list 10,200 Endoscope System 12 Doctor 14 employees 16 Endoscope 18, 118 Display device 20 Light source device 22 Control device 24 medical support devices 26 subjects 28, 132 Large intestine 30 lights 32, 136 Intestinal wall 34 cars 35, 118A screen 35A first display area 35B second display area 39 Endoscopy video image 40 individual images 40A0 to 40A8, 130A, 130A1 to 130A8, 130B, 130B1 to 130B8, 130C, 130C0 to 130C8, 130D, 130D0 to 130D8, 160A, 160A1 to 160A8 Separation range 42,138 lumens 43, 134 fold 44 Help Information 44A visible information 46 Control unit 48 Insert part 50 distal end section 50A distal end face 52 Camera 54 Lighting device 54A, 54B Lighting window 56 Treatment tool opening 58 Treatment tool 60 Treatment tool insertion port 62 Universal cables 64, 116 Receiving device 66, 78, 102 Computer Bus 68, 88, 112 70, 80, 104 external interface 72, 82, 106 processor 74, 84, 108 memory 76, 86, 110 data storage 82A Detection Unit 82B Control 90 medical support program 92, 92A Lumen detection model 94, 94A Time series model 100 Information processing device 106A Training Data Generation Unit 106B Learning Implementation Unit 106C Model Construction Unit 116A keyboard 116B Mouse 120 Processing program for machine learning 122 Example image set 122A, 122B Example image 124 Annotator 126, 126A, 152A2 ground truth data 128, 128A Training data 139, 139A Lumen correspondence position 140, 140A Single image 142 Model 144 Inference result 146 errors 148 Adjustment value 150 Model Construction Processing Program 151 image 152A, 152B Time series data set 152A1, 152B1 Time series image 153 lumen detection processing 154, 154A Transition number 156, 156A Transition probability 157 Confidence level information 157A map 158 Confidence level 162 Unit direction vector 164, 164A, 164B: Direction vector 166 Vector sum 168 lumen existence range 170, 170A marking 171 Identification 172 Outer contour line 173 line segment 174, 174A Condition Information 177, 177A Lumen prediction processing 202 External Device 204 Network C1, C2, C3 Middle CD1, CD2, CD3 Circumference CL center line CL1 to CL8 center line TH1 first threshold TH2 second threshold QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] WO 2020 / 194472A

[0002]

Claims

[1] Medical support device (24), comprising: a processor (82) configured to: a medical image (40) produced by imaging the interior of a luminal organ (28); and several lumen specification information (170), which is information that is able to specify an existence position of a lumen (42) shown in the medical image (40), selectively outputs in the medical image (40), wherein the medical image (40) is classified into a first medical image and a second medical image, which is obtained later than the first medical image, the multiple lumen specification information (170) contain first lumen specification information and second lumen specification information, the first lumen specification information is generated based on information obtained from a trained model (92) in a case where the first medical image is input into the trained model (92), and is information that is able to specify a first existence position, which is the existence position of the lumen (42) shown in the first medical image, in the first medical image, and the second lumen specification information is generated based on information obtained from a time series model (94) in a case in which time series information, which is information relating to one or more of the first lumen specification information obtained in time series, is input into the time series model (94) and is information that is able to specify a second existence position, which is the existence position of the lumen (42) shown in the second medical image. [2] Medical support device (24) according to claim 1, wherein the processor (82) is configured to output the second lumen specification information in a case where the second medical image is an image that triggers a faulty specification trigger condition, the faulty specification of the second existence position by the trained model (92). [3] Medical support device (24) according to claim 2, wherein the multiple lumen specification information (170) includes third lumen specification information, the processor (82) is configured to output the third lumen specification information in a case where the second medical image is an image that does not meet the faulty specification trigger condition, and the third lumen specification information is generated based on information obtained from the trained model (92) in a case where the second medical image is input into the trained model (92) and is information that is able to specify a third existence position, which is the existence position of the lumen (42) shown in the second medical image, in the second medical image. [4] Medical support device (24) according to claim 1, wherein the multiple lumen specification information (170) includes third lumen specification information, the processor (82) is configured to output the third lumen specification information in a case where the second medical image is an image that does not satisfy a faulty specification trigger condition, the faulty specification of the second existence position by the trained model (92), and the third lumen specification information is generated based on information obtained from the trained model (92) in a case where the second medical image is input into the trained model (92) and is information that is able to specify a third existence position, which is the existence position of the lumen (42) shown in the second medical image, in the second medical image. [5] Medical support device (24) according to claim 2, wherein the faulty specification trigger condition includes: a first condition in which at least part of the lumen (42) is not shown in the second medical image, a second condition in which at least one dark section that differs from the lumen (42) is shown in the second medical image, a third condition in which an obstruction that obstructs the lumen (42) is shown in the second medical image, a fourth condition where image quality triggers the faulty specification, and / or a fifth condition in which a section of the luminal organ (28) that triggers the faulty specification is shown in the second medical image. [6] Medical support device (24) according to claim 2, wherein the trained model (92) generates a confidence level (158) indicating that the lumen for each of several subdivided areas (160A1-160A8) obtained by subdividing the second medical image or an image corresponding to the second medical image is shown in a case where the second medical image is input into the trained model (92), and The faulty specification trigger condition includes a confidence level condition relating to the reliability of the confidence level (158) for each of the several subdivided areas (160A1-160A8). [7] Medical support device (24) according to claim 6, wherein the confidence level condition includes a condition in which the confidence level (158) for each of the several subdivided ranges (160A1-160A8) is equal to or greater than a second threshold (TH2) that is less than a first threshold (TH1) and is less than the first threshold. [8] Medical support device (24) according to claim 1, wherein the first medical image or an image (157A) corresponding to the first medical image has several separation areas (160A1-160A8) obtained by separating the first medical image or the image (157A) corresponding to the first medical image along a circumferential direction (CD2), the trained model (92) generates several confidence levels (158) corresponding to the multiple separation ranges (160A1-160A8) and indicating that the lumen (42) is shown for each of the multiple separation ranges (160A1-160A8) in a case where the first medical image or the image (157A) corresponding to the first medical image is input into the trained model (92), the first lumen specification information is information that is able to specify the first existence position more accurately than the separation ranges (160A1-160A8) in the first medical image or the image (158) corresponding to the first medical image, and the first lumen specification information is generated based on the multiple separation ranges (160A1-160A8) and the multiple confidence levels (158). [9] Medical support device (24) according to claim 8, wherein the first lumen specification information is information that is able to specify the first existence position with a higher resolution than the multiple separation areas (160A1-160A8) along the circumferential direction (CD2). [10] Medical support device (24) according to claim 8, wherein a direction from a reference position (C2) of the first medical image or the image (157A) corresponding to the first medical image to an existence position of each of the multiple separation regions (160A1-160A8) is determined by multiple first vectors (162), a direction from the reference position (C2) to a lumen existence area, (168) which contains the first existence position, is determined by a second vector (166), the second vector (166) is a sum of at least two third vectors (164) obtained by adding the confidence level (158) as a weight to at least two first vectors (162) among the several first vectors (162), and The first lumen specification information is information that is determined on the basis of the second vector (166). [11] Medical support device (24) according to claim 1, wherein the second medical image or an image corresponding to the second medical image (157A) has a central area of ​​the second medical image and several separation areas (160A1-160A8) obtained by radially separating the surroundings of the central area in the second medical image, the trained model (92) generates a confidence level (158) indicating that the lumen (42) is shown for the central region and each of the multiple separation regions (160A1-160A8) in a case where the second medical image is input into the trained model (92), and In a case where the confidence level (158) for each of the mean and multiple separation areas (160A1-160A8) is equal to or greater than a third threshold (TH1), or the confidence level (158) for each of the mean and multiple separation areas is less than a fourth threshold (TH2) that is less than the third threshold (TH1), information based on a processing result obtained by the trained model (92) is preferentially output over information based on a processing result obtained by the time series model (94). [12] Medical support device (24) according to claim 1, wherein the trained model generates a confidence level indicating that the lumen is shown in the first medical image, in a case where the first medical image is input into the trained model, and In a case where the confidence level is equal to or less than a reference value, the output of the lumen specification information (170) that was output before generating the confidence level equal to or less than the reference value is continued. [13] Medical support device (24) according to claim 12, wherein in a case where a state in which the confidence level (158) generated by the trained model is equal to or less than the reference value persists for a predetermined period each time the first medical image is input into the trained model (92), an output level of the lumen specification information that is output before the start of the predetermined period gradually decreases during the predetermined period. [14] Medical support device (24) according to claim 1, wherein the output of the lumen specification information (170) is implemented by displaying the lumen specification information (170) on a screen (35). [15] Medical support device (24) according to claim 14, wherein the medical image (40) and / or an image (157A) corresponding to the medical image (40) is displayed on the screen (35), and the lumen specification information (170) is displayed on the screen (35) in a state in which it is comparable to the medical image (40) or the image (157A) corresponding to the medical image (40) displayed on the screen (35). [16] Medical support device (24) according to claim 15, wherein the lumen specification information (170) is superimposed on the medical image (40) and / or the image (157A) corresponding to the medical image (40). [17] Medical support device (24) according to claim 15, wherein the lumen specification information (170) displayed on the screen (35) is updated in accordance with the display of the medical image (40) and / or the image (157A) corresponding to the medical image (40). [18] Endoscope system (200), comprising: the medical support device (24) according to any one of claims 1 to 17; and an endoscope (16), wherein the medical image (40) is generated by imaging the interior of the luminal organ (28) with the endoscope (16). [19] Medical support procedure, comprehensive: Acquiring a medical image (40) produced by imaging the interior of a luminal organ (28); and selective output of multiple lumen specification information (170), which is information capable of specifying an existence position of a lumen (42) shown in the medical image (40) in the medical image (40), wherein the medical image (40) is classified into a first medical image and a second medical image, which is obtained later than the first medical image, the multiple lumen specification information (170) contain first lumen specification information and second lumen specification information, the first lumen specification information is generated based on information obtained from a trained model (92) in a case where the first medical image is input into the trained model (92), and is information that is able to specify a first existence position, which is the existence position of the lumen (42) shown in the first medical image, in the first medical image, and the second lumen specification information is generated based on information obtained from a time series model (94) in a case in which time series information, which is information relating to one or more of the first lumen specification information obtained in time series, is input into the time series model (94) and is information that is able to specify a second existence position, which is the existence position of the lumen (42) shown in the second medical image. [20] Storage medium (86) storing a program (90) executable by a computer (78) for performing medical support processing, comprising: Acquiring a medical image (40) produced by imaging the interior of a luminal organ (28); and selective output of multiple lumen specification information (170), which is information capable of specifying an existence position of a lumen (42) shown in the medical image (40) in the medical image (40), wherein the medical image (40) is classified into a first medical image and a second medical image, which is obtained later than the first medical image, the multiple lumen specification information (170) contain first lumen specification information and second lumen specification information, the first lumen specification information is generated based on information obtained from a trained model (92) in a case where the first medical image is input into the trained model (92), and is information that is able to specify a first existence position, which is the existence position of the lumen (42) shown in the first medical image, in the first medical image, and the second lumen specification information is generated based on information obtained from a time series model (94) in a case in which time series information, which is information relating to one or more of the first lumen specification information obtained in time series, is input into the time series model (94) and is information that is able to specify a second existence position, which is the existence position of the lumen (42) shown in the second medical image.

Images