Medical support devices, endoscopic systems, medical support methods, and programs

Abstract

The present invention provides a medical support device, an endoscope system, a medical support method, and a program that can estimate the shape of the endoscope's insertion tube when it is inserted into a lumen organ, without using an external device that recognizes the shape of the insertion tube when it is inserted into a lumen organ. [Solution] The medical support device includes a processor. The processor inputs multiple endoscopic images obtained by imaging the inside of a tubular organ with an endoscope inserted into the tubular organ into a trained model, causing the trained model to generate rotation information around the long axis of the insertion part of the endoscope, and generates shape information representing the shape of the insertion part based on the rotation information.

Description

Technical Field

[0001] The present disclosure relates to a medical support device, an endoscope system, a medical support method, and a program.

Background Art

[0002] Patent Document 1 discloses an endoscope shape detection device including attitude detection sensor means in which a plurality of rotation angle detection means that are arranged on the insertion portion of an endoscope, detect the rotation angle of the arranged points, and convert them into electrical signals are arranged on three orthogonal axes, attitude detection means for sampling the output of the attitude detection sensor means at predetermined intervals, and shape detection means for detecting the insertion shape of the endoscope from a plurality of attitude information sampled by the plurality of attitude detection means.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] One embodiment according to the present disclosure provides a medical support device, an endoscope system, a medical support method, and a program that can estimate the shape of the insertion portion of an endoscope when the insertion portion of the endoscope is inserted into a luminal organ without using an external device that recognizes the shape of the insertion portion when the insertion portion of the endoscope is inserted into a luminal organ.

Means for Solving the Problems

[0005] A first aspect according to the present disclosure is a medical support device including a processor, the processor inputs a plurality of endoscope images obtained by imaging the inside of a luminal organ with an endoscope inserted into the luminal organ into a learned model, thereby generating rotation information around the long axis of the insertion portion of the endoscope with respect to the learned model, and generating shape information representing the shape of the insertion portion based on the rotation information.

[0006] A second aspect of this disclosure is a medical support device according to the first aspect, wherein rotation information is obtained based on the cumulative result of the accumulation of changes in characteristic information obtained from endoscope images between multiple endoscope images.

[0007] A third aspect of this disclosure is a medical support device according to the second aspect, wherein the feature information includes lumen position information indicating the position of a lumen included in a tubular organ within an endoscopic image, and the rotation information is obtained based on the cumulative result of the accumulation of changes in the lumen position information between multiple endoscopic images.

[0008] A fourth aspect of this disclosure is a medical support device according to the second or third aspect, wherein the feature information includes gravity direction information that can identify the direction of gravity, and the rotation information is obtained based on the cumulative result of the accumulation of changes in gravity direction information between multiple endoscopic images.

[0009] A fifth aspect of the present disclosure is a medical support device relating to any one of the first to fourth aspects, wherein the rotation information includes information relating to the relative rotation angle and direction of rotation of the second position with respect to a first position of the insertion portion, or information relating to the absolute rotation angle and direction of rotation of the first and second positions with respect to a reference angle.

[0010] A sixth aspect of this disclosure is a medical support device relating to any one of the first to fifth aspects, wherein when the tip position of the insertion portion is rotated by 180 degrees or more relative to the base position of the insertion portion, the shape information includes information indicating that the insertion portion forms a loop.

[0011] A seventh aspect of this disclosure is a medical support device according to the sixth aspect, wherein the rotation information includes rotation direction information that can identify the direction of rotation around a major axis, and the shape information includes loop classification information which is information that classifies the shape of the loop based on the rotation direction information.

[0012] The eighth aspect of this disclosure is a medical support device according to the seventh aspect, wherein the loop classification information includes information classifying loops as alpha loops or inverse alpha loops based on rotation direction information.

[0013] The ninth aspect of this disclosure is a medical support device relating to any one of the sixth to eighth aspects, wherein the processor outputs information on how to release a loop based on rotational information and / or shape information.

[0014] The tenth aspect of this disclosure is a medical support device relating to any one of the first to ninth aspects, wherein a processor inputs a plurality of endoscopic images and a plurality of images including the part of the insertion part that is at the operator's fingertips into a trained model, causing the trained model to generate rotation information, which is information based on rotation around the long axis of the part.

[0015] The eleventh aspect of this disclosure is a medical support device relating to any one of the first to tenth aspects, wherein the tubular organ is the large intestine.

[0016] A twelfth aspect of the present disclosure is a medical support device comprising a processor, wherein the processor inputs a plurality of endoscopic images obtained by imaging the inside of a tubular organ with an endoscope inserted into the tubular organ into a trained model, thereby causing the trained model to generate shape information representing the shape of the insertion portion of the endoscope.

[0017] A thirteenth aspect of this disclosure is a medical support device comprising a processor, which inputs the cumulative result of the cumulative changes between multiple endoscopic images obtained by imaging the inside of a tubular organ with an endoscope inserted into the tubular organ into a trained model, thereby causing the trained model to generate shape information representing the shape of the insertion portion of the endoscope.

[0018] A fourteenth aspect of this disclosure is an endoscope system comprising a medical support device relating to any one of the first to thirteenth aspects, and an output device that outputs shape information generated by the medical support device and / or information based on the shape information generated by the medical support device.

[0019] The 15th aspect according to the present disclosure is a medical support method including: inputting a plurality of endoscopic images obtained by imaging the inside of a luminal organ with an endoscope inserted into the luminal organ into a learned model to cause the learned model to generate rotation information around the long axis of the insertion portion of the endoscope; and generating shape information representing the shape of the insertion portion based on the rotation information.

[0020] The 16th aspect according to the present disclosure is a program for causing a computer to execute a process including: inputting a plurality of endoscopic images obtained by imaging the inside of a luminal organ with an endoscope inserted into the luminal organ into a learned model to cause the learned model to generate rotation information around the long axis of the insertion portion of the endoscope; and generating shape information representing the shape of the insertion portion based on the rotation information.

Brief Description of Drawings

[0021] [Figure 1] It is a conceptual diagram showing an example of an aspect in which an endoscope system is used by a doctor. [Figure 2] It is a conceptual diagram showing an example of the overall configuration of an endoscope system. [Figure 3] It is a block diagram showing an example of the hardware configuration of the electrical system of an endoscope system. [Figure 4] It is a conceptual diagram showing an example of an aspect in which the insertion portion of an endoscope loops inside the large intestine. [Figure 5] It is a conceptual diagram showing an example of the characteristics of an α loop and an inverse α loop. [Figure 6] It is a block diagram showing an example of the main functions of a processor included in a medical support device and an example of information stored in a storage. [Figure 7] It is a block diagram showing an example of the hardware configuration of the electrical system of an information processing device. [Figure 8] It is a conceptual diagram showing an example of an aspect in which teacher data is generated by an information processing device. [Figure 9] It is a conceptual diagram showing an example of a sample image. [Figure 10] This is a conceptual diagram showing an example of teacher data generated when a lumen appears in one of a plurality of divided regions obtained by radially dividing the example image shown in FIG. 9. [Figure 11] This is a conceptual diagram showing an example of the processing content in an information processing apparatus when a lumen recognition model is generated by performing machine learning using teacher data on a model. [Figure 12] This is a block diagram showing an example of the hardware configuration of the electrical system of the information processing apparatus. [Figure 13] This is a conceptual diagram showing an example of the processing content in an information processing apparatus when a rotation recognition model is constructed based on a dataset group. [Figure 14] This is a conceptual diagram showing an example of the processing content of the lumen recognition processing performed by the recognition unit. [Figure 15] This is a conceptual diagram showing an example of the processing content of the rotation recognition processing performed by the recognition unit. [Figure 16] This is a conceptual diagram showing an example of the processing content by the control unit. [Figure 17] This is a flowchart showing an example of the flow of medical support processing. [Figure 18] This is a block diagram showing an example of the main functions of the processor included in the medical support apparatus according to the second embodiment, and an example of the information stored in the storage. [Figure 19] This is a conceptual diagram showing an example of the mode in which teacher data according to the second embodiment is generated. [Figure 20] This is a conceptual diagram showing an example of the processing content in an information processing apparatus when a gravity direction recognition model is generated by performing machine learning using teacher data on a model. [Figure 21] This is a conceptual diagram showing an example of the processing content in an information processing apparatus when the rotation recognition model according to the second embodiment is generated. [Figure 22] This is a conceptual diagram showing an example of the processing content of the gravity direction recognition processing performed by the processor of the medical support apparatus. [Figure 23]This is a conceptual diagram showing an example of the processing content of the rotation recognition process according to the second embodiment, which is performed by the processor of the medical support device. [Figure 24] This is a conceptual diagram showing an example of the processing content in the information processing device when a rotation recognition model according to the third embodiment is generated. [Figure 25] This is a conceptual diagram showing an example of the processing content of the rotation recognition process according to the third embodiment, which is performed by the processor of the medical support device. [Figure 26] This is a conceptual diagram showing an example of the processing content in the information processing device when a shape recognition model according to the fourth embodiment is generated. [Figure 27] This is a conceptual diagram showing an example of the processing content of the shape recognition process according to the fourth embodiment, which is performed by the processor of the medical support device. [Figure 28] This is a conceptual diagram illustrating an example of the processing involved when release method information corresponding to shape information is derived using a release method derivation table. [Figure 29] This is a conceptual diagram showing an example of the processing content in the information processing device when a shape recognition model according to the fifth embodiment is generated. [Figure 30] This is a conceptual diagram showing an example of the processing content of the shape recognition process according to the fifth embodiment, which is performed by the processor of the medical support device. [Figure 31] This is a conceptual diagram illustrating an example of a series of processes in which a processor within a computer sends a processing request to an external device via a network, the external device executes the processing in response to the request, and the processor within the computer receives the processing result from the external device. [Modes for carrying out the invention]

[0022] Hereinafter, examples of embodiments of the medical support device, endoscopic system, medical support method, and program related to this disclosure will be described with reference to the attached drawings. This disclosure is also applicable to programs and computer program products.

[0023] First, let's explain the terminology used in the following explanation.

[0024] CPU stands for "Central Processing Unit". GPU stands for "Graphics Processing Unit". GPGPU stands for "General-Purpose computing on Graphics Processing Units". NPU stands for "Neural Processing Unit". APU stands for "Accelerated Processing Unit". TPU stands for "Tensor Processing Unit". RAM stands for "Random Access Memory". ASIC stands for "Application Specific Integrated Circuit". PLD stands for "Programmable Logic Device". FPGA stands for "Field-Programmable Gate Array". SoC stands for "System-on-a-chip". SSD stands for "Solid State Drive". CD-ROM stands for "Compact Disc Read Only Memory". DVD-ROM stands for "Digital Versatile Disc Read Only Memory". USB stands for "Universal Serial Bus". EL stands for "Electro-Luminescence". CMOS stands for "Complementary Metal Oxide Semiconductor". CCD stands for "Charge Coupled Device". AI stands for "Artificial Intelligence". WLI stands for "White Light Imaging". BLI stands for "Blue Light Imaging". LCI stands for "Linked Color Imaging". NBI stands for "Narrow Band Imaging". I / F stands for "Interface". LAN stands for "Local Area Network".WAN stands for "Wide Area Network." 5G stands for "5th Generation Mobile Communication System."

[0025] In the following description, a signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, a processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU, GPU, GPGPU, NPU, APU, or TPU.

[0026] In the following explanation, signed memory refers to memory such as RAM where information is temporarily stored, and is used as work memory by the processor.

[0027] In the following description, signed storage refers to one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory, magnetic disks, or magnetic tapes. Another example of storage is cloud storage.

[0028] In the following embodiments, a signed external interface (I / F) is responsible for the exchange of various types of information between multiple interconnected devices. An example of an external interface is a USB interface. The external interface may also be a communication interface including a communication processor and an antenna. The communication interface is responsible for communication between multiple computers. An example of a communication standard applicable to the communication interface is a wireless communication standard including 5G, Wi-Fi®, or Bluetooth®.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0030] [First Embodiment] Figure 1 is a conceptual diagram showing an example of how the endoscopic system 10 is used. As shown in Figure 1, the endoscopic system 10 is used by a physician 12 in endoscopic examinations, etc. Endoscopic examinations are assisted by staff 14 such as nurses.

[0031] The endoscope system 10 is connected to a communication device (not shown) in a communicative manner, and information obtained by the endoscope system 10 is transmitted to the communication device. Examples of communication devices include a server for managing various types of information such as electronic medical records, a personal computer, and / or a tablet terminal. The communication device receives the information transmitted from the endoscope system 10 and performs processing using the received information (for example, processing to save it to an electronic medical record, etc.).

[0032] The endoscope system 10 comprises an endoscope 16, a display device 18, a light source device 20, a control device 22, and a medical support device 24. In this first embodiment, the endoscope system 10 is an example of the “endoscope system” according to the disclosure, the endoscope 16 is an example of the “endoscope” according to the disclosure, the display device 18 is an example of the “output device” according to the disclosure, and the medical support device 24 is an example of the “medical support device” according to the disclosure.

[0033] The endoscopic system 10 is a modality for performing medical procedures on the large intestine 28, a tubular organ located inside the body of a subject 26 (e.g., a patient), using an endoscope 16. In this first embodiment, the large intestine 28 is the object to be observed by the physician 12.

[0034] The endoscope 16 is used by a physician 12 and inserted into the body of a subject 26. In this first embodiment, the endoscope 16 is inserted into the large intestine 28 of the subject 26. In this first embodiment, the large intestine 28 is an example of a "luminal organ" according to this disclosure.

[0035] The endoscopic system 10 uses an endoscope 16 inserted into the colon 28 of the subject 26 to image the inside of the colon 28, including the lumen 42, and to perform various medical procedures on the colon 28 as needed.

[0036] The large intestine 28 has a lumen 42. An endoscope 16 is inserted into the lumen 42. The location of the lumen 42 within the large intestine 28 can be medically determined based on the morphological patterns of multiple folds 43, which are characteristic regions within the large intestine 28 (for example, the shape and orientation of the multiple folds 43). As will be described in detail later, in this first embodiment, the location of the lumen 42 is recognized by an AI that has been trained on various information such as the morphological patterns of the multiple folds 43, and the recognition result is provided to the physician 12 as visually understandable information. In this first embodiment, the lumen 42 is an example of a "lumen" as described herein.

[0037] The endoscope system 10 acquires and outputs an image showing the inside of the large intestine 28, including the lumen 42, by imaging the inside of the large intestine 28, including the lumen 42. In this first embodiment, the endoscope system 10 has an optical imaging function that images the reflected light obtained by irradiating the inside of the large intestine 28 with light 30 and reflecting it off the intestinal wall 32 of the large intestine 28.

[0038] While this example illustrates endoscopic examination of the large intestine (28), this is merely one example, and this disclosure also applies to endoscopic examinations of tubular organs such as the esophagus, stomach, duodenum, or trachea.

[0039] The light source device 20, the control device 22, and the medical support device 24 are installed on the wagon 34. The wagon 34 has multiple platforms arranged vertically, with the medical support device 24, the light source device 20, and the control device 22 installed from the lower platform to the upper platform. In addition, the display device 18 is installed on the top platform of the wagon 34.

[0040] The control device 22 controls the entire endoscopy system 10. The control device 22 performs various processing on the images obtained by imaging the intestinal wall 32 with the endoscope 16. The medical support device 24, under the control of the control device 22, performs AI-based processing on the images processed by the control device 22 and outputs various information including the processing results of the AI-based processing. Examples of output destinations for various information include the display device 18, stationary storage media (for example, storage installed in the endoscopy system 10, or storage on a server that is communicatively connected to the endoscopy system 10), and / or portable storage media (for example, a memory card or a USB flash drive).

[0041] The display device 18 displays various information (for example, various information output from the medical support device 24). An example of the display device 18 is a liquid crystal display or an EL display. Alternatively, a tablet terminal with a display may be used instead of the display device 18, or together with the display device 18.

[0042] The display device 18 displays a screen 35. The screen 35 includes multiple display areas. These multiple display areas are arranged side by side within the screen 35. In the example shown in Figure 1, a first display area 35A and a second display area 35B are shown as an example of multiple display areas. The size of the first display area 35A is larger than the size of the second display area 35B. The first display area 35A is used as the main display area, and the second display area 35B is used as a sub-display area. The size relationship between the first display area 35A and the second display area 35B is not limited to this, and any size relationship that fits within the screen 35 is acceptable.

[0043] The first display area 35A displays an endoscopic video 39. The endoscopic video 39 is obtained by performing various processes on multiple images in a time series obtained by imaging the inside of the large intestine 28 of the subject 26 with the endoscope 16. The intestinal wall 32 shown in the endoscopic video 39 includes the lumen 42 as a region of interest (i.e., observation area) that the physician 12 focuses on, and the physician 12 can visually recognize the appearance of the intestinal wall 32, including the lumen 42, through the endoscopic video 39.

[0044] The image displayed in the first display area 35A is one frame 40 included in a video that consists of multiple frames 40 arranged in chronological order. In other words, the first display area 35A displays multiple frames 40 arranged in chronological order at a predetermined frame rate (for example, tens of frames / second or tens of frames / second). In this first embodiment, frame 40 is an example of an "endoscopic image" according to the disclosure.

[0045] An example of a moving image displayed in the first display area 35A is a moving image using the live view method. The live view method is merely one example; the moving image may also be one that is temporarily stored in memory or the like before being displayed, such as a moving image using the post-view method. Alternatively, each frame contained in a recording moving image stored in memory or the like may be reproduced and displayed on the screen 35 (for example, the first display area 35A) as an endoscopic moving image 39.

[0046] The display position of the second display area 35B is the lower right of the front view within the screen 35. The display position of the second display area 35B can be anywhere within the screen 35 of the display device 18, but it is preferable that it be displayed in a position that can be compared with the endoscopic video 39. The second display area 35B displays auxiliary information 44 that assists the physician 12 in making medical judgments during an endoscopic examination. The auxiliary information 44 is information that the physician 12 refers to. Examples of auxiliary information 44 include various information about the subject 26 into which the endoscope 16 is inserted, and / or various information obtained through medical support processing described later.

[0047] Figure 2 is a conceptual diagram showing an example of the overall configuration of the endoscope system 10. As shown in Figure 2, the endoscope 16 comprises an operating section 46 and an insertion section 48. The insertion section 48 is formed in a tubular shape and partially bends when the operating section 46 is operated. The insertion section 48 is inserted into the large intestine 28 (see Figure 1) while bending according to the shape of the large intestine 28, in accordance with the operation of the operating section 46 by the physician 12 (see Figure 1). In this first embodiment, the insertion section 48 is an example of the "insertion section" according to this disclosure.

[0048] The tip 50 of the insertion section 48 is provided with a camera 52, an illumination device 54, and an opening 56 for a treatment instrument. Part of the camera 52 (e.g., the imaging optical system) and part of the illumination device 54 (e.g., the irradiation optical system) are exposed from the tip surface 50A of the tip 50.

[0049] The camera 52 is mounted on the endoscope 16 and is inserted into the body cavity of the subject 26 (in this case, the lumen 42 as an example) to image the area to be observed. An example of the camera 52 is a CMOS camera. However, this is merely an example, and other types of cameras such as CCD cameras may also be used. In this first embodiment, the camera 52 images the inside of the large intestine 28, including the lumen 42, thereby generating an image showing the inside of the large intestine 28 including the lumen 42. The image generated by the camera 52 is an image with a circular shape. For example, the image generated by the camera 52 is processed by the control device 22 into a shape in which the upper and lower ends are masked. As a result, as shown in Figure 1, an image is generated as frame 40 in which the upper and lower edges are straight lines and the left and right edges are arc-shaped.

[0050] The illumination device 54 has illumination windows 54A and 54B. The illumination windows 54A and 54B are provided on the tip surface 50A. The illumination device 54 irradiates light 30 (see Figure 1) through the illumination windows 54A and 54B. Examples of the types of light 30 irradiated from the illumination device 54 include light for WLI (e.g., white light), light for LCI (e.g., a combination of red, green, and blue light), light for BLI (e.g., blue light), and / or light for NBI (e.g., a combination of blue and green light). The camera 52 optically images the inside of the large intestine 28 while the large intestine 28 is irradiated with light 30 (see Figure 1) by the illumination device 54.

[0051] The treatment instrument opening 56 is an opening for allowing the treatment instrument 58 to protrude from the tip portion 50. The treatment instrument opening 56 is also used as a suction port for aspirating blood and bodily waste, and as an outlet for discharging fluids. Examples of fluids include gases (e.g., air) and / or liquids (e.g., water).

[0052] The operating section 46 has a treatment instrument insertion port 60, and the treatment instrument 58 is inserted into the insertion section 48 through the treatment instrument insertion port 60. The treatment instrument 58 passes through the insertion section 48 and protrudes to the outside through the treatment instrument opening 56. In the example shown in Figure 2, a puncture needle is shown as the treatment instrument 58 protruding from the treatment instrument opening 56. Here, a puncture needle is used as an example of the treatment instrument 58, but this is merely an example, and the treatment instrument 58 may be a grasping forceps, a papillotome knife, a snare, a catheter, a guidewire, a cannula, and / or a puncture needle with a guide sheath, etc.

[0053] The endoscope 16 is connected to the light source device 20 and the control device 22 via a universal code 62. The control device 22 is connected to the medical support device 24 and the reception device 64. The medical support device 24 is connected to the display device 18. In other words, the control device 22 is connected to the display device 18 via the medical support device 24.

[0054] Here, the medical support device 24 is exemplified as an external device for extending the functions performed by the control device 22, and therefore, an example is given in which the control device 22 and the display device 18 are indirectly connected via the medical support device 24. However, this is merely one example. For example, the display device 18 may be directly connected to the control device 22. In this case, for example, the functions of the medical support device 24 may be incorporated into the control device 22, or the control device 22 may be equipped with a function that causes a server (not shown) to perform the same processing as the processing performed by the medical support device 24 (for example, the medical support processing described later), and receives and uses the processing results from the server.

[0055] The reception device 64 receives instructions from the doctor 12 and outputs the received instructions as electrical signals to the control device 22. Examples of the reception device 64 include a keyboard, mouse, touch panel, foot switch, microphone, and / or remote control device.

[0056] The control device 22 controls the light source device 20, exchanges various signals with the camera 52, and exchanges various signals with the medical support device 24.

[0057] The light source device 20 emits light under the control of the control device 22 and supplies light 30 (see Figure 1) to the illumination device 54. The illumination device 54 has a built-in light guide, and the light 30 supplied from the light source device 20 is irradiated through the illumination windows 54A and 54B via the light guide. The control device 22 causes the camera 52 to take images while light 30 is irradiated from the illumination windows 54A and 54B. The control device 22 generates multiple frames 40 in chronological order by processing the outline of the images obtained by the camera 52 and adjusting the image quality, etc. Then, the control device 22 outputs the endoscopic video 39, which includes the generated multiple frames 40 in chronological order, to a predetermined output destination (for example, the medical support device 24).

[0058] The medical support device 24 performs various processes on the endoscopic video 39 input from the control device 22 to support medical procedures (in this case, endoscopic examination as an example). The medical support device 24 outputs the processed endoscopic video 39 to a predetermined output destination (for example, the display device 18).

[0059] It should be noted that, although this explanation describes an example in which the endoscopic video 39 output from the control device 22 is output to the display device 18 via the medical support device 24, this is merely one example. For example, the control device 22 and the display device 18 may be connected, and the endoscopic video 39, which has undergone various processing by the medical support device 24, may be displayed on the display device 18 via the control device 22.

[0060] Figure 3 is a block diagram showing an example of the electrical hardware configuration of the endoscope system 10. As shown in Figure 3, the control unit 22 comprises a computer 66, a bus 68, and an external interface 70. The computer 66 comprises a processor 72, memory 74, and storage 76. The processor 72, memory 74, storage 76, and external interface 70 are connected to the bus 68. The processor 72 controls the entire control unit 22. The memory 74 and storage 76 are used by the processor 72.

[0061] The external I / F 70 is responsible for the exchange of various types of information between the processor 72 and one or more devices located outside the control device 22 (hereinafter also referred to as the "first external device").

[0062] A camera 52 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 is responsible for the exchange of various information between the camera 52 and the processor 72. The processor 72 controls the camera 52 via the external I / F 70. The processor 72 also acquires images generated by the imaging of the inside of the large intestine 28 (see Figure 1) by the camera 52 via the external I / F 70, and generates endoscopic moving images 39 (see Figure 1) by performing various processing on the acquired images.

[0063] The light source device 20 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 is responsible for the exchange of various information between the light source device 20 and the processor 72. The light source device 20 supplies light 30 to the illumination device 54 under the control of the processor 72. The illumination device 54 irradiates with the light 30 supplied from the light source device 20.

[0064] A receiving device 64 is connected to the external I / F 70 as one of the first external devices. The processor 72 receives instructions from the receiving device 64 via the external I / F 70 and executes processing according to the received instructions.

[0065] The medical support device 24 includes a computer 78 and an external I / F 80. The computer 78 includes a processor 82, memory 84, and storage 86. The processor 82, memory 84, storage 86, and external I / F 80 are connected to a bus 88. In this first embodiment, the computer 78 is an example of a “computer” according to the disclosure, and the processor 82 is an example of a “processor” according to the disclosure.

[0066] Since the hardware configuration of computer 78 (i.e., processor 82, memory 84, and storage 86) is basically the same as that of computer 66, a description of the hardware configuration of computer 78 will be omitted here.

[0067] The external I / F 80 is responsible for the exchange of various types of information between the processor 82 and one or more devices located outside the medical support device 24 (hereinafter also referred to as the "second external device").

[0068] The control device 22 is connected to the external I / F 80 as one of the second external devices. In the example shown in Figure 3, the external I / F 70 of the control device 22 is connected to the external I / F 80. The external I / F 80 is responsible for the exchange of various information between the processor 82 of the medical support device 24 and the processor 72 of the control device 22. For example, the processor 82 acquires endoscopic video images 39 (see Figure 1) from the processor 72 of the control device 22 via the external I / F 70 and 80, and performs various processing on the acquired endoscopic video images 39. The various processing performed by the processor 82 includes AI-based processing (for example, lumen recognition processing 166, which uses the lumen recognition model 92 described later, and rotation recognition processing 172, which uses the rotation recognition model 94).

[0069] A display device 18 is connected to the external I / F 80 as one of the second external devices. The processor 82 controls the display device 18 via the external I / F 80 to display various information (for example, endoscopic video images 39 that have undergone various processing) on ​​the display device 18.

[0070] Incidentally, the large intestine 28 has a complex shape, and it can be difficult to insert the insertion part 48. For example, as shown in Figure 4, in the sigmoid colon and transverse colon, which are not fixed to the abdominal wall, the insertion part 48 may form a loop due to the shape of the intestinal tract, the pressure of the intestinal tract, and the physical effects on the large intestine 28 when the insertion part 48 is manipulated. When such a loop is formed, it becomes difficult to insert the insertion part 48 into the deeper part of the large intestine 28. If insertion of the insertion part 48 becomes difficult, it also places a physical burden on the subject 26. Therefore, it is very important to allow the physician 12 to know in real time what shape the insertion part 48 is in inside the large intestine 28.

[0071] As a device to allow physicians 12 to understand the shape of the insertion portion 48 within the colon 28, an endoscopic shape observation device is known that combines a dedicated scope that generates a magnetic field with an external device. Although this endoscopic shape observation device can display the shape of the insertion portion 48 in real time on a display, it requires a dedicated scope and an external device, which makes the preparation and operation of the equipment time-consuming and lacks convenience.

[0072] Furthermore, conventional methods have the drawback of making it difficult to specifically identify the loop shape of the insertion portion 48 within the large intestine 28. Typical loop shapes include α-loops and inverse α-loops. For example, as shown in Figure 5, an α-loop is formed when the insertion portion 48 rotates clockwise by 180 degrees or more around its long axis (i.e., around the long axis when viewed from the proximal end to the anterior end of the insertion portion 48 along its long axis) within the large intestine 28. On the other hand, an inverse α-loop is formed when the insertion portion 48 rotates counterclockwise by 180 degrees or more around its long axis within the large intestine 28. α-loops and inverse α-loops mainly occur in highly mobile parts of the intestine such as the sigmoid colon and transverse colon.

[0073] Thus, while both the case where an α-loop is formed and the case where an inverse α-loop is formed involve rotation of 180 degrees or more around the long axis of the insertion portion 48, the direction of rotation around the long axis of the insertion portion 48 differs. When an α-loop is formed, the multiple frames 40 obtained in the process leading up to the formation of the α-loop rotate counterclockwise by 180 degrees or more around the center of the frame 40. On the other hand, when an inverse α-loop is formed, the multiple frames 40 obtained in the process leading up to the formation of the inverse α-loop rotate clockwise by 180 degrees or more around the center of the frame 40.

[0074] Therefore, in this first embodiment, in order to solve the above-mentioned problems, medical support processing is performed by the processor 82 by utilizing the fact that different phenomena occur on frame 40 when an α loop is formed and when an inverse α loop is formed, as shown in Figure 6 as an example.

[0075] Figure 6 is a block diagram showing an example of the essential functions of the processor 82 included in the medical support device 24, and an example of the information stored in the storage 86. As shown in Figure 6, the storage 86 stores a medical support program 90. In this first embodiment, the medical support program 90 is an example of the "program" according to this disclosure.

[0076] The processor 82 reads the medical support program 90 from the storage 86 and performs medical support processing by executing the read medical support program 90 on the memory 84. The medical support processing is realized by the recognition unit 82A and the control unit 82B operating according to the medical support program 90 executed by the processor 82 on the memory 84.

[0077] Storage 86 stores the lumen recognition model 92, the rotation recognition model 94, and the information derivation table 96. As will be described in detail later, the lumen recognition model 92 and the rotation recognition model 94 are machine learning models and are used by the recognition unit 82A. An example of a machine learning model is a neural network (e.g., a recurrent neural network, a two-dimensional convolutional neural network, and / or a three-dimensional convolutional neural network). The information derivation table 96 is used by the control unit 82B. An example of the information derivation table 96 is a lookup table in which pairs of input values ​​and corresponding output values ​​are represented as a predefined table.

[0078] Figure 7 is a block diagram showing an example of the electrical hardware configuration of an information processing device 100 used to generate a lumen recognition model 92 and a rotation recognition model 94. As shown in Figure 7, the information processing device 100 includes a computer 102 and an external interface 104. The computer 102 includes a processor 106, memory 108, and storage 110. The processor 106, memory 108, storage 110, and external interface 104 are connected to a bus 112.

[0079] Since the hardware configuration of computer 102 (i.e., processor 106, memory 108, and storage 110) is basically the same as that of computer 66, a description of the hardware configuration of computer 102 will be omitted here.

[0080] The information processing device 100 is equipped with a receiving device 116. The receiving device 116 is a keyboard and / or mouse, etc., and receives instructions from the user of the information processing device 100. The receiving device 116 is connected to the bus 112. The processor 106 receives instructions received by the receiving device 116 and operates according to the received instructions.

[0081] The display device 118 displays various information, including images. An example of the display device 118 is a liquid crystal display or an EL display. The display device 118 is connected to the bus 112. The processor 106 performs various processes and displays the results obtained on the display device 118.

[0082] The external interface 104 controls the exchange of various types of information between the processor 106 and one or more devices located outside the information processing device 100 (hereinafter also referred to as the "third external device"). The medical support device 24 is connected to the external interface 104 as one of the third external devices. In the example shown in Figure 7, the external interface 80 of the medical support device 24 is connected to the external interface 104. The external interface 104 controls the exchange of various types of information between the processor 82 of the medical support device 24 (see Figures 3 and 6) and the processor 106 of the information processing device 100. For example, the information processing device 100 generates a lumen recognition model 92 and a rotation recognition model 94, and transmits the generated lumen recognition model 92 and rotation recognition model 94 to the medical support device 24 via the external interfaces 80 and 104 in response to a request from the medical support device 24.

[0083] The storage 110 stores the first machine learning processing program 120. The processor 106 reads the first machine learning processing program 120 from the storage 110 and performs the first machine learning processing by executing the read first machine learning processing program 120 on the memory 108. The first machine learning processing is realized by the processor 106 operating as a training data generation unit 106A and a first learning execution unit 106B according to the first machine learning processing program 120 executed on the memory 108.

[0084] The storage 110 contains the example image set 122. As will be explained in more detail later, the example image set 122 is used by the training data generation unit 106A.

[0085] Figure 8 is a conceptual diagram showing an example of the processing content of the training data generation unit 106A. As shown in Figure 8, the information processing device 100 is used by the annotator 124. The annotator 124 refers to an operator who adds annotations for machine learning to the given data (i.e., an operator who performs labeling).

[0086] In the example shown in Figure 8, a keyboard 116A and a mouse 116B are shown as an example of the reception device 116. The annotator 124 gives instructions to the computer 102 via the keyboard 116A and mouse 116B.

[0087] The example image set 122 includes multiple example images 122A, each containing different content. An example image 122A is a predefined medical image used for object recognition processing (for example, the process by which the recognition unit 82A recognizes the lumen 42 based on frame 40 and lumen recognition model 92). A predefined medical image used for object recognition processing is an image corresponding to frame 40. In other words, an image corresponding to frame 40 can also be said to be an image that represents frame 40. To put it another way, an image that represents frame 40 can also be said to be an image that shows a sample of frame 40. Here, a first example of an image that shows a sample of frame 40 is an image obtained by actually imaging the inside of the large intestine with a camera. A second example of an image that shows a sample of frame 40 is a virtually created image (for example, an image generated by a generative AI such as Stable Diffusion or Midjourney).

[0088] The training data generation unit 106A acquires example image 122A from the example image set 122 according to instructions received by the reception device 116. The training data generation unit 106A displays example image 122A on the screen 118A of the display device 118. With example image 122A displayed on screen 118A, the annotator 124 instructs the training data generation unit 106A via the reception device 116 to specify the lumen-corresponding position, which is the position of the lumen within example image 122A that is depicted in example image 122A. Based on the lumen-corresponding position instructed via the reception device 116, the training data generation unit 106A generates training data 128 by associating correct answer data 126 with example image 122A. Associating the correct answer data 126 with the example image 122A is achieved by assigning annotations that allow for the identification of lumen-corresponding locations within the example image 122A to the correct answer data 126.

[0089] In this way, the training data generation unit 106A generates multiple training data sets 128 by repeatedly performing the process of associating each of the example images 122A included in the example image set 122 with the correct answer data 126 according to the instructions given by the annotator 124.

[0090] Figure 9 is a conceptual diagram showing an example of the structure of example image 122A. As shown in Figure 9, example image 122A shows the inside of the large intestine 132. In the example shown in Figure 9, the intestinal wall 136 with multiple folds 134 and the lumen 138 are shown in example image 122A.

[0091] Example image 122A is divided into multiple partitioned regions 130A. These partitioned regions 130A include eight partitioned regions 130A1 to 130A8. The partitioned regions 130A1 to 130A8 are regions that radiate from the center C1 of example image 122A toward the outer edge of example image 122A, and are arranged along the circumferential direction CD1 of example image 122A (in other words, around the center C1).

[0092] Figure 10 is a conceptual diagram showing an example of how the training data generation unit 106A generates training data 128 by associating the correct answer data 126 with the example image 122A.

[0093] As shown in Figure 10, with the example image 122A displayed on screen 118A, the annotator 124 instructs the training data generation unit 106A via the receiving device 116 to specify the lumen corresponding position 139, which is the position of the lumen 138 shown in the example image 122A. The training data generation unit 106A superimposes a circular frame 140 onto the example image 122A according to the instructions received by the receiving device 116, and positions the frame 140 to surround the lumen 138 shown in the example image 122A. The frame 140 is a mark that defines the lumen corresponding position 139 within the example image 122A. That is, the position of the area enclosed by the frame 140 within the example image 122A is the lumen corresponding position 139. The size and position of the frame 140 can be freely changed on screen 118A according to the instructions received by the receiving device 116. Here, the frame 140 is circular in shape, but it may be a shape other than a circle. Also, the size of the frame 140 can be changed according to the instructions received by the receiving device 116.

[0094] The annotator 124, with the frame 140 positioned around the lumen 138, provides a confirmation instruction to the teacher data generation unit 106A via the receiving device 116, which instructs the unit to confirm the lumen-corresponding position 139. As a result, the teacher data generation unit 106A confirms the lumen-corresponding position 139.

[0095] The training data generation unit 106A identifies the divisional region 130A with the largest overlapping area with the frame 140 that defines the lumen-corresponding position 139 from among multiple divisional regions 130A. Then, the training data generation unit 106A generates training data 128 by associating the correct data 126 as an annotation that can identify the divisional region 130A in which the lumen 138 is visible with the identified divisional region 130A (in the example shown in Figure 10, divisional region 130A2).

[0096] Figure 11 is a conceptual diagram showing an example of how a lumen recognition model 92 is generated by machine learning using training data 128 performed by the first learning execution unit 106B.

[0097] As shown in Figure 11, in the information processing device 100, the first learning execution unit 106B acquires the training data 128 generated by the training data generation unit 106A. Then, the first learning execution unit 106B performs machine learning using the training data 128. The following will be explained in detail with reference to Figure 11.

[0098] In the example shown in Figure 11, the first learning execution unit 106B performs processing using model 142. An example of model 142 is a neural network. Examples of neural networks include a recurrent neural network, a two-dimensional convolutional neural network, and / or a three-dimensional convolutional neural network. The first learning execution unit 106B inputs the example image 122A included in the training data 128 to model 142. Upon receiving the example image 122A, model 142 performs inference and outputs the inference result 144. The first learning execution unit 106B calculates the error 146 between the inference result 144 and the correct answer data 126 included in the training data 128.

[0099] The first learning execution unit 106B calculates a number of adjustment values ​​148 that minimize the error 146. Then, the first learning execution unit 106B optimizes the model 142 by adjusting a number of optimization variables within the model 142 using the number of adjustment values ​​148. Examples of the number of optimization variables within the model 142 include weights (in other words, connection weights) that indicate the strength of the connections between layers (i.e., the strength of the connections between neurons), and biases (in other words, offset values) that control the activation of neurons (i.e., values ​​used to adjust the output of neurons).

[0100] The first learning execution unit 106B repeatedly performs the learning process of inputting example image 122A into model 142, calculating error 146, calculating multiple adjustment values ​​148, and adjusting multiple optimization variables within model 142 using multiple training data 128. That is, for each of the multiple example image 122A included in the multiple training data 128, the first learning execution unit 106B optimizes model 142 by adjusting multiple optimization variables within model 142 using multiple adjustment values ​​148 calculated to minimize the error 146. The lumen recognition model 92 is generated by optimizing model 142 in this way. The lumen recognition model 92 is transmitted from the information processing device 100 to the medical support device 24 via external I / F 80 and 104 (see Figure 7) and received by the medical support device 24. Then, in the medical support device 24, the lumen recognition model 92 is stored in storage 86 by the processor 82 (see Figure 6). The lumen recognition model 92 stored in storage 86 is used by the recognition unit 82A (see Figure 6).

[0101] As an example, as shown in Figure 12, in the information processing device 100, the storage 110 stores the second machine learning processing program 150. The processor 106 reads the second machine learning processing program 150 from the storage 110 and performs the second machine learning processing by executing the read second machine learning processing program 150 on the memory 108. The second machine learning processing is realized by the processor 106 operating as a second learning execution unit 106C according to the second machine learning processing program 150 executed on the memory 108.

[0102] Furthermore, the storage 110 stores the dataset group 152. The dataset group 152 is used by the second learning execution unit 106C.

[0103] Figure 13 shows a conceptual diagram illustrating an example of how a rotation recognition model 94 is generated by machine learning using the dataset group 152 performed by the second learning execution unit 106C.

[0104] As shown in Figure 13, the dataset group 152 is a collection of multiple datasets 152A. The contents of the multiple datasets 152A are different from each other. Dataset 152A is training data in which the lumen position information set 152A1 and the ground truth data 152A2 are associated. In the same manner as the training data 128 shown in Figure 8 is generated by the training data generation unit 106A, the generation of dataset 152A is achieved by associating the ground truth data 152A2 with the lumen position information set 152A1.

[0105] The correct answer data 152A2 is data that can identify the rotation direction and rotation angle around the long axis of the insertion part of the endoscope (for example, the same endoscope as endoscope 16) when an α-loop is formed, data that can identify the rotation direction and rotation angle around the long axis of the insertion part of the endoscope when an inverse α-loop is formed, or data that can identify the rotation direction and rotation angle around the long axis of the insertion part of the endoscope when no loop is formed.

[0106] The lumen position information set 152A1 includes multiple lumen position information 152A1a (here, as an example, three or more lumen position information 152A1a) arranged in chronological order. The lumen position information 152A1a is one of the first to third types of information. The first type of information is information that can identify the position of the lumen within each frame in multiple frames included in endoscopic video images obtained when the inside of the large intestine is imaged by the endoscope during the process until the α-loop of the endoscope insertion part is formed during an endoscopic examination. The second type of information is information that can identify the position of the lumen within each frame in multiple frames included in endoscopic video images obtained when the inside of the large intestine is imaged by the endoscope during the process until the inverse α-loop of the endoscope insertion part is formed during an endoscopic examination. The third type of information is information that can identify the position of the lumen within each frame in multiple frames included in endoscopic video images obtained when the inside of the large intestine is imaged by the endoscope when the loop of the endoscope insertion part is not formed during an endoscopic examination.

[0107] Lumen position information 152A1a is information that allows the location of any one of the eight divided regions 154 to be identified as the location where the lumen is visible. The eight divided regions 154 are obtained by dividing the frame included in the endoscopic video image into eight parts, in the same manner as obtaining the eight divided regions 130A. The eight divided regions 154 are arranged at 45-degree intervals along the center C2 of the frame included in the endoscopic video image.

[0108] The lumen position information set 152A1 contains one of the first to third time-series information. The first time-series information is information obtained during an endoscopic examination from the time the insertion part of the endoscope is inserted into the large intestine until the α-loop of the insertion part is formed, with the lumen position information 152A1a arranged in time series. The first time-series information can also be described as the cumulative result of the change in the position of the lumen over multiple frames until the position of the lumen has moved at least halfway around the eight divided regions 154 counterclockwise.

[0109] The second time-series information is a time-series arrangement of lumen position information 152A1a obtained for a number of frames during an endoscopic examination, from the time the insertion part of the endoscope is inserted into the large intestine until the inverse alpha loop of the insertion part is formed. The second time-series information can also be described as the cumulative result of the changes in the position of the lumen over multiple frames until the position of the lumen has completed at least half a clockwise rotation through the eight divided regions 154.

[0110] The third time-series information is a time-series arrangement of lumen position information 152A1a for a predetermined number of frames (for example, the number of frames corresponding to statistical values ​​such as the average, median, mode, maximum, or minimum number of frames obtained until an α-loop or inverse α-loop is formed) in an endoscopic examination, from the time the insertion part of the endoscope is inserted into the large intestine until a loop of the insertion part is formed.

[0111] Here, for the lumen position information set 152A1 which contains the first time-series information, data that can identify the rotation direction (e.g., clockwise) and rotation angle (e.g., an angle of 180 degrees or more) around the long axis of the insertion part of the endoscope when an α-loop is formed is associated as ground truth data 152A2.

[0112] Furthermore, for the lumen position information set 152A1, which includes second time-series information, data is associated as ground truth data 152A2 that can identify the rotation direction (e.g., counterclockwise) and rotation angle (e.g., an angle of 180 degrees or more) around the long axis of the endoscope insertion section when an inverse alpha loop is formed.

[0113] Furthermore, for the lumen position information set 152A1 which includes third time-series information, the correct data 152A2 is associated with data that can identify the direction of rotation (e.g., clockwise, counterclockwise, or neither clockwise nor counterclockwise) and the angle of rotation (e.g., an angle less than 180 degrees) when no loop is formed.

[0114] In the information processing device 100, the second learning execution unit 106C performs machine learning using the dataset group 152. This will be explained in detail below with reference to Figure 13.

[0115] The second learning execution unit 106C performs processing using model 156. An example of model 156 is a neural network similar to model 142 shown in Figure 11. The second learning execution unit 106C obtains dataset 152A from dataset group 152. Then, the second learning execution unit 106C inputs multiple lumen position information 152A1a, which are arranged in time series within the lumen position information set 152A1 included in dataset 152A obtained from dataset group 152, into model 156 in chronological order. When model 156 receives multiple lumen position information 152A1a arranged in time series, it performs inference and outputs the inference result 158. The second learning execution unit 106C calculates the error 160 between the inference result 158 ​​and the ground truth data 152A2 included in dataset 152A obtained from dataset group 152.

[0116] The second learning execution unit 106C calculates multiple adjustment values ​​162 that minimize the error 160. Then, the second learning execution unit 106C optimizes the model 156 by adjusting multiple optimization variables within the model 156 using the multiple adjustment values ​​162. Examples of multiple optimization variables within the model 156 include weights and biases.

[0117] The second learning execution unit 106C repeatedly performs the learning process of inputting the lumen position information set 152A1 into the model 156, calculating the error 160, calculating multiple adjustment values ​​162, and adjusting multiple optimization variables within the model 156, using all datasets 152A included in the dataset group 152. That is, for each of the lumen position information sets 152A1 included in all datasets 152A, the second learning execution unit 106C optimizes the model 156 by adjusting multiple optimization variables within the model 156 using multiple adjustment values ​​162 calculated to minimize the error 160. The rotation recognition model 94 is generated by optimizing the model 156 in this way. The rotation recognition model 94 is transmitted from the information processing device 100 to the medical support device 24 via external I / F 80 and 104 (see Figure 7) and received by the medical support device 24. The medical support device 24 then stores the rotation recognition model 94 in storage 86 by the processor 82 (see Figure 6). The rotation recognition model 94 stored in storage 86 is used by the recognition unit 82A (see Figure 6).

[0118] Figure 14 shows an example of the processing performed by the recognition unit 82A. As shown in Figure 14, the image 164 obtained by the camera 52, which images the intestinal wall 32 inside the large intestine 28 including the lumen 42, is acquired by the recognition unit 82A. The recognition unit 82A generates a frame 40 by performing various processing on the image 164. In the example shown in Figure 14, the frame 40 shows the intestinal wall 32 with folds 43 and the lumen 42.

[0119] The control unit 82B acquires the frame 40 from the recognition unit 82A and displays the acquired frame 40 in the first display area 35A.

[0120] The recognition unit 82A performs a lumen recognition process 166 on the frame 40. The lumen recognition process 166 is a process that recognizes the lumen 42 shown in the frame 40 by using the lumen recognition model 92 stored in the storage 86 (in other words, a process that identifies the location of the lumen 42 shown in the frame 40 within the frame 40 by using the lumen recognition model 92).

[0121] The recognition unit 82A inputs the frame 40 to the lumen recognition model 92, causing the lumen recognition model 92 to generate lumen position information 168. The lumen position information 168 is information indicating the position of the lumen 42 contained in the large intestine 28 within the frame 40. The lumen position information 168 is information that allows the position of any one of the eight divided regions 170 to be identified as the position where the lumen is captured. The eight divided regions 170 are obtained by dividing the frame 40 into eight sections in the same manner as obtaining the eight divided regions 130A. The eight divided regions 170 are arranged at 45-degree intervals around the center C3 of the frame 40. In this first embodiment, the lumen position information 168 is an example of the "feature information" and "lumen position information" related to this disclosure.

[0122] As an example, as shown in Figure 15, the recognition unit 82A maintains multiple lumen position information 168 in a time series. Here, the multiple lumen position information 168 in a time series can also be described as the cumulative result (for example, the accumulated cumulative result) of the changes in feature information (i.e., feature-representing information) obtained from the frame 40 across multiple frames 40. The recognition unit 82A generates rotation information 174 based on the cumulative result of the changes in feature information obtained from the frame 40 across multiple frames 40. Here, an example of the cumulative result of the changes in feature information obtained from the frame 40 across multiple frames 40 is the multiple lumen position information 168 in a time series obtained by the recognition unit 82A performing a lumen recognition process 166 (see Figure 14) on each of the multiple frames 40 obtained in a time series.

[0123] The recognition unit 82A performs rotation recognition processing 172 on multiple lumen position information 168 in a time series (here, as an example, three or more lumen position information 168). Rotation recognition processing 172 is a process that recognizes rotation information 174 from multiple lumen position information 168 in a time series by using a rotation recognition model 94 stored in storage 86 (in other words, a process that identifies the rotation direction and rotation angle around the long axis of the insertion part 48 by using the rotation recognition model 94).

[0124] The recognition unit 82A inputs a plurality of lumen position information 168 in a time series to the rotation recognition model 94, thereby causing the rotation recognition model 94 to generate rotation information 174. In this first embodiment, the lumen recognition model 92 and the rotation recognition model 94 are examples of "trained models" according to this disclosure. Also, in this first embodiment, the rotation information 174 is an example of "rotation information" according to this disclosure.

[0125] The rotation information 174 includes information that can identify the rotation angle (for example, clockwise, counterclockwise, or neither clockwise nor counterclockwise) and rotation direction of the insertion part 48 around its major axis. The rotation direction and rotation angle around the major axis of the insertion part 48 refer to the relative rotation angle and rotation direction of the insertion part 48 at the second position with respect to the first position of the insertion part 48, or the absolute rotation angle and rotation direction of the insertion part 48 at the first and second positions with respect to a reference angle (for example, the angle of gravity recognized by AI-type or non-AI-type image recognition processing, or a predetermined angle). Here, the first position is the reference position of the insertion part 48. An example of the reference position of the insertion part 48 is the position of the insertion part 48 held by the physician 12. The second position is a position that is compared with the first position. An example of a position that is compared with the first position is the tip surface 50A shown in Figure 2.

[0126] As an example, as shown in Figure 16, the information derivation table 96 stored in the storage 86 is a table that associates rotation information 174 with shape information 176 and release method information 178 corresponding to the rotation information 174. The shape information 176 is information that represents the shape of the insertion part 48 within the large intestine 28 (e.g., α-loop, inverse α-loop, or non-loop). For example, if the tip position of the insertion part 48 is rotated by 180 degrees or more relative to the proximal end position of the insertion part 48 (e.g., the position of the insertion part 48 that is grasped by the physician 12), the shape information 176 indicates that the insertion part 48 forms a loop. The shape information 176 when the tip position of the insertion part 48 is rotated by 180 degrees or more relative to the proximal end position of the insertion part 48 includes loop classification information, which is information that classifies the shape of the loop based on the direction of rotation identified from the rotation information 174. An example of loop classification information is information that classifies the loop of the insertion part 48 as an α-loop or an inverse α-loop based on the direction of rotation identified from the rotation information 174.

[0127] To explain this more specifically, when the tip position of the insertion part 48 is rotated clockwise by 180 degrees or more around the long axis of the insertion part 48 relative to the base position of the insertion part 48, the shape information 176 indicates that the insertion part 48 forms an α-loop. Similarly, when the tip position of the insertion part 48 is rotated counterclockwise by 180 degrees or more around the long axis of the insertion part 48 relative to the base position of the insertion part 48, the shape information 176 indicates that the insertion part 48 forms an α-loop. Furthermore, when the tip position of the insertion part 48 is not rotated by 180 degrees or more relative to the base position of the insertion part 48, the shape information 176 indicates that the insertion part 48 does not form a loop.

[0128] The release method information 178 is information regarding a method for releasing a loop when the shape represented by the shape information 176 associated with the rotation information 174 is an α-loop or an inverse α-loop.

[0129] The control unit 82B acquires rotation information 174 from the recognition unit 82A. The control unit 82B then generates shape information 176 and release method information 178 from the recognition unit 82A based on the rotation information 174. In the example shown in Figure 16, the control unit 82B derives shape information 176 and release method information 178 corresponding to the rotation information 174 from the information derivation table 96 by referring to the information derivation table 96 stored in the storage 86. In this first embodiment, shape information 176 is an example of "shape information" in this disclosure, and release method information 178 is an example of "information relating to a loop release method" and "information based on shape information" in this disclosure.

[0130] In the example shown in Figure 16, the release method information 178 is directly derived from the rotation information 174. However, this is merely one example, and the shape information 176 may be derived first from the rotation information 174, and then the release method information 178 may be derived from the shape information 176. In this case, for example, first, the control unit 82B refers to a first table in which the rotation information 174 and the shape information 176 are associated, and derives the shape information 176 corresponding to the rotation information 174 from the first table. Then, the control unit 82B refers to a second table in which the shape information 176 and the release method information 178 are associated, and derives the release method information 178 corresponding to the derived shape information 176 from the second table. Alternatively, the release method information 178 may be derived from both the rotation information 174 and the shape information 176. In this case, the control unit 82B derives the release method information 178 corresponding to the derived rotation information 174 and shape information 176 from the third table by referring to a third table in which both rotation information 174 and shape information 176 are associated with release method information 178.

[0131] The control unit 82B performs display control on the display device 18 based on rotation information 174, shape information 176, and release method information 178, etc. Specifically, the control unit 82B displays the rotation information 174 acquired from the recognition unit 82A as visible information in the second display area 35B, and also displays the shape information 176 and release method information 178 derived from the information derivation table 96 as visible information in the second display area 35B. In the example shown in Figure 16, the rotation angle and rotation direction specified by the rotation information 174 are displayed in text as part of the auxiliary information 44 in the second display area 35B. In addition, the type of shape represented by the shape information 176 is displayed in text and a schematic diagram as part of the auxiliary information 44 in the second display area 35B. Furthermore, the release method indicated by the release method information 178 is displayed in text as part of the auxiliary information 44 in the second display area 35B.

[0132] Here, a visible display using the display device 18 is shown as an example of outputting rotation information 174, shape information 176, and release method information 178, but this is merely one example. For example, rotation information 174, shape information 176, and release method information 178 may be output as audible information in the form of sound, stored in some storage medium (e.g., memory and / or magnetic tape, etc.), or recorded on a medium by a printer.

[0133] Next, an example of the flow of medical support processing performed by the endoscope system 10 will be explained with reference to Figure 17. The flow of medical support processing shown in Figure 17 is an example of the "medical support method" related to this disclosure.

[0134] In the medical support processing shown in Figure 17, first, in step ST100, the recognition unit 82A acquires an image 164 from the camera 52 and generates a frame 40 by performing various processing on the acquired image 164 (see Figure 14). Then, the control unit 82B displays the latest frame 40 generated by the recognition unit 82A in the first display area 35A. After the processing in step ST100 is completed, the medical support processing proceeds to step ST102.

[0135] In step ST102, the recognition unit 82A generates lumen position information 168 for the lumen recognition model 92 by executing a lumen recognition process 166 using the lumen recognition model 92 (see Figure 14). After the process in step ST102 is executed, the medical support process moves to step ST104.

[0136] In step ST104, the recognition unit 82A stores the lumen position information 168 generated in step ST102 in chronological order. After the processing in step ST104 is completed, the medical support processing proceeds to step ST106.

[0137] In step ST106, the recognition unit 82A determines whether it has multiple lumen position information 168 (in this case, as an example, three or more lumen position information 168) in a time-series order. If, in step ST106, it is determined that multiple lumen position information 168 is not held in a time-series order (for example, if only one lumen position information 168 is held), the determination is denied, and the medical support process proceeds to step ST114. If, in step ST106, it is determined that multiple lumen position information 168 is held in a time-series order, the determination is affirmed, and the medical support process proceeds to step ST108.

[0138] In step ST108, the recognition unit 82A generates rotation information 174 for the rotation recognition model 94 by performing rotation recognition processing 172 on multiple lumen position information 168 that are held in chronological order (see Figure 15). After the processing in step ST108 is completed, the medical support processing moves on to step ST110.

[0139] In step ST110, the control unit 82B derives the corresponding shape information 176 and release method information 178 from the rotation information 174 generated in step ST108 by referring to the information derivation table 96 (see Figure 16). After the processing in step ST110 is completed, the medical support processing proceeds to step ST112.

[0140] In step ST112, the control unit 82B displays the shape information 176 and release method information 178 derived in step ST110, as well as the rotation information 174 generated in step ST108, as visualized auxiliary information 44 in the second display area 35B (see Figure 16). After the processing in step ST112 is executed, the medical support processing proceeds to step ST114.

[0141] In step ST114, the control unit 82B determines whether the conditions for terminating the medical support process have been met. An example of a condition for terminating the medical support process is that the endoscope system 10 has been given an instruction to terminate the medical support process (for example, that the instruction to terminate the medical support process has been received by the receiving device 64).

[0142] In step ST114, if the conditions for terminating the medical support process are not met, the judgment is denied, and the medical support process proceeds to step ST100 shown in Figure 17. In step ST100, if the conditions for terminating the medical support process are met, the judgment is affirmed, and the medical support process is terminated.

[0143] As described above, in the endoscopic system 10, multiple frames 40 obtained by imaging the inside of the large intestine 28 with the endoscope 16 inserted into the large intestine 28 are input to the lumen recognition model 92, and the lumen recognition model 92 generates lumen position information 168. Furthermore, multiple lumen position information 168 in a time series are input to the rotation recognition model 94, and the rotation recognition model 94 generates rotation information 174. Then, shape information 176 is derived based on the rotation information 174. Therefore, the shape of the insertion part 48 when the endoscope 16 is inserted into the large intestine 28 can be estimated without using an external device to recognize the shape of the insertion part 48 when the insertion part 48 of the endoscope 16 is inserted into the large intestine 28.

[0144] Furthermore, in the endoscope system 10, rotation information 174 is obtained based on the cumulative result of the accumulation of changes in feature information (i.e., feature-representing information) obtained from multiple frames 40. Here, lumen position information 168 is given as an example of feature information (see Figure 15). Therefore, accurate shape information 176 can be generated by utilizing the cumulative change in the position of the lumen 42 shown in the frame 40. In addition, rotation information 174 can be obtained more easily than when a dedicated sensor for obtaining rotation information 174 is used, or when a dedicated device for obtaining rotation information 174 is used outside the endoscope system 10.

[0145] Furthermore, in the endoscope system 10, the rotation information 174 includes information regarding the relative rotation angle and direction of rotation of the insertion section 48 at a second position relative to a first position of the insertion section 48, or information regarding the absolute rotation angle and direction of rotation of the insertion section 48 at the first and second positions relative to a reference angle. Therefore, by utilizing the relative or absolute rotation information 174 of multiple positions of the insertion section 48, the shape of the insertion section 48 can be estimated with high accuracy.

[0146] Furthermore, in the endoscope system 10, the shape information 176 when the tip position of the insertion section 48 is rotated by 180 degrees or more relative to the proximal position of the insertion section 48 includes information indicating that the insertion section 48 forms a loop. Therefore, the physician 12 can estimate with high accuracy whether the insertion section 48 forms a loop by referring to the shape information 176.

[0147] Furthermore, in the endoscope system 10, when the tip position of the insertion section 48 is rotated by 180 degrees or more relative to the proximal position of the insertion section 48, the shape information 176 includes loop classification information, which is information that classifies the shape of the loop based on the direction of rotation identified from the rotation information 174. Therefore, the physician 12 can estimate the type of loop shape of the insertion section 48 with high accuracy by referring to the shape information 176.

[0148] Here, the loop classification information is information that classifies the loop of the insertion part 48 as an α-loop or an inverse α-loop based on the direction of rotation identified from the rotation information 174. Therefore, the physician 12 can accurately estimate whether the shape of the loop of the insertion part 48 is an α-loop or an inverse α-loop by referring to the shape information 176.

[0149] Furthermore, the endoscope system 10 outputs release method information 178 based on the rotation information 174. The release method information 178 is information regarding the method of releasing the loop of the insertion section 48. Therefore, it is possible to assist the physician 12 in selecting the appropriate operation to release the loop of the insertion section 48.

[0150] [Second Embodiment] In the first embodiment described above, an example of a configuration in which rotation information 174 is obtained based on the cumulative result of the cumulative change of lumen position information 168 between multiple frames 40 (see Figure 15) was illustrated, but this is merely one example. In this second embodiment, an example of a configuration in which rotation information 174 is obtained based on information other than the cumulative result of the cumulative change of lumen position information 168 between multiple frames 40 will be described. In this second embodiment, the same reference numerals are used for the components described in the first embodiment above, and their descriptions are omitted. Only the parts that differ from the first embodiment will be described.

[0151] Figure 18 is a block diagram showing an example of the essential functions of the processor 82 included in the medical support device 24 according to the second embodiment, and an example of the information stored in the storage 86. As shown in Figure 18, the storage 86 stores a medical support program 200. In this second embodiment, the medical support program 200 is an example of the "program" according to this disclosure.

[0152] The processor 82 reads the medical support program 200 from the storage 86 and performs medical support processing by executing the read medical support program 200 on the memory 84. The medical support processing is realized by the recognition unit 82A1 and the control unit 82B operating according to the medical support program 200 executed by the processor 82 on the memory 84.

[0153] Storage 86 stores the gravity direction recognition model 202, the rotation recognition model 204, and the information derivation table 96. As will be described in more detail later, the gravity direction recognition model 202 and the rotation recognition model 204 are machine learning models and are used by the recognition unit 82A1. An example of a machine learning model is a neural network (e.g., a recurrent neural network, a two-dimensional convolutional neural network, and / or a three-dimensional convolutional neural network).

[0154] Figure 19 is a conceptual diagram showing an example of how the processor 106 of the information processing device 100 generates training data 208 by associating the correct answer data 206 with the example image 122A.

[0155] As shown in Figure 19, with the example image 122A displayed on screen 118A, the annotator 124 instructs the processor 106 via the receiving device 116 to locate the liquid retention position 212 within the example image 122A, which is the position of the liquid 210 retained on the intestinal wall 136 within the large intestine 132 shown in the example image 122A. The processor 106 superimposes a circular frame 214 onto the example image 122A, surrounding the liquid retention position 212, according to the instructions received by the receiving device 116. The frame 214 is a mark that defines the liquid retention position 212 within the example image 122A. Here, the shape of the frame 214 is circular, but it may be a shape other than a circle. The size of the frame 214 can also be changed according to the instructions received by the receiving device 116.

[0156] The annotator 124, with the frame 214 positioned around the liquid retention position 212, provides a confirmation instruction to the processor 106 via the receiving device 116, which is an instruction to confirm the liquid retention position 212. As a result, the processor 106 confirms the liquid retention position 212.

[0157] The processor 106 generates training data 208 by associating the example image 122A with the correct data 206, which is information that can identify the direction of gravity within the example image 122A. The correct data 206 is a vector that starts at the center C4 of the example image 122A and ends at the center of the liquid retention position 212.

[0158] In this way, the processor 106 generates multiple training data 208 by repeatedly associating each of the example images 122A included in the example image set 122 with the correct answer data 206 according to the instructions given by the annotator 124.

[0159] Figure 20 is a conceptual diagram showing an example of how a gravity direction recognition model 202 is generated in the information processing device 100 by machine learning using training data 208 performed by the processor 106.

[0160] As shown in Figure 20, in the information processing device 100, the processor 106 performs machine learning using the training data 208 generated in the manner described above. The following will be explained in detail with reference to Figure 20.

[0161] In the example shown in Figure 20, the processor 106 performs processing using the model 213. An example of the model 213 is the neural network exemplified in the first embodiment described above. The processor 106 inputs the example image 122A included in the training data 208 into the model 213. Upon receiving the example image 122A, the model 213 performs inference and outputs the inference result 215. The processor 106 calculates the error 216 between the inference result 215 and the correct answer data 206 included in the training data 208.

[0162] The processor 106 calculates multiple adjustment values ​​218 that minimize the error 216. Then, the processor 106 optimizes the model 213 by adjusting multiple optimization variables within the model 213 using the multiple adjustment values ​​218. Examples of multiple optimization variables within the model 213 include weights and biases.

[0163] The processor 106 repeatedly performs the learning process using multiple training data sets 208, which involves inputting example images 122A included in the training data 208 into the model 213, calculating the error 216, calculating multiple adjustment values ​​218, and adjusting multiple optimization variables within the model 213. In other words, for each of the multiple example images 122A included in the multiple training data sets 208, the processor 106 optimizes the model 213 by adjusting multiple optimization variables within the model 213 using multiple adjustment values ​​218 calculated to minimize the error 216. This optimization of the model 213 generates the gravity direction recognition model 202. The gravity direction recognition model 202 is transmitted from the information processing device 100 to the medical support device 24 via external I / F 80 and 104 (see Figure 7), and received by the medical support device 24. The medical support device 24 then stores the gravity direction recognition model 202 in storage 86 by the processor 82 (see Figure 18). The gravity direction recognition model 202 stored in storage 86 is used by the recognition unit 82A1 (see Figure 18).

[0164] As shown in Figure 21, the storage 110 of the information processing device 100 stores a dataset group 220. The dataset group 220 is a collection of multiple datasets 220A. The contents of the multiple datasets 220A are different from each other. A dataset 220A is training data in which the gravity direction information set 220A1 and the correct answer data 152A2 described in the first embodiment are associated. The generation of dataset 220A is achieved by associating the correct answer data 152A2 with the gravity direction information set 220A1, in the same manner as how the training data 128 shown in Figure 8 described in the first embodiment is generated by the training data generation unit 106A.

[0165] The gravity direction information set 220A1 includes multiple gravity direction information sets 220A1a arranged in chronological order (here, as an example, three or more gravity direction information sets 220A1a). The gravity direction information set 220A1a is one of the fourth to sixth pieces of information. The fourth piece of information is information indicating the direction of gravity (e.g., a direction vector) that can be identified from each of the multiple frames included in the endoscopic video image obtained when the inside of the large intestine is imaged by the endoscope during the process until the α-loop of the endoscope insertion section is formed during an endoscopy. The fifth piece of information is information indicating the direction of gravity (e.g., a direction vector) that can be identified from each of the multiple frames included in the endoscopic video image obtained when the inside of the large intestine is imaged by the endoscope during the process until the inverse α-loop of the endoscope insertion section is formed during an endoscopy. The sixth piece of information is information indicating the direction of gravity (e.g., a direction vector) that can be identified from each of the multiple frames included in the endoscopic video image obtained when the inside of the large intestine is imaged by the endoscope when the loop of the endoscope insertion section is not formed during an endoscopy.

[0166] The gravity direction information set 220A1 contains one of the fourth to sixth time-series information. The fourth time-series information is information obtained during an endoscopic examination from the time the insertion part of the endoscope is inserted into the large intestine until the α-loop of the insertion part is formed, with the gravity direction information 220A1a arranged in time series. The fourth time-series information can also be described as the cumulative result of the change in the gravity direction over multiple frames until the gravity direction has completed at least half a counterclockwise rotation.

[0167] The fifth time-series information is a time-series arrangement of gravity direction information 220A1a obtained for a number of frames during an endoscopic examination, from the time the insertion part of the endoscope is inserted into the large intestine until the inverse alpha loop of the insertion part is formed. The fifth time-series information can also be described as the cumulative result of the changes in the gravity direction over multiple frames until the gravity direction completes at least half a clockwise rotation.

[0168] The sixth time-series information is a time-series arrangement of gravity direction information 220A1a for a predetermined number of frames (for example, the number of frames corresponding to statistical values ​​such as the average, median, mode, maximum, or minimum of the number of frames obtained until an α-loop or inverse α-loop is formed) in an endoscopic examination, from the time the insertion part of the endoscope is inserted into the large intestine until a loop of the insertion part is formed.

[0169] Here, for the gravity direction information set 220A1, which contains the fourth time-series information, data that can identify the rotation direction (e.g., clockwise) and rotation angle (e.g., an angle of 180 degrees or more) around the long axis of the endoscope insertion section when an α loop is formed is associated as ground truth data 152A2.

[0170] Furthermore, for gravity direction information set 220A1, which contains fifth time-series information, data is associated as ground truth data 152A2 that can identify the rotation direction (e.g., counterclockwise) and rotation angle (e.g., an angle of 180 degrees or more) around the long axis of the endoscope insertion section when an inverse alpha loop is formed.

[0171] Furthermore, for gravity direction information set 220A1, which contains sixth time-series information, ground truth data 152A2 is associated with data that can identify the rotation direction (e.g., clockwise, counterclockwise, or neither clockwise nor counterclockwise) and rotation angle (e.g., an angle less than 180 degrees) when no loop is formed.

[0172] In the information processing device 100, the processor 106 performs machine learning using the dataset group 220. This will be explained in detail below with reference to Figure 21.

[0173] The processor 106 performs processing using the model 222. An example of the model 222 is the neural network exemplified in the first embodiment described above. The processor 106 obtains dataset 220A from dataset group 220. Then, the processor 106 inputs multiple gravity direction information 220A1a, which are arranged in time series within the gravity direction information set 220A1 included in dataset 220A obtained from dataset group 220, into the model 222 in time series. When the model 222 receives multiple gravity direction information 220A1a arranged in time series, it performs inference and outputs the inference result 224. The processor 106 calculates the error 226 between the inference result 224 and the ground truth data 152A2 included in dataset 220A obtained from dataset group 220.

[0174] The processor 106 calculates multiple adjustment values ​​228 that minimize the error 226. Then, the processor 106 optimizes the model 222 by adjusting multiple optimization variables within the model 222 using the multiple adjustment values ​​228. Examples of multiple optimization variables within the model 222 include weights and biases.

[0175] The processor 106 repeatedly performs the learning process of inputting the gravity direction information set 202A1 into the model 222, calculating the error 226, calculating multiple adjustment values ​​228, and adjusting multiple optimization variables within the model 222, using all datasets 220A included in the dataset group 220. That is, for each of the gravity direction information sets 220A1 included in all datasets 220A, the processor 106 optimizes the model 222 by adjusting multiple optimization variables within the model 222 using multiple adjustment values ​​228 calculated to minimize the error 226. The rotation recognition model 204 is generated by optimizing the model 222 in this way. The rotation recognition model 204 is transmitted from the information processing device 100 to the medical support device 24 via external I / F 80 and 104 (see Figure 7), and received by the medical support device 24. Then, the rotation recognition model 204 is stored in the storage 86 by the processor 82 in the medical support device 24 (see Figure 18). The rotation recognition model 204 stored in storage 86 is used by the recognition unit 82A1 (see Figure 18).

[0176] Figure 22 shows an example of the processing content of the recognition unit 82A1. The recognition unit 82A1 differs from the recognition unit 82A described in the first embodiment above in that it performs gravity direction recognition processing 230 on the frame 40 instead of lumen recognition processing 153.

[0177] The gravity direction recognition process 230 is a process that recognizes the direction of gravity within the large intestine 28 shown in frame 40 by using the gravity direction recognition model 202 stored in storage 86 (in other words, a process that identifies the direction of gravity within the large intestine 28 shown in frame 40 by using the gravity direction recognition model 202).

[0178] The recognition unit 82A1 inputs the frame 40 to the gravity direction recognition model 202, thereby causing the gravity direction recognition model 202 to generate gravity direction information 232. The gravity direction information 232 is information that can identify the direction of gravity within the large intestine 28. An example of the gravity direction information 232 is a vector that starts at the center C5 of the frame 40 and ends at the center of the liquid retention position 236 (here, as an example, the position of a circular frame surrounding the center of the liquid 234) which is the position within the frame 40 of the liquid 234 that is retained on the intestinal wall 32 within the large intestine 28 as seen in the frame 40. In this second embodiment, the gravity direction information 232 is an example of the "feature information" and "gravity direction information" related to this disclosure.

[0179] As an example, as shown in Figure 23, the recognition unit 82A1 holds multiple gravity direction information 232 in a time series. Here, the multiple gravity direction information 232 in a time series can be said to be the cumulative result of the accumulation of changes in feature information (i.e., feature-representing information) obtained from the frame 40 across multiple frames 40. Similar to the recognition unit 82A described in the first embodiment above, the recognition unit 82A1 generates rotation information 174 based on the cumulative result of the accumulation of changes in feature information obtained from the frame 40 across multiple frames 40. Here, an example of the cumulative result of changes in feature information obtained from the frame 40 across multiple frames 40 is the multiple gravity direction information 232 in a time series obtained by the recognition unit 82A1 executing gravity direction recognition processing 230 (see Figure 22) for each of the multiple frames 40 obtained in a time series.

[0180] The recognition unit 82A1 performs a rotation recognition process 238 on multiple gravity direction information 232 in a time series (here, as an example, three or more pieces of gravity direction information 232). The rotation recognition process 238 is a process that recognizes rotation information 174 from multiple gravity direction information 232 in a time series by using a rotation recognition model 204 stored in the storage 86 (in other words, a process that identifies the rotation direction and rotation angle around the long axis of the insertion unit 48 by using the rotation recognition model 204).

[0181] The recognition unit 82A1 inputs a plurality of gravity direction information 232 in a time series to the rotation recognition model 204, thereby causing the rotation recognition model 204 to generate rotation information 174. In this second embodiment, the gravity direction recognition model 202 and the rotation recognition model 204 are examples of "trained models" according to this disclosure. Also, in this second embodiment, the rotation information 174 is an example of "rotation information" according to this disclosure.

[0182] The rotation information 174 obtained in this manner is used by the control unit 82B in the same manner as in the first embodiment. This provides the same effects as in the first embodiment.

[0183] [Third Embodiment] In the first embodiment described above, an example of a configuration in which rotation information 174 is obtained based on the cumulative result of the cumulative change of lumen position information 168 between multiple frames 40 (see Figure 15) was illustrated, and in the second embodiment described above, an example of a configuration in which rotation information 174 is obtained based on the cumulative result of the cumulative change of gravity direction information 232 between multiple frames 40 (see Figure 23) was illustrated, but the disclosure is not limited thereto. In this third embodiment, an example of a configuration in which rotation information 174 is obtained based on information other than the cumulative result of the cumulative change of lumen position information 168 between multiple frames 40 and the cumulative result of the cumulative change of gravity direction information 232 between multiple frames 40 will be described. In this third embodiment, the same reference numerals are used for the components described in the first and second embodiments above, and their descriptions are omitted. Only the parts that differ from the first and second embodiments will be described.

[0184] Figure 24 is a conceptual diagram showing an example of how a rotation recognition model 242 is generated in the information processing device 100 by machine learning using the dataset group 240 performed by the processor 106.

[0185] As shown in Figure 24, the dataset group 240 is a collection of multiple datasets 240A. The contents of the multiple datasets 240A are different from each other. Dataset 240A is training data in which the example dataset 240A1 and the correct answer data 152A2 are associated. In the same manner as the training data 128 shown in Figure 8 is generated by the training data generation unit 106A, the generation of dataset 240A is achieved by associating the correct answer data 152A2 with the example dataset 240A1.

[0186] The example dataset 240A1 includes multiple lumen position information 152A1a (in this case, three or more lumen position information 152A1a as an example) arranged in time series, and multiple hand images 244 (in this case, three or more hand images 244 as an example) arranged in time series. One hand image 244 is associated with each lumen position information 152A1a.

[0187] The hand-held image 244 is an image obtained by capturing the area of ​​the endoscope insertion section that includes the part of the endoscope that is held by the physician's hands (i.e., the physician operating the endoscope insertion section) using a camera. In addition to the physician's hands, the hand-held image 244 also shows the part of the endoscope insertion section that the physician is holding. The hand-held image 244 is an image obtained by capturing the area of ​​the endoscope insertion section that includes the part of the endoscope that is held by the physician's hands (i.e., the physician operating the endoscope insertion section) using a camera at the same timing used to obtain example image 122A, which was used to generate the lumen position information 152A1a to which the hand-held image 244 is associated (i.e., the lumen position information 152A1a that forms a pair with the hand-held image 244). In other words, the lumen position information 152A1a and the hand-held image 244, which are in a corresponding relationship, are obtained at the same timing.

[0188] In the information processing device 100, the processor 106 performs machine learning using the dataset group 240. This will be explained in detail below with reference to Figure 24.

[0189] Processor 106 executes processing using Model 246. An example of Model 246 is a neural network similar to Model 142 shown in Figure 11. Processor 106 obtains dataset 240A from dataset group 240. Then, Processor 106 inputs each of the multiple lumen position information 152A1a arranged in time series within the example dataset 240A1 included in dataset 240A obtained from dataset group 240, and each of the local images 244 associated with each lumen position information 152A1a, into Model 246 in time series. When Model 246 receives the multiple lumen position information 152A1a arranged in time series and the multiple local images 244 arranged in time series, it performs inference and outputs the inference result 248. Processor 106 calculates the error 250 between the inference result 248 and the ground truth data 152A2 included in dataset 240A obtained from dataset group 240.

[0190] The processor 106 calculates multiple adjustment values ​​252 that minimize the error 250. Then, the processor 106 optimizes the model 246 by adjusting multiple optimization variables within the model 246 using the multiple adjustment values ​​252. Examples of multiple optimization variables within the model 246 include weights and biases.

[0191] The processor 106 repeatedly performs the learning process, which involves inputting the example dataset 240A1 into the model 246, calculating the error 250, calculating multiple adjustment values ​​252, and adjusting multiple optimization variables within the model 246, using all datasets 240A included in the dataset group 240. That is, for each pair of all lumen position information sets 152A1 and all local images 244 included in all datasets 240A, the processor 106 optimizes the model 246 by adjusting multiple optimization variables within the model 246 using multiple adjustment values ​​252 calculated to minimize the error 250. The rotation recognition model 242 is generated by optimizing the model 246 in this way. The rotation recognition model 242 is transmitted from the information processing device 100 to the medical support device 24 via external I / F 80 and 104 (see Figure 7), and received by the medical support device 24. The medical support device 24 then stores the rotation recognition model 242 in storage 86 by the processor 82. The rotation recognition model 242 stored in storage 86 is used by processor 82.

[0192] In this case, for example, as shown in Figure 25, the processor 82 maintains multiple lumen position information 168 and multiple local images 254 in a time-series manner. One local image 254 is associated with one lumen position information 168.

[0193] The hand-held image 254 is an image obtained by capturing a region of the insertion section 48 of the endoscope 16 that includes the area around the doctor's hand (i.e., the area around the doctor's hand operating the insertion section 48 of the endoscope 16) with a camera. In addition to the doctor's hand, the hand-held image 254 also shows the portion of the insertion section 48 of the endoscope 16 that the doctor is holding. The hand-held image 254 is an image obtained by capturing a region of the insertion section 48 of the endoscope 16 that includes the area around the doctor's hand (i.e., the area around the doctor's hand operating the insertion section 48 of the endoscope 16) with a camera at the imaging timing used to obtain the frame 40 used to generate the lumen position information 168 to which the hand-held image 254 is associated (i.e., the lumen position information 168 that forms a pair with the hand-held image 254). In other words, the lumen position information 168 and the hand-held image 254, which are in a corresponding relationship, are obtained at the same timing.

[0194] Here, the set of multiple lumen position information 168 in a time series and multiple local images 254 in a time series can be said to be the cumulative result of the accumulation of changes in feature information (i.e., feature-representing information) obtained from frame 40 across multiple frames 40. The processor 82 generates rotation information 174 based on the cumulative result of the accumulation of changes in feature information obtained from frame 40 across multiple frames 40.

[0195] The processor 82 performs rotation recognition processing 256 on a combination of multiple lumen position information 168 (here, as an example, three or more lumen position information 168) in a time series and multiple hand images 254 in a time series. The rotation recognition processing 256 is a process that recognizes rotation information 174 from a combination of multiple lumen position information 168 and multiple hand images 254 in a time series by using a rotation recognition model 242 stored in the storage 86 (i.e., a rotation recognition model 242 obtained in the manner described using the example shown in Figure 24) (in other words, a process that identifies the direction of rotation and the angle of rotation around the long axis of the insertion part 48 by using the rotation recognition model 242).

[0196] The processor 82 inputs a combination of multiple lumen position information 168 and multiple hand images 254 in a time-series sequence to the rotation recognition model 242 in a time-series sequence, causing the rotation recognition model 242 to generate rotation information 174. The rotation information 174 obtained in this way is used by the control unit 82B in the same manner as in the first embodiment. This provides the same effects as in the first embodiment. Furthermore, since the rotation information 174 is estimated using an AI method that also uses multiple hand images 254 in a time-series sequence in addition to the multiple lumen position information 168 in a time-series sequence, it is possible to obtain rotation information 174 with higher accuracy compared to the case where the rotation information 174 is estimated only from the multiple lumen position information 168 in a time-series sequence.

[0197] In this third embodiment, the lumen recognition model 92 and the rotation recognition model 242 are examples of "trained models" as defined in this disclosure. Also, in this third embodiment, the rotation information 174 is an example of "rotation information" as defined in this disclosure. Furthermore, in this third embodiment, the physician 12 is an example of a "practitioner" as defined in this disclosure, and the multiple hand images 244 are examples of "multiple images" as defined in this disclosure.

[0198] [Fourth Embodiment] In the first embodiment described above, an example was given in which shape information 176 corresponding to rotation information 174 is derived from the information derivation table 96, but this is merely one example. In this fourth embodiment, an example of how shape information 176 corresponding to rotation information 174 is derived using the AI ​​method will be described. In this fourth embodiment, the same reference numerals are used for the components described in the first to third embodiments above, and their descriptions are omitted. Only the parts that differ from the first to third embodiments will be described.

[0199] Figure 26 is a conceptual diagram showing an example of how a shape recognition model 260 is generated by machine learning using the dataset group 258 performed by the processor 106.

[0200] As shown in Figure 26, the dataset group 258 is a collection of multiple datasets 258A. The contents of the multiple datasets 258A are different from each other. Dataset 258A is training data in which the example image set 258A1 and the correct answer data 262 are associated. In the same manner as the training data 128 shown in Figure 8 is generated by the training data generation unit 106A, the generation of dataset 258A is achieved by associating the correct answer data 262 with the example image set 258A1.

[0201] The correct answer data 262 is data that can identify the shape of the insertion site of the endoscope within the large intestine (e.g., alpha loop, inverted alpha loop, or non-loop). For example image set 258A1, the correct answer data 262 is associated with data that can identify an alpha loop, data that can identify an inverted alpha loop, or data that can identify a non-loop.

[0202] Example image set 258A1 contains multiple example images 122A arranged in chronological order (here, as an example, three or more example images 122A). Example image set 258A1 contains one of the 7th to 9th time-series information.

[0203] The seventh time-series information is information obtained by imaging with the endoscope at a predetermined frame rate (e.g., tens of frames / second) from the time the insertion part of the endoscope is inserted into the large intestine until the α-loop of the insertion part is formed during an endoscopic examination, and these images are arranged in chronological order as multiple example images 122A.

[0204] The eighth time-series information consists of multiple endoscopic images obtained by imaging the endoscope at a predetermined frame rate from the time the insertion tube of the endoscope is inserted into the large intestine until the inverse alpha loop of the insertion tube is formed, arranged in chronological order as multiple example images 122A.

[0205] The ninth time-series information is information in which endoscopic images are arranged in time as multiple example images 122A for a predetermined number of frames (for example, the number of frames corresponding to statistical values ​​such as the mean, median, mode, maximum, or minimum of the number of frames obtained until an α-loop or inverse α-loop is formed) when the insertion part of the endoscope is inserted into the large intestine but no loop of the insertion part is formed during an endoscopic examination.

[0206] Here, for example image set 258A1 containing the seventh time series information, data that can identify the α loop is associated as the correct answer data 262. Similarly, for example image set 258A1 containing the eighth time series information, data that can identify the inverse α loop is associated as the correct answer data 262. Furthermore, for example image set 258A1 containing the ninth time series information, data that can identify the non-loop is associated as the correct answer data 262.

[0207] In the information processing device 100, the processor 106 performs machine learning using the dataset group 258. This will be explained in detail below with reference to Figure 26.

[0208] Processor 106 performs processing using Model 264. An example of Model 264 is a neural network similar to Model 142 shown in Figure 11. Processor 106 obtains dataset 258A from dataset group 258. Then, Processor 106 inputs multiple example images 122A, which are arranged in chronological order within the example image set 258A1 included in dataset 258A obtained from dataset group 258, into Model 264 in chronological order. Upon receiving the multiple example images 122A arranged in chronological order, Model 264 performs inference and outputs the inference result 266. Processor 106 calculates the error 268 between the inference result 266 and the ground truth data 262 included in dataset 258A obtained from dataset group 258.

[0209] The processor 106 calculates several adjustment values ​​270 that minimize the error 268. Then, the processor 106 optimizes the model 264 by adjusting several optimization variables within the model 264 using the several adjustment values ​​270. Examples of several optimization variables within the model 264 include weights and biases.

[0210] The processor 106 repeatedly performs the learning process, which involves inputting the example image set 258A1 into the model 264, calculating the error 268, calculating multiple adjustment values ​​270, and adjusting multiple optimization variables within the model 264, using all datasets 258A included in the dataset group 258. That is, for each example image set 258A1 included in all datasets 258A, the processor 106 optimizes the model 264 by adjusting multiple optimization variables within the model 264 using multiple adjustment values ​​270 calculated to minimize the error 268. The shape recognition model 260 is generated by optimizing the model 264 in this way. The shape recognition model 260 is transmitted from the information processing device 100 to the medical support device 24 via external I / F 80 and 104 (see Figure 7), and received by the medical support device 24. Then, the shape recognition model 260 is stored in the storage 86 by the processor 82 in the medical support device 24. The shape recognition model 260 stored in storage 86 is used by processor 82.

[0211] In this case, for example, as shown in Figure 27, the processor 82 holds multiple frames 40 in a time series (for example, three or more frames 40). The processor 82 performs a shape recognition process 272 on the multiple frames 40 in a time series. The shape recognition process 272 is a process that recognizes shape information 176 from the multiple frames 40 in a time series by using a shape recognition model 260 stored in the storage 86 (i.e., a shape recognition model 260 obtained in the manner described using the example shown in Figure 26) (in other words, a process that uses the shape recognition model 260 to determine whether the shape of the insertion part 48 of the endoscope 16 inside the large intestine 28 is an alpha loop, an inverse alpha loop, or not a loop).

[0212] The processor 82 inputs multiple frames 40 in a time series order to the shape recognition model 260, causing the shape recognition model 260 to generate shape information 176.

[0213] In this fourth embodiment, the multiple frames 40 are an example of "multiple endoscopic images" according to the disclosure. Also, in this fourth embodiment, the large intestine 28 is an example of a "luminal organ" according to the disclosure. Also, in this fourth embodiment, the endoscope 16 is an example of an "endoscope" according to the disclosure. Also, in this fourth embodiment, the insertion part 48 is an example of an "insertion part" according to the disclosure. Also, in this fourth embodiment, the shape recognition model 260 is an example of a "trained model" according to the disclosure. Also, in this fourth embodiment, the shape information 176 is an example of "shape information" according to the disclosure.

[0214] As an example, as shown in Figure 28, the storage 86 stores a release method derivation table 274. The release method derivation table 274 is a table in which shape information 176 and release method information 178 that are in a corresponding relationship with each other are associated. For example, if the shape information 176 is information that can identify an α-loop, then information on how to release the α-loop is associated with the shape information 176 as release method information 178. Also, for example, if the shape information 176 is information that can identify an inverse α-loop, then information on how to release the inverse α-loop is associated with the shape information 176 as release method information 178.

[0215] The processor 82 refers to the release method derivation table 274 and derives release method information 178 corresponding to the shape information 176 generated in the manner described in the example shown in Figure 27 from the release method derivation table 274. Then, the processor 82 displays the shape information 176 and the release method information 178, etc., as visible information in the second display area 35B. In this way, the same effects as in each of the above embodiments can be obtained.

[0216] In this fourth embodiment, we have given an example in which shape information 176 is generated by the shape recognition model 260 and release method information 178 is derived from the release method derivation table 274, but this is merely one example. For example, both the shape information 176 and the release method information 178 may be generated by a trained model obtained by optimizing the model through machine learning of both the shape (e.g., α-loop, inverse α-loop, and non-loop) and the release method (e.g., α-loop release method and inverse α-loop release method).

[0217] [Fifth Embodiment] In the fourth embodiment described above, an example was given in which shape information 176 is derived using an AI method with multiple frames 40 arranged in a time series, but this is merely one example. In this fifth embodiment, an example will be described in which shape information 176 is derived using an AI method with multiple lumen position information 152A1a arranged in a time series. In this fifth embodiment, the same reference numerals are used for the components described in the first to fourth embodiments above, and their descriptions are omitted. Only the parts that differ from the first to fourth embodiments will be described.

[0218] Figure 29 is a conceptual diagram showing an example of how a shape recognition model 278 is generated by machine learning using the dataset group 276 performed by the processor 106.

[0219] As shown in Figure 29, the dataset group 276 is a collection of multiple datasets 276A. The contents of the multiple datasets 276A are different from each other. Dataset 276A is training data in which the lumen position information set 152A1 is associated with the ground truth data 262 described in the fourth embodiment above. In the same manner as the training data 128 shown in Figure 8 is generated by the training data generation unit 106A, the generation of dataset 276A is achieved by associating the ground truth data 262 with the lumen position information set 152A1.

[0220] As described in the first embodiment above, the lumen position information set 152A1 includes one of the first to third time-series information. For lumen position information set 152A1 containing the first time-series information, data capable of identifying the α-loop is associated as the ground truth data 262. For lumen position information set 152A1 containing the second time-series information, data capable of identifying the inverse α-loop is associated as the ground truth data 262. Furthermore, for lumen position information set 152A1 containing the third time-series information, data capable of identifying the non-loop is associated as the ground truth data 262.

[0221] In the information processing device 100, the processor 106 performs machine learning using the dataset group 276. This will be explained in detail below with reference to Figure 29.

[0222] Processor 106 performs processing using Model 280. An example of Model 280 is a neural network similar to Model 142 shown in Figure 11. Processor 106 obtains dataset 276A from dataset group 276. Then, Processor 106 inputs multiple lumen position information 152A1a, which are arranged in time series within the lumen position information set 152A1 contained in dataset 276A obtained from dataset group 276, into Model 280 in chronological order. When Model 280 receives multiple lumen position information 152A1a arranged in time series, it performs inference and outputs the inference result 282. Processor 106 calculates the error 284 between the inference result 282 and the ground truth data 262 contained in dataset 276A obtained from dataset group 276.

[0223] The processor 106 calculates multiple adjustment values ​​286 that minimize the error 284. Then, the processor 106 optimizes the model 280 by adjusting multiple optimization variables within the model 280 using the multiple adjustment values ​​286. Examples of multiple optimization variables within the model 280 include weights and biases.

[0224] The processor 106 repeatedly performs the learning process of inputting the dataset 276A into the model 280, calculating the error 284, calculating multiple adjustment values ​​286, and adjusting multiple optimization variables within the model 280, using all datasets 276A included in the dataset group 276. That is, for each of the lumen position information sets 152A1 included in all datasets 276A, the processor 106 optimizes the model 280 by adjusting multiple optimization variables within the model 280 using multiple adjustment values ​​286 calculated to minimize the error 284. The shape recognition model 278 is generated by optimizing the model 280 in this way. The shape recognition model 278 is transmitted from the information processing device 100 to the medical support device 24 via external I / F 80 and 104 (see Figure 7), and received by the medical support device 24. Then, the shape recognition model 278 is stored in the storage 86 by the processor 82 in the medical support device 24. The shape recognition model 278 stored in the storage 86 is used by the processor 82.

[0225] In this case, for example, as shown in Figure 30, the processor 82 maintains a plurality of lumen position information 168 in a time-series manner, similar to the example shown in Figure 15 of the first embodiment described above. The processor 82 performs a shape recognition process 288 on the plurality of lumen position information 168 in a time-series manner (here, as an example, three or more lumen position information 168). The shape recognition process 288 is a process that recognizes shape information 176 from the plurality of lumen position information 168 in a time-series manner by using a shape recognition model 278 stored in the storage 86 (i.e., a shape recognition model 278 obtained in the manner described using the example shown in Figure 29) (in other words, a process that uses the shape recognition model 278 to determine whether the shape of the insertion portion 48 of the endoscope 16 within the large intestine 28 is an α-loop, an inverse α-loop, or not a loop).

[0226] The processor 82 inputs multiple lumen position information 168 in a time series to the shape recognition model 278 in a time series, thereby causing the shape recognition model 278 to generate shape information 176. In this fifth embodiment, the same processing as shown in Figure 28 described in the fourth embodiment is performed by the processor 82. In this way, the same effects as in each of the above embodiments can be obtained.

[0227] In this fifth embodiment, the multiple frames 40 are an example of "multiple endoscopic images" according to the disclosure. Also, in this fifth embodiment, the multiple lumen position information 168 are an example of "accumulated results obtained by imaging the inside of a lumen organ with an endoscope whose insertion part is inserted into the lumen organ" according to the disclosure, where the changes in feature information contained in each of the multiple endoscopic images are accumulated among the multiple endoscopic images. Also, in this fifth embodiment, the large intestine 28 is an example of "lumen organ" according to the disclosure. Also, in this fifth embodiment, the endoscope 16 is an example of "endoscope" according to the disclosure. Also, in this fifth embodiment, the insertion part 48 is an example of "insertion part" according to the disclosure. Also, in this fifth embodiment, the shape recognition model 278 is an example of "trained model" according to the disclosure. Also, in this fifth embodiment, the shape information 176 is an example of "shape information" according to the disclosure.

[0228] In the embodiments described above, examples of how medical support processing is performed by the computer 78 were given, but this disclosure is not limited thereto, and at least some of the processing included in the medical support processing may be performed by a device located outside the computer 78. An example of this case will be described below with reference to Figure 31.

[0229] Figure 31 is a conceptual diagram showing an example of the configuration of the endoscope system 300. The endoscope system 300 is an example of the "endoscope system" according to this disclosure. The endoscope system 300 differs from the endoscope system 10 described in the above embodiment in that it has an external device 302.

[0230] The external device 302 is connected to the computer 78 via a network 304 (e.g., WAN and / or LAN, etc.) so that it can communicate with it.

[0231] An example of an external device 302 is at least one server that directly or indirectly sends and receives data with the computer 78 via the network 304. The external device 302 receives processing execution instructions from the processor 82 of the computer 78 via the network 304. The external device 302 then executes the processing according to the received processing execution instructions and transmits the processing results to the computer 78 via the network 304. The processor 82 of the computer 78 receives the processing results transmitted from the external device 302 via the network 304 and executes processing using the received processing results.

[0232] Examples of processing execution instructions include instructions to have the external device 302 perform at least a portion of the medical support processing.

[0233] One example of at least a part of the medical support processing (i.e., processing to be performed by the external device 302) is the lumen recognition processing 166. In this case, the external device 302 performs the lumen recognition processing 166 according to the processing execution instructions given from the processor 82 via the network 304, and transmits the lumen position information 168 to the computer 78 via the network 304. The computer 78, through the processor 82, receives the lumen position information 168 and performs processing using the received lumen position information 168.

[0234] A second example of at least a part of the medical support processing (i.e., processing to be performed by the external device 302) is rotation recognition processing 172, 238, or 256. In this case, the external device 302 executes rotation recognition processing 172, 238, or 256 according to processing execution instructions given from the processor 82 via the network 304, and transmits rotation information 174 to the computer 78 via the network 304. The computer 78's processor 82 receives the rotation information 174 and executes processing using the received rotation information 174.

[0235] A third example of at least a part of the medical support processing (i.e., processing to be performed by the external device 302) is the shape recognition processing 272 or 288. In this case, the external device 302 executes the shape recognition processing 272 or 288 according to the processing execution instructions given from the processor 82 via the network 304, and transmits the shape information 176 to the computer 78 via the network 304. In the computer 78, the processor 82 receives the shape information 176 and performs processing using the received shape information 176.

[0236] A fourth example of at least a part of the medical support processing (i.e., processing to be performed by the external device 302) is gravity direction recognition processing 230. In this case, the external device 302 performs gravity direction recognition processing 230 according to the processing execution instructions given from the processor 82 via the network 304, and transmits gravity direction information 232 to the computer 78 via the network 304. In the computer 78, the processor 82 receives the gravity direction information 232 and performs processing using the received gravity direction information 232.

[0237] A fifth example of at least a part of the medical support processing (i.e., processing to be performed by the external device 302) is the processing performed by the control unit 82B (for example, the processing described in each of the embodiments above). In this case, the external device 302 performs the processing performed by the control unit 82B in accordance with the processing execution instructions given from the processor 82 via the network 304, and transmits the processing results (for example, visible information to be displayed on the screen 35, etc.) to the computer 78 via the network 304. In the computer 78, the processor 82 receives the processing results and performs the same processing as in each of the embodiments above using the received processing results.

[0238] The external device 302 may be implemented through cloud computing. Cloud computing is merely one example; the external device 302 may also be implemented through network computing such as fog computing, edge computing, or grid computing.

[0239] In each of the above embodiments, each process is executed on any computer. Furthermore, any computer may execute these processes using a processor as hardware, a program as software, or a combination thereof. In this case, the processor is configured to work in cooperation with the program to execute the various processes in each of the above embodiments, and can function as a unit or means in these embodiments. Also, the execution order of the processes by the processor is not limited to the order described and may be changed as appropriate. Any computer may be a general-purpose computer, a computer designed for a specific purpose, a workstation, or any other system capable of executing each process.

[0240] A processor may consist of one or more hardware components, and the type of hardware is not limited. For example, a processor may consist of hardware such as a CPU, MPU, and / or programmable logic devices such as FPGAs, dedicated circuits for executing specific processes such as ASICs, GPUs, or NPUs. The type of hardware may also be a combination of different types of hardware. When multiple hardware components are configured to execute one or more processes of a processor, the multiple hardware components may reside in physically separate devices or in the same device. Furthermore, in any embodiment, the order of each process performed by the processor is not limited to the order described above and may be changed as appropriate. Hardware is composed of electrical circuits (circuitry) that combine circuit elements such as semiconductor elements.

[0241] Furthermore, the program may be firmware or software such as microcode. Alternatively, the program may be, for example, a set of program modules, each function of which may be implemented by a processor configured to perform its respective function. The program may be program code or multiple code segments stored on one or more non-temporary computer-readable media (e.g., storage media and / or other storage). The program may be divided and stored on multiple non-temporary computer-readable media located in physically separate devices. Program code or code segments may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, instructions, data structures, or program statements. Program code or code segments may be connected to other code segments or hardware circuits by sending and receiving information, data, arguments, parameters, or memory contents.

[0242] Furthermore, while the above embodiments describe examples in which the medical support program 90 or 200 (hereinafter referred to as "medical support program" without reference numerals) is pre-stored in the storage 86 (i.e., installed), this disclosure is not limited thereto. The medical support program may be provided in a form stored on a storage medium such as a CD-ROM, DVD-ROM, or USB memory. Alternatively, the medical support program may be provided in a form that can be downloaded from an external device via a network.

[0243] The technology disclosed herein extends to all program products. Program products include all forms of products for providing programs. For example, program products include programs provided via networks such as the Internet, as well as non-temporary computer-readable storage media such as CD-ROMs, DVDs, and USB memory sticks on which programs are stored.

[0244] Furthermore, the above medical support process is merely an example. Therefore, it goes without saying that you may remove unnecessary steps, add new steps, or change the order of processing, as long as it does not deviate from the main purpose.

[0245] The descriptions and illustrations presented above are detailed explanations of the parts related to this disclosure and are merely examples of this disclosure. For example, the above explanation of the structure, function, operation, and effect is an example of the structure, function, operation, and effect of the parts related to this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace parts of the descriptions and illustrations presented above, as long as you do not deviate from the spirit of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the parts related to this disclosure, explanations of common technical knowledge, etc., that do not require special explanation to enable the implementation of this disclosure have been omitted from the descriptions and illustrations presented above.

[0246] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference. [Explanation of symbols]

[0247] 10,200 Endoscopy Systems 12 Doctors 14 Staff 16 Endoscopy 18,118 Display device 20 Light source device 22 Control device 24 Medical support devices 26 subjects 28,132 large intestine 30 light 32,136 intestinal wall 34 Wagon 35,118A screen 35A 1st display area 35B 2nd display area 39 Endoscopic video 40 frames 130A,130A1~130A8,154,170 Segmented area 42,138 lumen 43,134 folds 44. Supplementary Information 46 Control section 48 Insertion part 50 Tip 50A tip surface 52 Cameras 54 Lighting equipment 54A, 54B Lighting window 56 Treatment opening 58 Treatment tools 60 Instrument insertion port 62 Universal Code 64,116 Reception device 66,78,102 computers 68,88,112 bus 70, 80, 104 External I / F 72,82,106 processors 74,84,108 memory 76,86,110 storage 82A,82A1 recognition section 82B Control Unit 90,200 medical support programs 92 Lumen Recognition Model 94,204,242 rotation recognition models 96 Information Derivation Table 100 Information Processing Devices 106A Training Data Generation Unit 106B First Learning Execution Unit 106C Second Learning Execution Unit 116A Keyboard 116B Mouse 120 First Machine Learning Processing Program 122,158A1 Example Image Set 122A Example Image 124 Annotator 126,152A2,206,262 Correct data 128,208 training data 139 Lumen-corresponding position 140,214 slots Models 142, 156, 213, 222, 246, 264, 280 144,158,215,224,248,266,282 Inference result 146,160,216,226,250,268,284 error 148,162,218,228,252,270,286 Adjusted values 150 Second Machine Learning Processing Program 151 images 152,220,240,258,276 datasets 152A, 220A, 240A, 258A, 276A dataset 152A1 Lumen Position Information Set 152A1a,168 Luminal position information 164 images 166 Lumen Recognition Processing 172,238,256 Rotation recognition processing 174 Rotation information 176 Shape information 178 Release method information 202 Gravity direction recognition model 210,234 Liquid 212,236 Liquid retention position 220A1 Gravity direction information set 220A1a,232 Gravity direction information 230 Gravity direction recognition processing 244,254 Hand image 260,278 Shape recognition model 272,288 Shape recognition processing 274 Release method derivation table 302 External device 304 Network C1,C2,C3,C4,C5 Center CD1 Circumferential direction

Claims

1. Equipped with a processor, The aforementioned processor, By inputting multiple endoscopic images obtained by imaging the inside of a tubular organ with an endoscope inserted into the tubular organ into a trained model, the trained model is made to generate rotational information about the long axis of the insertion part of the endoscope. Based on the rotation information, shape information representing the shape of the insertion part is generated. Medical support device.

2. The rotation information is obtained based on the cumulative result of the accumulation of changes in feature information obtained from the endoscope images between multiple endoscope images. The medical support device according to claim 1.

3. The characteristic information includes lumen position information indicating the position of the lumen included in the tubular organ within the endoscopic image, The rotation information is obtained based on the cumulative result of the cumulative change in the lumen position information between multiple endoscopic images. The medical support device according to claim 2.

4. The aforementioned feature information includes gravity direction information that can specify the direction of gravity, The rotation information is obtained based on the cumulative result of the cumulative change of the gravity direction information between multiple endoscopic images. The medical support device according to claim 2.

5. The rotation information includes information relating to the relative rotation angle and direction of the second position relative to the first position of the insertion portion, or information relating to the absolute rotation angle and direction of the first and second positions relative to a reference angle. The medical support device according to claim 1.

6. When the tip position of the insertion portion is rotated by 180 degrees or more relative to the base position of the insertion portion, the shape information includes information indicating that the insertion portion forms a loop. The medical support device according to claim 1.

7. The rotation information includes rotation direction information that can specify the direction of rotation around the long axis, The shape information includes loop classification information, which is information that classifies the shape of the loop based on the rotation direction information. The medical support device according to claim 6.

8. The loop classification information includes information that classifies the loop as an α-loop or an inverse α-loop based on the rotation direction information. The medical support device according to claim 7.

9. The processor outputs information regarding the method for releasing the loop based on the rotation information and / or the shape information. The medical support device according to claim 6.

10. The aforementioned processor, By inputting the aforementioned multiple endoscopic images and multiple images including the part of the insertion section closest to the operator's hand into the trained model, the trained model is instructed to generate rotation information based on information relating to the rotation of the part around its long axis. The medical support device according to claim 1.

11. The aforementioned tubular organ is the large intestine. The medical support device according to claim 1.

12. Equipped with a processor, The aforementioned processor, By inputting multiple endoscopic images obtained by imaging the inside of a tubular organ with an endoscope inserted into the organ into a trained model, the trained model generates shape information representing the shape of the insertion portion of the endoscope. Medical support device.

13. Equipped with a processor, The aforementioned processor, By inputting the cumulative result of the changes in feature information contained in each of the multiple endoscopic images obtained by imaging the inside of a tubular organ with an endoscope inserted into the tubular organ into a trained model, the trained model generates shape information representing the shape of the insertion part of the endoscope. Medical support device.

14. A medical support device according to any one of claims 1 to 13, The medical support device comprises an output device that outputs shape information generated by the medical support device, and / or information based on the shape information generated by the medical support device. Endoscopic system.

15. By inputting multiple endoscopic images obtained by imaging the inside of a tubular organ with an endoscope inserted into the tubular organ into a trained model, the trained model generates rotational information about the long axis of the insertion part of the endoscope, and This includes generating shape information representing the shape of the insertion portion based on the rotation information. Medical support methods.

16. By inputting multiple endoscopic images obtained by imaging the inside of a tubular organ with an endoscope inserted into the tubular organ into a trained model, the trained model generates rotational information about the long axis of the insertion part of the endoscope, and A program for causing a computer to perform a process that includes generating shape information representing the shape of the insertion part based on the rotation information.

Images