Program, information processing method, information processing device, and diagnosis support system

Abstract

The program of the present application causes a computer to execute the following processing: acquire an endoscope image of a subject from an endoscope; acquire a three-dimensional medical image of the inside of the subject by a unit capable of imaging the inside of the subject with a three-dimensional image using an X-ray CT, an X-ray cone beam CT, an MRI-CT, an ultrasonic diagnostic apparatus, or the like; generate a virtual endoscope image reconstructed from the three-dimensional medical image on the basis of the acquired endoscope image; derive distance image information in the endoscope image on the basis of the virtual endoscope image and the endoscope image; and output operation support information on the endoscope operation on the basis of the distance image information and the three-dimensional medical image.

Description

TECHNICAL FIELD

[0001] The present technology relates to a program, an information processing method, an information processing apparatus, and a diagnosis support system.

[0002] This application claims priority based on Japanese Application No. 2020-056712 filed on March 26, 2020, and incorporates by reference the entire disclosure of the Japanese Application. BACKGROUND

[0003] In a tumor examination of a patient, in particular, in a tubular organ site such as a trachea and bronchus, an upper digestive tract, a pancreas, a biliary tract, or an intestinal tract, an endoscope is inserted into the relevant tubular organ site, and most of the examination is performed by an image of the inserted endoscope. However, in two-dimensional image information of the endoscope image, the distance between pixels is not known, and there is a geometric distortion of the image, and the error at the time of image measurement is large, and thus it is difficult to directly provide image diagnosis support information using the endoscope image. In this regard, the virtual endoscope disclosed in Patent Literature 1 provides a virtual endoscope image using data of an X-ray CT (Computed Tomography) image. The virtual endoscope image is created from a three-dimensional image of the X-ray CT.

[0004] Prior Art Documents

[0005] Patent Literature

[0006] Patent Literature 1: Japanese Patent Application Publication No. 2002-238887 SUMMARY

[0007] Problems to be Solved by the Invention

[0008] However, the virtual endoscope disclosed in Patent Literature 1 displays only a cross-sectional X-ray CT reconstruction image (virtual endoscope image), and does not consider providing information related to endoscope operation based on the endoscope image and the virtual endoscope image to perform diagnosis support.

[0009] On the other hand, an object of the present application is to provide a program and the like that can provide information on endoscope operation to perform effective diagnosis support.

[0010] Technical Solution for Solving the Problem

[0011] The program of one embodiment of the present disclosure causes a computer to execute processing of: acquiring an endoscope image of a subject from an endoscope; acquiring a three-dimensional medical image of an inside of the subject, which is imaged with a three-dimensional image by a unit that is able to image the inside of the subject using an X-ray CT, an X-ray cone beam CT, an MRI-CT, an ultrasonic diagnostic apparatus, or the like; generating a virtual endoscope image reconstructed from the three-dimensional medical image on the basis of the acquired endoscope image; deriving distance image information in the endoscope image on the basis of the virtual endoscope image and the endoscope image; and outputting operation support information on the endoscope operation on the basis of the distance image information and the three-dimensional medical image.

[0012] The information processing method of one embodiment of the present disclosure causes a computer to execute processing of: acquiring an endoscope image of a subject from an endoscope; acquiring a three-dimensional medical image of an inside of the subject, which is imaged with a three-dimensional image by a unit that is able to image the inside of the subject using an X-ray CT, an X-ray cone beam CT, an MRI-CT, an ultrasonic diagnostic apparatus, or the like; generating a virtual endoscope image reconstructed from the three-dimensional medical image on the basis of the acquired endoscope image; deriving distance image information in the endoscope image on the basis of the virtual endoscope image and the endoscope image; and outputting operation support information on the endoscope operation on the basis of the distance image information and the three-dimensional medical image.

[0013] The information processing apparatus of one embodiment of the present disclosure includes: an endoscope image acquisition unit that acquires an endoscope image of a subject from an endoscope; a three-dimensional medical image acquisition unit that acquires a three-dimensional medical image of an inside of the subject, which is imaged with a three-dimensional image by a unit that is able to image the inside of the subject using an X-ray CT, an X-ray cone beam CT, an MRI-CT, an ultrasonic diagnostic apparatus, or the like; a generation unit that generates a virtual endoscope image reconstructed from the three-dimensional medical image on the basis of the acquired endoscope image; a derivation unit that derives distance image information in the endoscope image on the basis of the virtual endoscope image and the endoscope image; and an output unit that outputs operation support information on the endoscope operation on the basis of the distance image information and the three-dimensional medical image.

[0014] The diagnostic support system of one embodiment of the present disclosure includes an endoscope, an automatic operation mechanism that performs automatic operation of the endoscope, and an endoscope processor that acquires an endoscope image of a subject from the endoscope. The endoscope processor includes a three-dimensional medical image acquisition unit that acquires a three-dimensional medical image of the inside of the subject by a unit that can perform imaging of the inside of the subject with a three-dimensional image using an X-ray CT, an X-ray cone beam CT, an MRI-CT, an ultrasonic diagnostic apparatus, or the like, a generation unit that generates a virtual endoscope image reconstructed from the three-dimensional medical image on the basis of the acquired endoscope image, an extraction unit that extracts distance image information in the endoscope image on the basis of the virtual endoscope image and the endoscope image, and an output unit that outputs operation support information on the endoscope operation to the automatic operation mechanism on the basis of the distance image information and the three-dimensional medical image. The automatic operation mechanism performs automatic operation of the endoscope in accordance with the operation support information output from the output unit.

[0015] Effects of Invention

[0016] According to the present disclosure, a procedure for providing information on endoscope operation and performing effective diagnostic support can be provided. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a schematic view illustrating an outline of a diagnostic support system according to Embodiment 1.

[0018] Figure 2 is a block diagram illustrating a structure example of an endoscope device included in the diagnostic support system.

[0019] Figure 3 is a block diagram illustrating a structure example of an information processing device included in the diagnostic support system.

[0020] Figure 4 is an explanatory view illustrating a data layout of an endoscope image DB.

[0021] Figure 5 is an explanatory view illustrating a process of outputting operation support information using an operation information learning model.

[0022] Figure 6 is an explanatory view illustrating a process of outputting a degree of agreement with an endoscope image using a degree of agreement learning model.

[0023] Figure 7 is a functional block diagram illustrating a function unit included in a control unit of the information processing device.

[0024] Figure 8 is an explanatory view illustrating an endoscope insertion distance (value of S coordinate).

[0025] Figure 9 is a diagram illustrating a correlation between an endoscope image and a three-dimensional medical image.

[0026] Figure 10 is a flowchart showing an example of processing steps performed by the control section of the information processing apparatus.

[0027] Figure 11 is a diagram illustrating one mode of an integrated image display screen.

[0028] Figure 12 is a functional block diagram illustrating a functional section included in the control section of the information processing apparatus according to Embodiment 2 (bending history).

[0029] Figure 13 is a flowchart showing an example of processing steps performed by the control section of the information processing apparatus.

[0030] Figure 14 is a diagram illustrating an outline of a diagnosis support system according to Embodiment 3 (automatic operation mechanism).

[0031] Figure 15 is a functional block diagram illustrating a functional section included in the control section of the information processing apparatus. DETAILED DESCRIPTION

[0032] (Embodiment 1)

[0033] Hereinafter, the present application will be specifically described with reference to the accompanying drawings showing embodiments thereof. Figure 1 is a diagram illustrating an outline of a diagnosis support system S according to Embodiment 1. The diagnosis support system S includes an endoscope apparatus 10, and an information processing apparatus 6 communicably connected to the endoscope apparatus 10.

[0034] The endoscope apparatus 10 transmits an image (captured image) captured by an imaging element of an endoscope 40 to an endoscope processor 20, and various image processing such as gamma correction, white balance correction, and shading correction are performed by the endoscope processor 20, thereby generating an endoscope image in a state that is easy for an operator to visually recognize. The endoscope apparatus 10 outputs (transmits) the generated endoscope image to the information processing apparatus 6. After the information processing apparatus 6 acquires the endoscope image transmitted from the endoscope apparatus 10, various information processing is performed based on the endoscope image, and information on diagnosis support is output.

[0035] The endoscope apparatus 10 includes the endoscope processor 20, the endoscope 40, and a display apparatus 50. The display apparatus 50 is, for example, a liquid crystal display apparatus or an organic EL (Electro Luminescence) display apparatus.

[0036] The display device 50 is provided on the upper layer of the wheeled housing rack 16. The endoscope processor 20 is housed in the middle layer of the housing rack 16. The housing rack 16 is disposed in the vicinity of an endoscopy bed, which is not shown. The housing rack 16 has a drawer-type shelf for loading a keyboard 15 connected to the endoscope processor 20.

[0037] The endoscope processor 20 has a substantially rectangular parallelepiped shape, and has a touch panel 25 on one face. A reading section 28 is disposed on the lower portion of the touch panel 25. The reading section 28 is, for example, a USB connector, an SD (Secure Digital) card slot, or a CD-ROM (Compact Disc Read Only Memory) drive, or the like, which is a connection interface for reading and writing a portable recording medium.

[0038] The endoscope 40 has an insertion section 44, an operation section 43, a general-purpose cord 49, and an observer connector 48. The operation section 43 is provided with a control button 431. The insertion section 44 is elongated, and one end thereof is connected to the operation section 43 via an elbow section 45. The insertion section 44 has, in order from the operation section 43 side, a flexible section 441, a bending section 442, and a distal end section 443. The bending section 442 is bent in response to an operation of a bending knob 433. In the insertion section 44, a physical detection device such as a 3-axis acceleration sensor, a gyro sensor, a geomagnetic sensor, a magnetic coil sensor, or a colonoscope insertion shape observation device (Colonavi) is installed, and a detection result of the physical detection device can be acquired when the endoscope 40 is inserted into a subject.

[0039] The general-purpose cord 49 is elongated, and one end thereof is connected to the operation section 43, and the other end thereof is connected to the observer connector 48. The general-purpose cord 49 is flexible. The observer connector 48 has a substantially rectangular parallelepiped shape. On the observer connector 48, a gas / water supply port 36 (see FIG. 2) for connecting a gas / water supply tube is provided. Figure 2 ).

[0040] Figure 2 is a block diagram showing a configuration example of the endoscope device 10 included in the diagnosis support system S. The control section 21 is an arithmetic control device for executing a program in the present embodiment. The control section 21 uses one or a plurality of CPUs (Central Processing Unit), GPUs (Graphics Processing Unit), or multicore CPUs, and the like. The control section 21 is connected to each of the hardware sections for constituting the endoscope processor 20 via a bus.

[0041] The main storage device 22 is, for example, a storage device such as a SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, or the like. The main storage device 22 temporarily stores information required in a processing process by the control section 21, and a program being executed in the control section 21. The auxiliary storage device 23 is, for example, a storage device such as a SRAM, a flash memory, or a hard disk, and is a storage device having a larger capacity than the main storage device 22. For example, an acquired captured image, an endoscope image generated, or the like can be saved as intermediate data in the auxiliary storage device 23.

[0042] The communication section 24 is a communication module or a communication interface for communicating with the information processing device 6 via a network by wired or wireless, such as a narrowband wireless communication module such as wifi (registered trademark), Bluetooth (registered trademark), or a broadband wireless communication module such as 4G, LTE, or the like. The touch panel 25 includes a display section such as a liquid crystal display panel, and an input section layered on the display section. The communication section 24 can communicate with a CT device, an MRI device (refer to Figure 5 ), an ultrasonic diagnostic device, or a storage device (not illustrated) for saving data output from these devices.

[0043] The display device I / F 26 is an interface for connecting the endoscope processor 20 and the display device 50. The input device I / F 27 is an interface for connecting the endoscope processor 20 and an input device such as a keyboard 15.

[0044] The light source 33 is, for example, a high-brightness white light source such as a white LED, a xenon lamp, or the like. The light source 33 is connected to the bus via an omitted driver. The lighting, extinguishing, and brightness change of the light source 33 are controlled by the control section 21. Illumination light emitted from the light source 33 is incident on the optical connector 312. The optical connector 312 is engaged with the scope connector 48, and provides the illumination light to the endoscope 40.

[0045] The pump 34 generates a pressure for a gas / water supply function of the endoscope 40. The pump 34 is connected to the bus via an omitted driver. The on / off and pressure change of the pump 34 are controlled by the control section 21. The pump 34 is connected to a gas / water supply port 36 provided on the scope connector 48 via a water supply tank 35.

[0046] The following describes a functional overview of the endoscope 40 connected to the endoscope processor 20. Inside the scope connector 48, the general-purpose cord 49, the operation section 43, and the insertion section 44, a fiber bundle, a cable bundle, a gas supply tube, a water supply tube, and the like are inserted. Illumination light emitted from the light source 33 is emitted from an illumination window provided on the distal end section 443 via the optical connector 312 and the fiber bundle. The range irradiated by the illumination light is captured by the image pickup element provided on the distal end section 443. The captured image is transmitted from the image pickup element to the endoscope processor 20 via the cable bundle and the electrical connector 311.

[0047] The control section 21 of the endoscope processor 20 functions as the image processing section 211 by executing the program stored in the main storage 22. The image processing section 211 performs various image processing such as gamma correction, white balance correction, and shading correction on the image (captured image) output from the endoscope 40 and outputs the image as an endoscope image.

[0048] Figure 3 is a block diagram showing a configuration example of the information processing apparatus 6 included in the diagnosis support system S. The information processing apparatus 6 includes a control section 62, a communication section 61, a storage section 63, and an input / output I / F 64. The information processing apparatus 6 is, for example, a server apparatus, a personal computer, or the like. The server apparatus includes not only a single server apparatus but also a cloud server apparatus or a virtual server apparatus composed of a plurality of computers. The information processing apparatus 6 can also be provided as a cloud server on an external network accessible from the endoscope processor 20.

[0049] The control section 62 includes one or a plurality of CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), or the like, which is an arithmetic processing apparatus having a time measuring function, and can perform various information processing, control processing, and the like related to the information processing apparatus 6 by reading out and executing the program P stored in the storage section 63. Alternatively, the control section 62 can be composed of a quantum computer chip, and the information processing apparatus 6 can be a quantum computer.

[0050] The storage section 63 includes a volatile storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, and a non-volatile storage area such as EEPROM or a hard disk. The program P and data to be referred to at the time of processing are stored in advance in the storage section 63. As for the program P stored in the storage section 63, a program P read from a recording medium 632 readable by the information processing apparatus 6 can also be stored. In addition, the program P can also be downloaded from an external computer not shown connected to a communication network not shown and stored in the storage section 63. Actual files (example files of neural networks (NN)) constituting a plurality of learning models (91, 92) described later are stored in the storage section 63. These actual files can also be made a part of the program P. In addition, an endoscope image DB 631 (DataBase) described later can also be stored in the storage section 63.

[0051] The communication section 61 is a communication module or a communication interface for communicating with the endoscope apparatus 10 by wired or wireless means, such as a narrowband wireless communication module such as wifi (registered trademark), Bluetooth (registered trademark), or a broadband wireless communication module such as 4G, LTE. The communication section 61 can communicate with a CT apparatus, an MRI apparatus (refer to Figure 5 ), an ultrasonic diagnostic apparatus, or a storage apparatus (not shown) for storing data output from these apparatuses.

[0052] The input / output I / F 64 conforms to a communication standard such as USB or DSUB, and is a communication interface for serial communication with an external device connected to the input / output I / F 64. The input / output I / F 64 is connected to the display section 7 such as a display, the input section 8 such as a keyboard, and the control section 62 outputs a result of information processing based on an execution command or an event input from the input section 8 to the display section 7.

[0053] Figure 4is an explanatory diagram illustrating a data layout of the endoscope image DB 631. The endoscope image DB 631 is stored in the storage section 63 of the information processing apparatus 6 and is constituted by a database management software such as an RDBMS (Relational DataBase Management System) installed in the information processing apparatus 6. Alternatively, the endoscope image DB 631 can be stored in a predetermined storage area accessible from the information processing apparatus 6 such as a storage device communicably connected to the information processing apparatus 6. Alternatively, the endoscope image DB 631 can be stored in the main storage device 22 of the endoscope apparatus 10. That is, the predetermined storage area includes the storage section 63 of the information processing apparatus 6, the main storage device 22 of the endoscope apparatus 10, and a storage device accessible from the information processing apparatus 6 or the endoscope apparatus 10. The information processing apparatus 6 can acquire the endoscope image, the examination date and time, and the attribute information of the subject output from the endoscope processor 20 and log in the endoscope image DB 631. Alternatively, the endoscope image, the examination date and time, and the attribute information of the subject output directly from the endoscope processor 20 can be logged in the endoscope image DB 631 directly.

[0054] The endoscope image DB 631 includes, for example, a subject master table and an image table, and the two tables are associated with each other by a common item (metadata), that is, a subject ID.

[0055] The subject master table includes, for example, a subject ID, a gender, a date of birth, and an age as management items (metadata). In the item (field) of the subject ID, ID information for uniquely identifying the subject who has undergone the endoscopic examination is stored. In the items (fields) of the gender and the date of birth, biological attributes of the gender and the date of birth of the subject ID are stored, and in the item (field) of the age, the age at the current time calculated from the date of birth is stored. These gender and age are managed as biological information of the subject by the subject master table.

[0056] The image table includes, for example, a subject ID, an examination date and time, an endoscope image, a frame number, an S coordinate (insertion distance), a three-dimensional medical image, a viewpoint position, a viewpoint direction, and a virtual endoscope image as management items (metadata).

[0057] The item (field) of the subject ID is used in association with the biological attributes of the subject managed by the subject master table, and stores the respective ID value of the relevant subject. In the item (field) of the examination date and time, the date and time at which the subject ID relevant subject underwent an endoscopic examination is stored. In the item (field) of the endoscopic image, the endoscopic image of the relevant subject ID is stored as object data. The endoscopic image can be a still image of, for example, jpeg format or a moving image of, for example, avi format. In the item (field) of the endoscopic image, information indicating the storage location (file path) of the endoscopic image stored as a file can also be stored.

[0058] In the item (field) of the frame number, if the endoscopic image is a moving image, the frame number of the moving image is stored. By storing the frame number of the moving image, even if the endoscopic image is a moving image, it can be handled in the same manner as a still image, and can be associated with the position information (coordinates in the in-vivo coordinate system) of the three-dimensional medical image or the virtual endoscopic image described later.

[0059] In the item (field) of the S coordinate (insertion distance), as the value of the S coordinate, the insertion distance of the endoscope 40 at the time of shooting of the endoscopic image stored in the same record is stored. The derivation of the insertion distance (S coordinate) will be described later.

[0060] In the item (field) of the three-dimensional medical image, a three-dimensional medical image of, for example, DICOM (Digital Imaging and Communications in Medicine) format generated based on data output from a unit capable of shooting the in-vivo with a three-dimensional image, such as a CT device (X-ray CT, X-ray cone beam CT) or an MRI device (MRI-CT), an ultrasonic diagnostic device, or the like, is stored as object data. Alternatively, information indicating the storage location (file path) of the three-dimensional medical image stored as a file can also be stored.

[0061] In the item (field) of the viewpoint position, the coordinates of the endoscope 40 in the in-vivo at the time of shooting of the endoscopic image, that is, the coordinates in the three-dimensional medical image coordinate system, are stored. The derivation of the viewpoint position will be described later.

[0062] In the item (field) of the viewpoint direction, the direction of the endoscope 40 at the time of shooting of the endoscopic image, that is, the rotation angle in the three-dimensional medical image coordinate system (coordinates in the in-vivo coordinate system), is stored. The derivation of the viewpoint direction will be described later.

[0063] In the item (field) of the virtual endoscope image, the virtual endoscope image generated from the three-dimensional medical image is stored as the object data. Information indicating the storage location (file path) of the virtual endoscope image saved as a file can also be stored. The virtual endoscope image is generated from the three-dimensional medical image to perform matching processing with the endoscope image, for example, the virtual endoscope image most consistent with the endoscope image is registered in the same record as the endoscope image. The generation of the virtual endoscope image and the like will be described later.

[0064] Figure 5 is an explanatory diagram that explains processing of outputting operation support information using the operation information learning model 91. The information processing apparatus 6 learns based on training data that takes the distance image information and the body cavity information included in the three-dimensional medical image described later as question data, and takes the operation support information including at least one of the insertion direction, the insertion amount, the insertion speed, and the target point coordinates indicating the insertion destination of the endoscope 40 as answer data, thereby constructing (generating) a neural network (operation information learning model 91) that takes the distance image information and the three-dimensional medical image as input and takes the operation support information including the insertion direction of the endoscope 40 and the like as output.

[0065] The operation information learning model 91 that learns using the training data is assumed to be used as part of the artificial intelligence software, that is, a program module. As described above, the operation information learning model 91 is used in the information processing apparatus 6 that has the control section 62 (CPU or the like) and the storage section 63, and the related operation is executed by the information processing apparatus 6 having the arithmetic processing capability, thereby constituting a neural network system. That is, the control section 62 of the information processing apparatus 6 operates as follows: in accordance with the instructions of the operation information learning model 91 stored in the storage section 63, it performs arithmetic processing to extract the feature amounts of the distance image information and the three-dimensional medical image input to the input layer, and outputs the operation support information including the insertion direction of the endoscope 40 and the like from the output layer.

[0066] The input layer has a plurality of neurons for accepting distance image information and body cavity information included in the three-dimensional medical image, and passes the input distance image information and the body cavity information included in the three-dimensional medical image to the intermediate layer. Details will be described later, but the distance image information is information derived on the basis of a virtual endoscope image corresponding to the acquired endoscope image, and is information on distances between pixels in the virtual endoscope image. The endoscope image and the corresponding virtual endoscope image are images in which the same region inside the body is imaged, and therefore the distance image information corresponds to information on distances between pixels in the endoscope image. The distances between pixels refer to distances in a three-dimensional medical image coordinate system (a body coordinate system), such as distances obtained by adding depths of two body sites included in the virtual endoscope image. In addition, information on a viewpoint position and a direction of the endoscope 40 can be added to the distance image information input to the operation information learning model 91. The body cavity information included in the three-dimensional medical image is curved surface data representing a shape (an inner wall shape of an organ) of an organ or the like into which the endoscope 40 is inserted in a three-dimensional region including an imaging region of the virtual endoscope image from which the distance image information is derived, and the curved surface data can be constituted by, for example, a polynomial or a point set.

[0067] The intermediate layer has, for example, a single phase or a multi-layer structure composed of one or a plurality of fully connected layers, and a plurality of neurons included in the fully connected layers respectively output information representing activation or deactivation on the basis of values of the input distance image information and the body cavity information included in the three-dimensional medical image. The information processing apparatus 6 optimizes parameters used in the operation processing of the intermediate layer, for example, using a backpropagation method or the like.

[0068] The output layer has one or a plurality of neurons outputting operation support information including an insertion direction of the endoscope 40 or the like, and outputs the operation support information on the basis of information representing activation or deactivation of each neuron output from the intermediate layer. The operation support information including the insertion direction of the endoscope 40 or the like can be, for example, information representing a plurality of coordinate values in a three-dimensional medical image coordinate system (a body coordinate system) and a rotation angle of the tip of the endoscope 40 sequentially passing through in the insertion direction, in a vector form. In addition, the operation support information can include a velocity component corresponding to a movement amount between adjacent coordinate values when the endoscope 40 sequentially passes through.

[0069] Distance image information and three-dimensional medical image problem data (problem data) used as training data, and operation support information including an insertion direction of the endoscope 40 or the like having relevance to these information are stored in large quantities as examination result data of the endoscope 40 performed at each medical institution, and by using these result data, training data can be generated in large quantities for learning the operation information learning model 91.

[0070] Figure 6 is an explanatory diagram that explains a process of outputting the degree of consistency with the endoscope image using the degree of consistency learning model 92. The information processing apparatus 6 learns training data in which the endoscope image and the virtual endoscope image are set as question data and information on the degree of consistency of the two images is set as answer data, thereby constructing (generating) a neural network (degree of consistency learning model 92) that inputs the endoscope image and the virtual endoscope image and outputs information such as a value indicating the degree of consistency of the two images. As with the operation information learning model 91, the degree of consistency learning model 92 that has learned using the training data is assumed to be used as a part of the program module of the artificial intelligence software.

[0071] The input layer has a plurality of neurons for accepting the pixel values of the endoscope image and the virtual endoscope image as input, and passes the input pixel values to the intermediate layer. The intermediate layer has a plurality of neurons for extracting the image feature amounts of the endoscope image and the virtual endoscope image, and passes the extracted image feature amounts of the two images to the output layer. The output layer has one or more neurons for outputting information on the degree of consistency of the input endoscope image and virtual endoscope image such as a value indicating the degree of consistency, and outputs the information on the degree of consistency based on the image feature amounts of the two images output from the intermediate layer.

[0072] When the degree of consistency learning model 92 is, for example, a CNN (Convolutional Neural Network), the intermediate layer has a structure in which a convolution layer for convolving the pixel values of each pixel input from the input layer and a pooling layer for mapping (compressing) the pixel values convolved in the convolution layer are alternately connected, and finally extracts the feature amounts of the endoscope image and the virtual endoscope image while compressing the pixel information of the endoscope image and the virtual endoscope image. The output layer includes, for example, a fully connected layer in which a cosine similarity derived from the inner product of the feature amount vectors of the image feature amounts of the two images is derived, and a soft-max layer in which a value (determination value) indicating the degree of consistency is derived based on the cosine similarity and output as information on the degree of consistency.

[0073] In constructing (generating) the consistency learning model 92, a knowledge base (learned model) such as a DCNN installed in a VGG16 model (caffemodel: VGG_ILSVRC_16_layers) can also be used to construct the consistency learning model 92 by performing transfer learning based on training data based on the endoscope image and the virtual endoscope image. The endoscope images used as the training data and the virtual endoscope images corresponding to the endoscope images are saved in large quantities as a result of the endoscope 40 examination and the CT device and the like performed at each medical institution, and by using these result data, it is possible to generate a large amount of training data for learning the consistency learning model 92.

[0074] In the present embodiment, the operation information learning model 91 and the consistency learning model 92 are described as examples of neural networks (NNs) such as CNNs, but these learning models (91, 92) are not limited to NNs, and can be learning models (91, 92) constructed using other learning algorithms such as SVMs (Support Vector Machines), Bayesian networks, and regression trees.

[0075] Figure 7 is a functional block diagram illustrating functions of the functional units included in the control unit of the information processing apparatus. The control unit 21 of the endoscope processor 20 (endoscope apparatus 10) functions as the image processing unit 211 by executing a program stored in the main storage 22. The control unit 62 of the information processing apparatus 6 functions as the acquisition unit 621, the viewpoint position derivation unit 622, the virtual endoscope image generation unit 623, the consistency determination unit 624, the distance image information derivation unit 625, and the operation support information output unit 626 by executing a program P stored in the storage unit 63.

[0076] The image processing unit 211 of the endoscope processor 20 performs various image processing such as gamma correction, white balance correction, and shading correction on the image (captured image) output from the endoscope, and outputs the image as an endoscope image. The image processing unit 211 outputs (transmits) the generated endoscope image and the examination date and time based on the time of capturing the endoscope image to the information processing apparatus 6. The image processing unit 211 can also output the subject ID input from the keyboard 15 to the information processing apparatus 6. In order to measure the surrounding environment of the endoscope 40, the image processing unit 211 can also output information on the insertion distance (S coordinate) of the endoscope 40 output from a sensor disposed on the insertion portion 44 (flexible tube) of the endoscope 40 to the information processing apparatus 6. The image processing unit 211 can also superimpose the information on the insertion distance of the endoscope 40 acquired from the sensor on the endoscope image and display the image in the display apparatus, for example.

[0077] A sensor for acquiring the distance of the insertion of the endoscope 40 into the body, i.e., the S coordinate, includes, for example, a temperature sensor, a light sensor, a pressure sensor, a wetness sensor (electrode), and a humidity sensor. For example, when the sensor is a light sensor, the light sensor is disposed inside the insertion section 44 (flexible tube), and light can be received even if the insertion section 44 (flexible tube) is inserted into the body. Thus, the portion where the light sensor receives more light can be recognized as outside the body, and the portion where the light sensor receives less light can be recognized as inside the body. Then, the control section 21 of the endoscope processor 20 can determine the light sensor located at the body cavity insertion site, i.e., the boundary position, based on the signal obtained by the light sensor, and thereby derive the distance (length) of the insertion section 44 (flexible tube) inserted into the body, i.e., the S coordinate.

[0078] If it is an upper endoscope, a roller encoder is attached to the unillustrated mouthpiece or the like that is connected to the insertion section 44 (flexible tube), and the roller encoder is rotated by an amount corresponding to the distance of the insertion section 44 (flexible tube) inserted into the body, whereby the distance of the insertion of the endoscope 40 into the body, i.e., the S coordinate, can be acquired. The roller encoder of the mouthpiece or the like is rotated as the insertion section 44 (flexible tube) is advanced and retracted, and the length between the distal end section 443 of the endoscope 40 inserted into the body and, for example, an opening section that communicates with a lumen such as a mouth or a nose, i.e., the insertion distance of the insertion section 44 (flexible tube), can be measured. The roller encoder is electrically connected to the endoscope processor 20, and transmits the measured distance to the endoscope processor 20. Alternatively, an optical or magnetic encoder can be used instead of the roller encoder.

[0079] In addition, if it is a lower endoscope, the insertion distance of the endoscope can be measured by attaching an object corresponding to a mouthpiece to the anal portion. When an auxiliary device for measuring the insertion distance of the endoscope 40 is attached to the body cavity insertion site that is the entrance of the subject, the distance of the insertion of the endoscope 40 into the body, i.e., the S coordinate, can be acquired by measuring the passing distance of the endoscope 40. The auxiliary device can be, for example, a magnetic field scale such as a linear scale attached to the insertion section (flexible tube) 44 and a linear magnetic head attached to the mouthpiece to measure the distance, or can be a mouthpiece of the endoscope 40 to which a roller is attached. In addition, if the endoscope 40 is inserted into a portion such as a nose or an anus, an auxiliary device similar to the mouthpiece, to which a roller is attached, can be used. Furthermore, a chip that records the insertion distance at constant intervals can be built into the insertion section (flexible tube) 44 of the endoscope 40. The endoscope processor 20 can acquire the distance of the insertion of the endoscope 40 into the body, i.e., the S coordinate, based on the S coordinate information recorded by the chip, which is obtained by the mouthpiece or the like.

[0080] The acquisition unit 621 acquires the subject ID, the examination date and time, the endoscope image, and the S coordinate (insertion distance) output from the endoscope processor 20. The acquisition unit 621 acquires the three-dimensional medical image of the subject on the basis of the acquired subject ID, the three-dimensional medical image being output from a unit capable of imaging the inside of the body with a three-dimensional image, such as a CT device, a cone beam CT device, or an MRI device, an ultrasonic diagnostic device, or the like. When the three-dimensional medical image output from another examination device capable of imaging the inside of the body with a three-dimensional image, such as a CT device, a cone beam CT device, or an MRI device, an ultrasonic diagnostic device, or the like, has been stored in, for example, an external server (not shown), the information processing device 6 can access the external server and acquire the three-dimensional medical image of the subject on the basis of the subject ID output from the endoscope processor 20.

[0081] The three-dimensional medical image is an image represented by volume data composed of tomographic data output from a unit capable of imaging the inside of the body with a three-dimensional image, such as a CT device, a cone beam CT device, or an MRI device, an ultrasonic diagnostic device, or the like, and an image represented by volume data output from a multi-slice CT device and an X-ray cone beam CT device using an X-ray flat panel. When an X-ray CT device or a cone beam CT device is used, the image can be taken with Dual Energy by, for example, an X-ray CT, and the composition (volume composition) of each pixel of the three-dimensional medical image is known by effective mass number (effective-Z). When an MRI device is used, the image can be attached with information on the composition (volume composition) of each pixel of the three-dimensional medical image, such as fat or lactic acid.

[0082] The acquisition unit 621 outputs the acquired S coordinate to the viewpoint position deriving unit 622. The viewpoint position deriving unit 622 derives the coordinate (coordinate in the body coordinate system) of the three-dimensional medical image corresponding to the S coordinate on the basis of the acquired S coordinate, that is, the viewpoint position at which the distal end portion 443 of the endoscope 40 is located at the time of imaging by the endoscope 40. Figure 8 is an explanatory view showing the insertion distance of the endoscope 40 (value of the S coordinate). As shown in the drawing, the digestive organ or the like imaged by the endoscope 40 is visualized in a three-dimensional shape by the three-dimensional medical image. A space is formed inside the inner wall of the digestive organ or the like, and the space becomes an insertion path into which the endoscope 40 is inserted. The insertion distance of the endoscope 40, that is, the S coordinate, is inside the insertion path (inside the inner wall of the digestive organ or the like), and the path length of the insertion path is substantially equal to the insertion distance at the position, and therefore, on the basis of the S coordinate, the coordinate of the distal end portion 443 of the endoscope 40 located inside the inner wall of the digestive organ or the like can be derived. The viewpoint position deriving unit 622 outputs information on the derived viewpoint position to the virtual endoscope image generating unit 623.

[0083] The acquisition unit 621 outputs the acquired three-dimensional medical image to the virtual endoscope image generation unit 623. The virtual endoscope image generation unit 623 generates a virtual endoscope image based on the acquired three-dimensional medical image and the viewpoint position acquired from the viewpoint position derivation unit 622. The virtual endoscope image is generated (reconstructed) based on a three-dimensional medical image of X-ray CT or MRI or X-ray cone beam CT that captures a trachea and bronchus or a tubular organ such as an intestinal tract, and is an image that shows the inside of an organ (inside of a body cavity) of the three-dimensional medical image by a virtual endoscope. For example, a CT image can be captured in a state where air is filled in a large intestine, and a virtual endoscope image of the large intestine can be generated (reconstructed) by volume rendering of the three-dimensional medical image obtained by the capturing from the inside of the large intestine.

[0084] The virtual endoscope image generation unit 623 extracts voxel data of an organ in a subject from the acquired three-dimensional medical image. The organ includes, for example, a large intestine, a small intestine, a kidney, a bronchus, or a blood vessel, but is not limited thereto, and can be another organ. In the present embodiment, voxel data of the large intestine is extracted and acquired. For example, as a method of extracting a large intestine region, specifically, first, axial tomographic images of a plurality of cross sections perpendicular to a body axis are reconstructed based on the three-dimensional medical image, and for each of the axial tomographic images, a boundary between a body surface and inside of the body is obtained based on an X-ray CT value of an X-ray absorption coefficient as a threshold value, and the inside of the body surface is separated into an outside and an inside region. For example, for the reconstructed axial tomographic images, a binarization process is performed according to the X-ray CT value, and a contour is extracted by a contour extraction process, and the inside of the extracted contour is extracted as an inside (human body) region. Next, for the axial tomographic images of the inside region, a binarization process is performed according to a threshold value, and candidates of the large intestine region in each of the axial tomographic images are extracted. Specifically, since there is air in the large intestine tube, a threshold value corresponding to the air CT value (for example, less than or equal to -600 HU (Hounsfield Unit)) is set, a binarization process is performed, and the air region in the inside of each of the axial tomographic images is extracted as a candidate of the large intestine region. The virtual endoscope image generation unit 623 reconstructs a center projection image in which voxel data in a plurality of ray directions extending in a radial line shape is projected onto a predetermined projection surface as a virtual endoscope image with a sight line vector based on the viewpoint position and a rotation angle set as a sight line direction. As a specific method of the center projection, for example, a known volume rendering method or the like can be used.

[0085] The virtual endoscope image generation section 623 generates a plurality of candidate virtual endoscope images by successively changing the viewpoint direction, i.e., the rotation angles (θx, θy, θz) in the three-dimensional medical image coordinate system, in predetermined units of, for example, 1°, starting from a viewpoint position corresponding to the distal end portion 443 of the endoscope 40. That is, the virtual endoscope image generation section 623 can generate a plurality of virtual endoscope images by projecting the three-dimensional shape of the inner wall of the digestive organ from the viewpoint position inside the digestive organ determined from the three-dimensional medical image, according to a plurality of rotation angles set as the viewpoint direction. The virtual endoscope image generation section 623 associates the plurality of virtual endoscope images generated and the viewpoint direction (rotation angles) used when generating the virtual endoscope images, and outputs them to the coincidence degree determination section 624.

[0086] The acquisition section 621 outputs the acquired endoscope image to the coincidence degree determination section 624. The coincidence degree determination section 624 determines the virtual endoscope image that is most coincident with the acquired endoscope image and the viewpoint direction (rotation angles) used when generating the most coincident virtual endoscope image, based on the acquired endoscope image, the plurality of virtual endoscope images acquired from the virtual endoscope image generation section 623, and the viewpoint direction (rotation angles) used when generating the virtual endoscope images. The coincidence degree determination section 624 derives the coincidence degree of the endoscope image and the virtual endoscope image by comparing the acquired endoscope image with the plurality of virtual endoscope images, respectively.

[0087] The coincidence degree determination section 624 includes a coincidence degree learning model 92 that outputs information indicating the degree of coincidence of the two images, such as a value, based on the input endoscope image and virtual endoscope image. The coincidence degree determination section 624 can also input the acquired endoscope image and virtual endoscope image into the coincidence degree learning model 92, and output the virtual endoscope image that becomes the highest value in the value (probability value) indicating the degree of coincidence output from the coincidence degree learning model 92 as the virtual endoscope image corresponding to the endoscope image.

[0088] Alternatively, the degree-of-consistency determination section 624 is not limited to the case where the degree-of-consistency learning model 92 is included, and, for example, the degree of consistency can be measured by taking an index such as the correlation between the shadow image of the endoscope image and the shadow image of the virtual endoscope image. If the degree of consistency of the virtual endoscope image with the endoscope image is to be viewed quantitatively, the degree of consistency can be viewed by looking at the degree of correlation of the shadow image information obtained from the luminance information, and determining the degree of consistency. Alternatively, the degree-of-consistency determination section 624 can also compare the degree of similarity between the plurality of virtual endoscope images configured and the endoscope image. The degree of similarity of the two images can be compared by known image processing, and either one of matching at the pixel data level or matching at the feature level extracted from the images can be used. The virtual endoscope image determined to have the highest degree of consistency with the endoscope image and the viewpoint direction (rotation angle) used to generate the virtual endoscope image can be registered in the endoscope image DB by the degree-of-consistency determination section 624. The virtual endoscope image determined to have the highest degree of consistency with the endoscope image and the viewpoint position and direction (rotation angle) used to generate the virtual endoscope image are output to the distance image information derivation section 625 by the degree-of-consistency determination section 624.

[0089] In the present embodiment, the degree-of-consistency determination section 624 determines the virtual endoscope image that is most consistent with the acquired endoscope image, but is not limited thereto. The degree-of-consistency determination section 624 can also determine the virtual endoscope image having a degree of consistency of a predetermined value or more as a virtual endoscope image that can be substantially equally viewed with the acquired endoscope image. By determining the virtual endoscope image having a degree of consistency of a predetermined value or more, it is not necessary to compare all of the virtual endoscope images generated as candidates, and reduction of the computational load and processing time of the information processing apparatus 6 can be achieved. Alternatively, the degree-of-consistency determination section 624 can also determine the virtual endoscope image having the smallest difference (difference index) from the acquired endoscope image as a virtual endoscope image that can be substantially equally viewed with the acquired endoscope image. The difference (difference index) of the endoscope image and the virtual endoscope image corresponds to the inverse of the degree of consistency of the endoscope image and the virtual endoscope image, and thus, by using an image comparison engine or the like that derives such a difference (difference index), the virtual endoscope image that can be substantially equally viewed with the acquired endoscope image can be efficiently acquired.

[0090] When the degree of consistency is not a predetermined value or more, the degree-of-consistency determination section 624 can generate a plurality of virtual endoscope images again by fine-tuning the viewpoint position acquired from the viewpoint position derivation section 622, and derive the degree of consistency of the plurality of virtual endoscope images generated again with the endoscope image, and determine the virtual endoscope image having the highest degree of consistency.

[0091] The distance image information deriving section 625 derives distance image information based on the virtual endoscope image acquired from the degree of coincidence determining section 624. The distance image information is information on distances between pixels in the virtual endoscope image. The distance between pixels refers to a distance in a three-dimensional medical image coordinate system (in-vivo coordinate system), for example, a distance obtained by adding the depths of two in-vivo sites included in the virtual endoscope image. The virtual endoscope image is an image obtained by projecting and transforming a three-dimensional medical image into two dimensions, and any point in the virtual endoscope image corresponds to a point in the three-dimensional medical image, which indicates the same position of an in-vivo site. The any point in the virtual endoscope image can be a pixel number (pixel coordinate) of the smallest unit in the image, or can be, for example, a center portion of a local region (a region composed of a plurality of pixels) that determines a predetermined in-vivo site. In this way, by specifying any two points, the distance between the two points in the three-dimensional medical image coordinate system can be derived. That is, the distance in the distance image information corresponds to the distance between two points in the three-dimensional medical image coordinate system corresponding to the two points in the virtual endoscope image.

[0092] The two points in the three-dimensional medical image are determined by the two points in the virtual endoscope image. Based on the coordinate values of the two points in the three-dimensional medical image determined, the distance and the vector of the two points can be derived. The distance and the vector of the two points in the three-dimensional medical image derived are assigned to the distance and the vector of the two points in the virtual endoscope image corresponding to the two points, and thus a distance image, that is, a virtual endoscope image in which distance information in the three-dimensional medical image coordinate system is added to the virtual endoscope image (distance image) can be generated. The distance image information deriving section 625 can output the virtual endoscope image (distance image) in which information on the distance between pixels is added as distance image information. In addition, the distance image information deriving section 625 can also output the position and the direction of the viewpoint of the endoscope 40 used when generating the virtual endoscope image are added.

[0093] The endoscope image corresponds to a virtual endoscope image reconstructed (generated) from a three-dimensional medical image based on the position (viewpoint position) and the direction of shooting (viewpoint direction) of the endoscope 40 that has shot the endoscope image. Therefore, the distance image information of the virtual endoscope image corresponding to the endoscope image can also be applied to the endoscope image. That is, the distance between two points in the endoscope image corresponds to the distance between two points in the virtual endoscope image corresponding to the endoscope image (distance in the distance image, distance in the three-dimensional medical image coordinate system). Therefore, by applying the distance image information included in the distance image (virtual endoscope image) to the endoscope image, the distance between in-vivo sites included in the endoscope image, the size of the in-vivo sites, and the like distance information (distance image information in the endoscope image) can be determined.

[0094] The operation support information output section 626 acquires the distance image information output from the distance image information derivation section 625 and the viewpoint position and direction (rotation angle) of the endoscope 40. The operation support information output section 626 acquires the three-dimensional medical image output from the acquisition section 621 and extracts the body cavity information included in the three-dimensional medical image. As described above, the body cavity information included in the three-dimensional medical image is curved surface data that represents, for example, the shape of an organ or the like into which the endoscope 40 is inserted (the inner wall shape of the organ) in a three-dimensional region including the imaging region of the virtual endoscope image from which the distance image information is derived.

[0095] The operation support information output section 626 includes an operation information learning model 91 that outputs operation support information including the insertion direction of the endoscope 40 and the like based on the input distance image information, the viewpoint position and direction of the endoscope 40, and the body cavity information included in the three-dimensional medical image represented by the curved surface data or the like. The operation support information output section 626 inputs the acquired distance image information, the viewpoint position and direction of the endoscope 40, and the body cavity information included in the three-dimensional medical image represented by the curved surface data or the like into the operation information learning model 91 and acquires the operation support information including the insertion direction of the endoscope 40 and the like output from the operation information learning model 91.

[0096] As described above, the operation support information output from the operation information learning model 91 includes, for example, information on the insertion direction, the insertion amount, or the insertion speed of the endoscope 40 from the viewpoint position and direction of the endoscope 40 to the target site representing the insertion destination at the time point at which the endoscope image is captured. This operation support information can be information in which a plurality of coordinate values in the three-dimensional medical image coordinate system (in-vivo coordinate system) and the rotation angle through which the distal end portion 443 of the endoscope 40 passes in order in the range from the viewpoint position of the endoscope 40 to the target site are represented in vector form or matrix form. By using the plurality of coordinate values and the rotation angle through which the endoscope 40 passes in order, the plurality of coordinate values can be connected as path points, and thus the path in which the endoscope 40 is inserted can be determined. The operation support information output section 626 can correct each coordinate value derived as a path point in accordance with the stiffness of the insertion portion (flexible tube) 44 and derive the coordinate value.

[0097] The operation support information output section 626 acquires the endoscope image output from the acquisition section 621, for example, generates image data in which the operation support information is superimposed on the endoscope image, and outputs the image data to the display section 7. The display section 7 displays the endoscope image in which the operation support information is superimposed on the endoscope image based on the image data acquired from the operation support information output section 626.

[0098] The subject ID, the examination date and time, the endoscope image, the S coordinate, the three-dimensional medical image, and the information on the viewpoint position and direction of the virtual endoscope image derived by the coincidence degree determination section 624, which are acquired by the acquisition section 621, are saved in association with each other in the endoscope image DB. That is, the control section of the information processing apparatus functions as a DB registration section, and registers and saves various images, information, or data acquired or derived by the acquisition section 621 and the coincidence degree determination section 624 in the endoscope image DB.

[0099] Figure 9 is an explanatory diagram regarding the association between the endoscope image and the three-dimensional medical image. In this diagram, the association in the three-dimensional medical image, the virtual endoscope image, and the endoscope image is expressed in an object-oriented manner.

[0100] As described above, the three-dimensional medical image, the virtual endoscope image, and the endoscope image registered in the endoscope image DB 631 are associated on the basis of the viewpoint position and the viewpoint direction at the time of imaging of the endoscope image. The viewpoint position corresponds to the coordinates (x, y, z) in the three-dimensional medical image coordinate system (in-vivo coordinate system). The viewpoint direction corresponds to the rotation angles (θx, θy, θz) on the x-axis, the y-axis, and the z-axis in the three-dimensional medical image coordinate system (in-vivo coordinate system).

[0101] Each pixel of the endoscope image corresponds to each pixel of the virtual endoscope image (the virtual endoscope image most coincident with the endoscope image). The virtual endoscope image is an image generated by projecting from the viewpoint position as a starting point, by performing vector transformation using the viewpoint vector defined by the viewpoint direction (rotation angle), on the basis of the three-dimensional medical image, and the coordinates in the three-dimensional medical image coordinate system (in-vivo coordinate system) are determined by the pixels of the virtual endoscope image.

[0102] As described above, since each pixel of the virtual endoscope image corresponds to each pixel of the endoscope image, the coordinates in the three-dimensional medical image coordinate system (in-vivo coordinate system) of the in-vivo site included in the endoscope image can be determined on the basis of the pixels of the virtual endoscope image. That is, the pixels (in-vivo sites) of the endoscope image can be associated with the coordinates in the three-dimensional medical image coordinate system (in-vivo coordinate system) with the virtual endoscope image as an intermediate medium.

[0103] The color information of the pixels of the endoscope image, the narrow band pixel information, and the like can also be added to the three-dimensional medical image, and the three-dimensional medical image can be registered in the endoscope image DB 631. When the pixel information of the endoscope image, such as the difference, the color information, and the like, is added to the three-dimensional medical image, it is preferable to perform brightness correction using the photographing light source 446. As described above, the distance between the pixel of the endoscope image and the viewpoint position (the position of the photographing light source 446) is derived on the three-dimensional medical image coordinate system. Therefore, the brightness included in the pixel information of the endoscope image can be corrected based on the reciprocal of the square of the derived distance. When there are a plurality of endoscope images including the pixel at the same coordinate on the three-dimensional medical image coordinate system, the nearest endoscope image can be given priority, and the endoscope image can be added to the three-dimensional medical image in a weighted average or a simple average manner according to the distance.

[0104] In photographing the three-dimensional medical image, when an X-ray CT apparatus or a cone beam CT apparatus is used, the image can be photographed by Dual Energy, for example, and the composition (body composition) of each pixel of the three-dimensional medical image can be known by effective mass number (effective-Z). In addition, when an MRI apparatus is used, the image can be added with information on the composition (body composition) of each pixel of the three-dimensional medical image, such as fat or lactic acid. As described above, by adding the effective mass number (effective-Z) and the information on the body composition, such as fat or lactic acid, to the composition of each pixel of the three-dimensional medical image, information can be provided to a doctor or the like, in which the added information is associated with the information of the endoscope image associated with the coordinate determined by each pixel of the three-dimensional medical image.

[0105] Figure 10 This is a flowchart showing an example of the processing steps performed by the control section 62 of the information processing apparatus 6. For example, the information processing apparatus 6 starts the processing of the flowchart based on the content input from the input section 8 connected to the apparatus.

[0106] The control section 62 of the information processing apparatus 6 acquires the relevant information of the examination date and time, the subject ID, the endoscope image, and the insertion distance output from the endoscope processor 20 (S101). The endoscope image acquired by the control section 62 from the endoscope processor 20 can be a still image or a moving image. The control section 62 acquires the relevant information of the insertion distance of the endoscope 40 output from the light sensor or the like, the examination date and time (the photographing date and time of the endoscope image), and the attribute information of the subject, such as the subject ID, at the same time as the endoscope image is acquired.

[0107] The control section 62 of the information processing apparatus 6 acquires a three-dimensional medical image output from an examination apparatus capable of imaging an inside of a body with a three-dimensional image, such as a CT apparatus or an MRI apparatus, an ultrasonic diagnostic apparatus, or the like (S102). The information processing apparatus 6 can be communicably connected to an examination apparatus capable of imaging an inside of a body with a three-dimensional image, such as a CT apparatus or an MRI apparatus, an ultrasonic diagnostic apparatus, or the like, and thereby acquire a three-dimensional medical image. Alternatively, when a three-dimensional medical image output from an examination apparatus capable of imaging an inside of a body with a three-dimensional image, such as a CT apparatus or an MRI apparatus, an ultrasonic diagnostic apparatus, or the like, has been stored in, for example, an external server (not illustrated), the information processing apparatus 6 can access the external server and acquire a three-dimensional medical image of a subject in accordance with a subject ID output from the endoscope processor 20. Alternatively, the endoscope processor 20 can be communicably connected to an examination apparatus capable of imaging an inside of a body with a three-dimensional image, such as a CT apparatus or an MRI apparatus, an ultrasonic diagnostic apparatus, or the like, and the control section 62 of the information processing apparatus 6 can acquire a three-dimensional medical image capable of imaging an inside of a body with a three-dimensional image, like a CT apparatus or an MRI apparatus, an ultrasonic diagnostic apparatus, via the endoscope processor 20.

[0108] The control section 62 of the information processing apparatus 6 derives a viewpoint position based on the insertion distance (S coordinate) (S103). The control section 62 acquires relevant information of the insertion distance (S coordinate) from, for example, a light sensor or the like disposed inside the insertion portion 44 (flexible tube) of the endoscope 40 via the endoscope processor 20, and derives a coordinate of the distal end portion 443 of the endoscope 40 inside the inner wall of a digestive organ or the like into which the endoscope 40 is inserted, based on the acquired insertion distance (S coordinate) and the three-dimensional medical image. The coordinate is a coordinate in a three-dimensional medical image coordinate system (in-vivo coordinate system) set with a predetermined point as an origin.

[0109] The control section 62 of the information processing apparatus 6 generates a plurality of candidate virtual endoscope images based on the viewpoint position (S104). The control section 62 sequentially changes a viewpoint direction, that is, a rotation angle (θx, θy, θz) in the three-dimensional medical image coordinate system, in a predetermined unit amount from the viewpoint position corresponding to the coordinate of the distal end portion 443 of the endoscope 40 as a starting point, and sequentially generates a plurality of candidate virtual endoscope images. For example, when the predetermined unit amount is 10°, the control section 62 can have 36 resolutions with respect to the rotation angle of each axis, that is, 46656 candidate virtual endoscope images can be generated.

[0110] The control section 62 of the information processing apparatus 6 determines a virtual endoscope image having the highest degree of consistency with the endoscope image from among the generated plurality of virtual endoscope images (S105). The control section 62 determines the virtual endoscope image having the highest degree of consistency with the endoscope image, for example, using the consistency learning model 92. Alternatively, the degree of consistency can be measured by taking an index such as a correlation between a shadow image of the endoscope image and a shadow image of the virtual endoscope image. The control section 62 determines the virtual endoscope image having the highest degree of consistency and the viewpoint position and direction (rotation angle) at the time of generation of the virtual endoscope image.

[0111] The control section 62 of the information processing apparatus 6 derives distance image information from the acquired virtual endoscope image or the like (S106). The control section 62 derives the distance image information, which is information on the distance between each pixel in the virtual endoscope image.

[0112] The control section 62 of the information processing apparatus 6 outputs operation support information on the basis of the distance image information and body cavity information included in the three-dimensional medical image (S107). The control section 62 extracts the body cavity information included in the acquired three-dimensional medical image as curved surface data representing the shape (inner wall shape of an organ) of an in-vivo organ or the like into which the endoscope 40 is inserted in a three-dimensional region including the imaging region of the virtual endoscope image from which the distance image information is derived. The control section 62 inputs the extracted curved surface data, the distance image information, and information representing the position and direction of the endoscope 40 to the operation information learning model 91 and outputs the operation support information output from the operation information learning model 91.

[0113] When the operation support information is output, the control section 62 can generate image data obtained by superimposing the operation support information on the endoscope image, the virtual endoscope image, or the three-dimensional medical image, and output the image data to, for example, the display section. The display section displays the endoscope image, the virtual endoscope image, or the three-dimensional medical image on which the operation support information is superimposed, on the basis of the image data output from the control section 62 of the information processing apparatus 6.

[0114] The operation support information superimposed on the endoscope image, the virtual endoscope image, or the three-dimensional medical image includes information on the direction of insertion, the amount of insertion, the speed of insertion, and a target point coordinate representing the destination of insertion of the endoscope 40, and thus useful information can be provided to an operator of the endoscope 40 such as a doctor, and diagnostic support can be provided to the doctor or the like.

[0115] Figure 11is an explanatory diagram showing one mode of the integrated image display screen 71. As described above, the control section 62 of the information processing apparatus 6 generates image data of an endoscope image, a virtual endoscope image, or a three-dimensional medical image including operation support information such as the insertion direction of the endoscope 40 superimposed thereon, and outputs it to the display section 7. The integrated image display screen 71 is an example of a display screen configured by this image data, and the display section 7 displays this integrated image display screen 71 based on this image data.

[0116] The integrated image display screen 71 includes, for example, a region displaying bibliographic items such as a subject ID, a region displaying an endoscope image, a region displaying a three-dimensional medical image, a region displaying a two-dimensional medical image, a region displaying a virtual endoscope image, and a region displaying relevant information of the currently displayed endoscope image and the viewpoint position at which the endoscope image was captured, and the like.

[0117] In the region displaying bibliographic items such as a subject ID, bibliographic items in terms of data management such as the subject ID adopted for determining a three-dimensional medical image corresponding to the endoscope image, the examination date and time of the endoscope 40, the generation date of the three-dimensional medical image, and the like are displayed.

[0118] In the region displaying an endoscope image, an endoscope image captured by the endoscope 40 at the current time is displayed in real time. In the endoscope image, operation support information such as the insertion direction of the endoscope 40 is superimposed and displayed. According to the setting of the display option described later, the endoscope image can be translucently displayed, and a body site inside a body wall surface displayed in the endoscope image and extracted from a virtual endoscope image corresponding to the endoscope image is displayed by a broken line or the like. The body site displayed by a broken line or the like in the translucent display can be, for example, a lesion candidate site extracted based on shape information of a body site determined from the three-dimensional medical image.

[0119] In the region displaying a three-dimensional medical image, a body site such as a digestive organ represented by the three-dimensional medical image is displayed as a three-dimensional object, and operation support information such as the viewpoint position of the endoscope 40 and the insertion direction from the viewpoint position is superimposed and displayed. By dragging an arbitrary portion of the three-dimensional object, the three-dimensional object can be rotated. In the three-dimensional medical image, the position of a lesion candidate site extracted based on shape information of a body site determined from the three-dimensional medical image can be displayed in a highlighted state, for example.

[0120] In the region displaying a two-dimensional medical image, a two-dimensional medical image projected from a region in which operation support information such as the insertion direction of the endoscope 40 is superimposed on the three-dimensional medical image is displayed. The projection vector adopted when generating the two-dimensional medical image can be determined in conjunction with the rotation state of the three-dimensional object displayed in the region displaying the three-dimensional medical image.

[0121] In the region where the virtual endoscope image is displayed, the virtual endoscope image having the highest degree of consistency with the endoscope image displayed in the region where the endoscope image is displayed is displayed. In the virtual endoscope image, as with the endoscope image, operation support information such as the insertion direction of the endoscope 40 can also be superimposed and displayed.

[0122] In the region where the viewpoint position and the like at which the endoscope image was captured is displayed, the position of the endoscope 40 in the body (the viewpoint position) and the direction of the viewpoint (the rotation angle) at the time when the endoscope image displayed in the region where the endoscope image is displayed was captured are displayed. As described above, the control section 62 (acquisition section 621) of the information processing apparatus 6 continuously acquires the endoscope image and the S coordinate indicating the insertion distance of the endoscope 40 from the endoscope 40 use processor, and continuously derives the position of the distal end portion of the endoscope 40 (the viewpoint position) based on the acquired S coordinate. In addition, the control section 62 (acquisition section 621) of the information processing apparatus 6 continuously derives the direction of the distal end portion of the endoscope 40 (the rotation angle) when determining the virtual endoscope image corresponding (equivalent) to the endoscope image based on the degree of consistency with the endoscope image. Therefore, in the region where the viewpoint position and the like at which the endoscope image was captured is displayed, the viewpoint position and the direction of the viewpoint of the endoscope 40 are displayed in real time in response to the operation of the endoscope 40 by the doctor or the like.

[0123] In the region where information of the currently displayed endoscope image is displayed, for example, the relevant information of the intrabody site or the pixel at the image center of the currently displayed endoscope image is displayed. As described above, the relevant information of the intrabody site (pixel) includes the X-ray-based material discrimination information, that is, the effective-Z in the three-dimensional medical image, or the relevant information of the composition (body composition) of each pixel of the three-dimensional medical image such as fat or lactic acid. Therefore, the relevant information of the effective-Z or the body composition extracted from the three-dimensional medical image based on the coordinate of the intrabody coordinate system indicating the image center of the endoscope image can be displayed in this region. In addition, as to whether there is a lesion in the intrabody site included in the currently displayed endoscope image, it can also be displayed in this region by using a learning model that inputs the endoscope image and outputs information on the presence or absence of a lesion. Such a learning model that outputs information on the presence or absence of a lesion based on the input endoscope image can use any object detection algorithm having a segmentation network function such as CNN or RCNN (Regions with Convolutional Neural Network), Fast RCNN, Faster RCNN, or SSD (Single Shot Multibox Detector), YOLO (You Only Look Once), and the like.

[0124] As described above, since the viewpoint position of the endoscope 40 can be continuously derived in correspondence with the operation of the endoscope 40 by the doctor or the like, relevant information such as the distance between the viewpoint position and the image center of the currently displayed endoscope image can also be calculated, and the calculated distance (distance from the viewpoint position to the image center pixel) can be displayed in the region.

[0125] The integrated image display screen 71 includes an input region that accepts input regarding the display form, in which, for example, a display option field 711 that sets a display option, and a display mode selection field 712 that accepts selection of a plurality of display modes are arranged.

[0126] In the display option field 711, a switch for setting whether to perform semi-transparent display of the endoscope image, the virtual endoscope image, or both images is provided. By checking the switch, the in-vivo site inside the in-vivo wall surface displayed by the endoscope image can be semi-transparently processed based on the shape data of the in-vivo site included in the virtual endoscope image or the three-dimensional medical image, and displayed with a broken line or the like. As described above, the in-vivo site inside the in-vivo wall surface can be, for example, a candidate lesion site extracted based on the shape information of the in-vivo site determined from the three-dimensional medical image.

[0127] In the display mode selection field 712, switches for selecting the virtual endoscope image, the two-dimensional medical image, and the three-dimensional medical image to be displayed together with the endoscope image are provided. By checking the switches corresponding to the images, the selected arbitrary image is displayed. According to the checking of each switch in the display mode selection field 712, it is possible to select to display any one or two of the virtual endoscope image, the two-dimensional medical image, and the three-dimensional medical image, or to display all of the three images. The display size of the image can be adjusted according to the number of images displayed.

[0128] According to the present embodiment, based on the endoscope image, a three-dimensional medical image of the inside of the subject is captured using an inspection device capable of capturing the inside of the subject with a three-dimensional image, such as an X-ray CT, an X-ray cone beam CT, an MRI-CT, or an ultrasonic diagnostic apparatus, a virtual endoscope image is reconstructed from the captured three-dimensional medical image, and distance image information in the endoscope image is derived based on the virtual endoscope image and the endoscope image. Thus, in addition to the endoscope image, the virtual endoscope image constituted by the three-dimensional medical image having three-dimensional spatial coordinate information is used, so that the distance image information in the endoscope image can be derived with high precision. Based on the derived distance image information and the three-dimensional medical image or the virtual endoscope image, operation support information regarding the endoscope operation can be effectively output. Note that the X-ray CT, the X-ray cone beam CT, the MRI-CT, or the ultrasonic diagnostic apparatus is an example of an inspection device capable of capturing the inside of the subject with a three-dimensional image, but is not limited thereto. The three-dimensional medical image is not limited to being acquired from any one of these inspection devices, but can be acquired from a plurality of inspection devices.

[0129] According to the present embodiment, since the operation support information includes the related information of the insertion direction or the insertion speed of the endoscope 40, by outputting the operation support information, the operator of the endoscope 40, such as a doctor, can be effectively provided with the diagnosis support regarding the endoscope 40 operation.

[0130] According to the present embodiment, the operation support information is superimposed and displayed on the endoscope image captured at the current time and the three-dimensional medical image, the two-dimensional medical image, or the virtual endoscope image corresponding to the endoscope image, for example, in the integrated image display screen 71. Thus, the visibility of the operation support information to the operator of the endoscope 40, such as a doctor, can be improved, and the operator of the endoscope 40, such as a doctor, can be effectively provided with the diagnosis support.

[0131] (Embodiment 2)

[0132] The information processing apparatus 6 of Embodiment 2 differs from Embodiment 1 in that the viewpoint position is corrected by the bending history acquired from the endoscope processor 20. Figure 12 is a functional block diagram illustrating functions of the functional sections included in the control section of the information processing apparatus according to Embodiment 2 (bending history).

[0133] The control section 21 of the processor 20 for endoscope acquires the bending history information of the endoscope 40 inserted into the body, and judges the insertion state of the endoscope 40 based on the acquired bending history information. For example, the control section 21 of the processor 20 for endoscope can detect the bending history information by using an endoscope insertion shape detection device (not shown) connected to the processor 20 for endoscope. The endoscope insertion shape detection device can be a device in which, for example, as disclosed in Japanese Patent Application Publication No. 2019-37643, a plurality of magnetic coils are arranged at predetermined intervals in the length direction of the insertion portion 44 inside the insertion portion 44 of the endoscope 40. The bending history information indicates bending-related physical parameters or information such as the bending angle, the bending direction, and the like.

[0134] As with Embodiment 1, the acquisition section 621 of the information processing apparatus 6 acquires the endoscope image and the like from the processor 20 for endoscope, and further acquires the bending history information. The acquisition section 621 outputs the acquired bending history information to the viewpoint position derivation section 622.

[0135] The viewpoint position derivation section 622 corrects the insertion distance (S coordinate) based on the acquired bending history information, and as with Embodiment 1, derives the viewpoint position based on the corrected insertion distance (S coordinate). The viewpoint position derivation section 622 detects the shape of the insertion portion 44 (for example, bending to the right by 30 degrees, and the like) by arithmetic processing according to the bending angle and the bending direction. The control section 21 recalculates (corrects) the insertion distance, that is, the S coordinate, based on the detected shape of the insertion portion 44. Thereafter, each functional section such as the virtual endoscope image generation section 623 performs the same processing as in Embodiment 1, and the operation support information output section 626 generates and outputs the operation support information as with Embodiment 1.

[0136] The viewpoint position derivation section 622 of the information processing apparatus 6 corrects the viewpoint position by the bending history acquired from the processor 20 for endoscope, but is not limited thereto. The control section 21 of the processor 20 for endoscope can correct the insertion distance based on the acquired bending history information, and output the corrected insertion distance to the information processing apparatus 6. The acquisition section 621 of the information processing apparatus 6 can acquire the viewpoint position corrected by the control section 21 of the processor 20 for endoscope based on the bending history information, and thereafter perform the same processing as in Embodiment 1.

[0137] Based on the information on the bending history, the information on the insertion distance, and the length of the insertion path of the endoscope 40 determined by the three-dimensional medical image, position information for associating the endoscope image with the three-dimensional medical image is derived. The information on the insertion distance is corrected using the information on the bending history, so that the accuracy of the insertion distance (S coordinate) can be improved. Therefore, the viewpoint position (coordinate) and the viewpoint direction (rotation angle) of the endoscope 40 in the three-dimensional medical image coordinate system at the time of capturing the endoscope image can be accurately determined, a suitable virtual endoscope image can be effectively generated, and the accuracy of the association between the endoscope image and the three-dimensional medical image can be further improved.

[0138] Figure 13 is a flowchart showing an example of processing steps performed by the control section of the information processing apparatus. For example, the information processing apparatus 6 starts the processing of this flowchart based on the content input from the input section 8 connected to the apparatus.

[0139] The control section 62 of the information processing apparatus 6 acquires the examination date and time, the subject ID, the endoscope image, the insertion distance, and the information on the bending history output from the endoscope processor 20 (S201). As in Embodiment 1, the control section 62 of the information processing apparatus 6 acquires the endoscope image and the like from the endoscope processor 20, and further acquires the information on the bending history detected by the endoscope insertion shape observation apparatus, for example, via the endoscope processor 20. Alternatively, the control section 62 of the information processing apparatus 6 can acquire the information on the bending history directly from the endoscope insertion shape observation apparatus.

[0140] The control section 62 of the information processing apparatus 6 acquires a three-dimensional medical image output from an examination apparatus capable of capturing the inside of the body with a three-dimensional image, such as a CT apparatus or an MRI apparatus, an ultrasonic diagnostic apparatus, and the like (S202). As in the processing S102 of Embodiment 1, the control section 62 of the information processing apparatus 6 performs the processing of S202.

[0141] The control section 62 of the information processing apparatus 6 corrects the insertion distance (S coordinate) based on the bending history output from the endoscope processor 20, and derives the viewpoint position (S203). The control section 62 derives the insertion portion 44 shape (for example, a rightward bending of 30 degrees, and the like) by arithmetic processing according to the bending angle and the bending direction included in the bending history, and recalculates (corrects) the insertion distance, that is, the S coordinate, based on the derived insertion portion 44 shape. The control section 62 derives the viewpoint position based on the corrected insertion distance (S coordinate), as in Embodiment 1.

[0142] The control section 62 of the information processing apparatus 6 generates a plurality of candidate virtual endoscope images based on the viewpoint position (S204). The control section 62 of the information processing apparatus 6 determines a virtual endoscope image having the highest degree of coincidence with the endoscope image among the generated plurality of virtual endoscope images (S205). The control section 62 of the information processing apparatus 6 derives distance image information from the acquired virtual endoscope image or the like (S206). The control section 62 of the information processing apparatus 6 outputs operation support information based on the distance image information and the body cavity information included in the three-dimensional medical image (S207). As with the processes S104, S105, S106, and S107 of Embodiment 1, the control section 62 of the information processing apparatus 6 performs the processes S204, S205, S206, and S207.

[0143] According to the present embodiment, the coordinate information of the endoscope 40 in the three-dimensional medical image coordinate system and the related information of the direction (viewpoint position) are derived based on the related information of the shape of the endoscope 40, the related information of the bending history, the related information of the insertion distance, and the three-dimensional medical image. Therefore, the position and the rotation angle (rotation angles of the x-axis, the y-axis, and the z-axis) of the endoscope 40 in the three-dimensional medical image coordinate system at the time of capturing the endoscope image can be determined in correspondence with the shape of the endoscope 40, and the distance image information in the endoscope image can be derived with high precision and effectively based on the position and the rotation angle of the endoscope 40.

[0144] (Embodiment 3)

[0145] Figure 14 is a schematic view showing an outline of a diagnosis support system S related to Embodiment 3 (automatic operation mechanism 434). Figure 15 is a functional block diagram exemplifying functional sections included in the control section 62 of the information processing apparatus 6. The diagnosis support system S of Embodiment 3 differs from that of Embodiment 1 in that the endoscope 40 included in the diagnosis support system S further has an automatic operation mechanism 434.

[0146] The endoscope 40 included in the diagnosis support system S has the automatic operation mechanism 434, which automatically operates the control button 431 and the bending knob 433 based on the operation support information output from the control section 62 (operation support information output section 626) of the information processing apparatus 6.

[0147] The automatic operation mechanism 434 is communicably connected to the information processing apparatus 6, and acquires (receives) operation support information output (transmitted) from the information processing apparatus 6. The automatic operation mechanism 434 includes, for example, a microcomputer (not shown) that generates an on / off signal or a pulse signal of the control button 431 or the bending knob 433 in accordance with the acquired operation support information, and a motor and a cam mechanism (not shown) that operate or drive the control button 431 or the bending knob 433 based on these signals output from the microcomputer. Thus, based on the operation support information output from the information processing apparatus 6, the automatic operation mechanism 434, the control button 431, the bending knob 433, and the like cooperate with each other, and perform automatic operations such as automatic insertion of the endoscope 40 into the subject in accordance with the operation support information.

[0148] The operation support information acquired by the automatic operation mechanism 434 from the control section 62 (operation support information output section 626) of the information processing apparatus 6 is not limited to the operation support information for the insertion and bending of the insertion section 44. For example, an air injection section (not shown) or a hand section (not shown) can be provided to the insertion section 44 of the endoscope 40, and the operation support information can include information related to operations such as injection of air by the air injection section or removal (sampling) of a lesion site by the hand section. That is, the operation support information output section 626 generates information related to the air injection section or the hand section operation based on the shape information of the internal site determined based on the acquired three-dimensional medical image and the distance image information, and includes the information in the operation support information, and outputs the information to the automatic operation mechanism 434. The automatic operation mechanism 434 can automatically operate the air injection section or the hand section based on the acquired operation support information. In addition, the information related to the air injection section or the hand section operation included in the operation support information can be superimposed and displayed on the endoscope image or the like on the integrated image display screen 71.

[0149] According to the present embodiment, based on the distance image information in the endoscope image and the three-dimensional medical image or the virtual endoscope image, the operation support information on the operation of the endoscope 40 can be effectively output, and the automatic operation mechanism 434 can perform automatic operation of the endoscope 40 in accordance with the operation support information output from the endoscope processor 20. Thus, a diagnosis support system S that effectively supports the operator who operates the endoscope 40 such as a doctor can be provided.

[0150] It should be understood that the embodiments disclosed herein are illustrative in all respects, rather than restrictive. The technical features described in each of the embodiments can be combined with each other, and all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be encompassed in the scope of the present application.

[0151] Explanation of Reference Signs

[0152] S Diagnosis support system

[0153] 10 endoscope device

[0154] 15 keyboard

[0155] 16 housing rack

[0156] 20 processor for endoscope

[0157] 21 control unit

[0158] 211 image processing unit

[0159] 22 main storage device

[0160] 23 auxiliary storage device

[0161] 24 communication unit

[0162] 25 touch panel

[0163] 26 display device I / F

[0164] 27 input device I / F

[0165] 28 reading unit

[0166] 31 connector for endoscope

[0167] 311 electrical connector

[0168] 312 optical connector

[0169] 33 light source

[0170] 34 pump

[0171] 35 water supply tank

[0172] 36 air / water supply port

[0173] 40 endoscope

[0174] 43 operation unit

[0175] 431 control button

[0176] 433 bending knob

[0177] 434 automatic operation mechanism

[0178] 44 insertion section (flexible tube)

[0179] 441 flexible section

[0180] 442 bending section

[0181] 443 tip section

[0182] 444 imaging section

[0183] 446 light source for imaging

[0184] 45 bend stop

[0185] 48 connector for viewer

[0186] 49 general-purpose cord

[0187] 50 display device

[0188] 6 information processing apparatus

[0189] 61 communication section

[0190] 62 control section

[0191] 621 acquisition section

[0192] 622 viewpoint position derivation section

[0193] 623 virtual endoscope image generation section

[0194] 624 degree of coincidence determination section

[0195] 625 distance image information derivation section

[0196] 626 operation support information output section

[0197] 63 storage section

[0198] 631 endoscope image DB

[0199] 632 recording medium

[0200] P program

[0201] 64 input / output I / F

[0202] 7 display section

[0203] 71 integrated image display screen

[0204] 711 display option field

[0205] 712 display mode selection field

[0206] 8 input section

[0207] 91 operation information learning model

[0208] 92 degree of coincidence learning model

Claims

1. A computer program product comprising a program that causes a computer to execute the following processes: acquiring an endoscope image of a subject from an endoscope; acquiring a three-dimensional medical image of an inside of the subject taken by a unit capable of taking an image of the inside of the subject with a three-dimensional image; generating a virtual endoscope image reconstructed from the three-dimensional medical image based on the acquired endoscope image; deriving distance image information in the endoscope image based on the virtual endoscope image and the endoscope image; acquiring information on an insertion distance of the endoscope inserted into the subject, a bending history; deriving information on coordinates and a direction of the endoscope in a coordinate system of the three-dimensional medical image based on the information on the bending history, the information on the insertion distance, and the three-dimensional medical image; and outputting operation support information on an operation of the endoscope based on the information on the coordinates and the direction of the endoscope, the distance image information, and the three-dimensional medical image. The program causes the computer to execute the following process: outputting the operation support information including information on an insertion direction or an insertion speed of the endoscope, an insertion distance, based on the distance image information and the three-dimensional medical image. The program causes the computer to execute the following process: deriving the distance image information including information on a size of an internal site or a distance between internal sites in the endoscope image based on the virtual endoscope image and the endoscope image. The program causes the computer to execute the following processes: acquiring information on a shape of the endoscope, deriving information on coordinates and a direction of the endoscope in a coordinate system of the three-dimensional medical image based on the information on the shape of the endoscope, the information on the bending history, the information on the insertion distance, and the three-dimensional medical image, and outputting the operation support information on the operation of the endoscope based on the information on the coordinates and the direction of the endoscope, the virtual endoscope image, and the endoscope image.

5. The computer program product according to any one of claims 1, 2, and 4, wherein the generation of the virtual endoscope image includes the following processes: generating a two-dimensional medical image obtained by projecting the three-dimensional medical image, and determining, as the virtual endoscope image, a two-dimensional medical image in which a difference from the endoscope image is below a predetermined value in the generated two-dimensional medical image.

6. The computer program product according to any one of claims 1, 2, and 4, wherein the generation of the virtual endoscope image includes the following processes: generating a two-dimensional medical image obtained by projecting the three-dimensional medical image, inputting the two-dimensional medical image and the endoscope image to a learning model that has learned to output a degree of agreement of the two-dimensional medical image and the endoscope image when the two-dimensional medical image and the endoscope image are input, acquiring a degree of agreement of the two-dimensional medical image and the endoscope image from the learning model, and determining, as the virtual endoscope image, a two-dimensional medical image in which the degree of agreement of the two-dimensional medical image and the endoscope image is above a predetermined value. ​ ​ ​ 2. The computer program product of claim 1, wherein, ​ ​ 3. The computer program product of claim 1 or 2, wherein, ​ ​ 4. The computer program product of claim 1, wherein, ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 7. An information processing method of causing a computer to execute the following processing: acquiring an endoscope image of a subject from an endoscope; acquiring a three-dimensional medical image of an inside of a body taken by a unit capable of taking an image of the inside of the body of the subject with a three-dimensional image; generating a virtual endoscope image reconstructed from the three-dimensional medical image, based on the acquired endoscope image; deriving distance image information in the endoscope image, based on the virtual endoscope image and the endoscope image; acquiring information on an insertion distance and a bending history of the endoscope inserted into the body of the subject; deriving information on coordinates and a direction of the endoscope in a coordinate system of the three-dimensional medical image, based on the information on the bending history, the information on the insertion distance, and the three-dimensional medical image; and outputting operation support information on an operation of the endoscope, based on the information on the coordinates of the endoscope, the distance image information, and the three-dimensional medical image.

8. An information processing apparatus including: an endoscope image acquisition section that acquires an endoscope image of a subject from an endoscope; a three-dimensional medical image acquisition section that acquires a three-dimensional medical image of an inside of a body taken by a unit capable of taking an image of the inside of the body of the subject with a three-dimensional image; a generation section that generates a virtual endoscope image reconstructed from the three-dimensional medical image, based on the acquired endoscope image; a derivation section that derives distance image information in the endoscope image, based on the virtual endoscope image and the endoscope image; and an output section, the three-dimensional medical image acquisition section acquires information on an insertion distance and a bending history of the endoscope inserted into the body of the subject, the derivation section derives information on coordinates and a direction of the endoscope in a coordinate system of the three-dimensional medical image, based on the information on the bending history, the information on the insertion distance, and the three-dimensional medical image, the output section outputs operation support information on an operation of the endoscope, based on the information on the coordinates of the endoscope, the distance image information, and the three-dimensional medical image.

9. A diagnosis support system including: an endoscope; and an endoscope processor that acquires an endoscope image of a subject from the endoscope; the endoscope processor includes: an automatic operation mechanism that performs automatic operation of the endoscope; a three-dimensional medical image acquisition section that acquires a three-dimensional medical image of an inside of a body taken by a unit capable of taking an image of the inside of the body of the subject with a three-dimensional image; a generation section that generates a virtual endoscope image reconstructed from the three-dimensional medical image, based on the acquired endoscope image; a derivation section that derives distance image information in the endoscope image, based on the virtual endoscope image and the endoscope image; and an output section, the three-dimensional medical image acquisition section acquires information on an insertion distance and a bending history of the endoscope inserted into the body of the subject, the derivation section derives information on coordinates and a direction of the endoscope in a coordinate system of the three-dimensional medical image, based on the information on the bending history, the information on the insertion distance, and the three-dimensional medical image, ​ ​ ​ The output unit outputs operation support information on operation of the endoscope, based on the coordinate information of the endoscope, the distance image information, and the three-dimensional medical image, The automatic operation mechanism performs automatic operation of the endoscope in accordance with the operation support information output from the output unit.

Images