Program, Information Processing Method, Information Processing Apparatus, and Diagnostic Support System

Abstract

To provide a program etc., capable of identifying the position of an endoscope apparatus for the lung field in the lung of an observation object in accordance with a respiration phase by the lung.SOLUTION: The program causes a computer to acquire information on the respiration phase of a subject, acquire a three-dimensional medical image obtained by imaging the inside of the body of the subject in synchronization with the respiration phase, acquire an endoscopic image of the subject from an endoscope for the lung field, identify the position of the endoscope for the lung field in the three-dimensional medical image synchronizing with the respiration phase, and output the identified position of the endoscope for the lung field in association with the three-dimensional medical image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present technology relates to a program, an information processing method, an information processing apparatus, and a diagnostic support system.

Background Art

[0002] An endoscope is a medical device that enables observation and treatment of a desired location by being inserted into a body cavity of a subject, and includes an imaging unit incorporated in a distal end portion of an insertion tube inserted into the body cavity, and an illumination device that illuminates an imaging field of the imaging unit. Patent Document 1 discloses an endoscope apparatus for observing inside the bile duct, pancreatic duct, etc., which includes a parent scope having an insertion portion that can be inserted up to the duodenum, and a child scope inserted into a forceps channel of the parent scope to enable observation and treatment up to the bile duct and pancreatic duct.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the endoscope apparatus described in Document 1, when the observation target is the lung (bronchus), the position of the endoscope apparatus in the lung (endoscope apparatus for lung field) corresponding to the respiratory phase of the lung is not considered.

[0005] On one side, an object is to provide a program or the like that can specify the position of an endoscope apparatus for lung field in the lung corresponding to the respiratory phase of the lung to be observed.

Means for Solving the Problems

[0006] A program according to one aspect of the present disclosure causes a computer to execute a process of acquiring information regarding a respiratory phase of a subject, acquiring a three-dimensional medical image obtained by imaging the inside of the subject in synchronization with the respiratory phase, acquiring an endoscopic image of the subject from an endoscope for the lung field, specifying the position of the endoscope for the lung field in the three-dimensional medical image synchronized with the respiratory phase, and outputting the specified position of the endoscope for the lung field in association with the three-dimensional medical image.

[0007] An information processing method according to one aspect of the present disclosure causes a computer to execute a process of acquiring information regarding a respiratory phase of a subject, acquiring a three-dimensional medical image obtained by imaging the inside of the subject in synchronization with the respiratory phase, acquiring an endoscopic image of the subject from an endoscope for the lung field, specifying the position of the endoscope for the lung field in the three-dimensional medical image synchronized with the respiratory phase, and outputting the specified position of the endoscope for the lung field in association with the three-dimensional medical image.

[0008] An information processing apparatus according to one aspect of the present disclosure includes a phase information acquisition unit that acquires information regarding a respiratory phase of a subject, a three-dimensional medical image acquisition unit that acquires a three-dimensional medical image obtained by imaging the inside of the subject in synchronization with the respiratory phase, an endoscopic image acquisition unit that acquires an endoscopic image of the subject from an endoscope for the lung field, a position specification unit that specifies the position of the endoscope for the lung field in the three-dimensional medical image synchronized with the respiratory phase, and an output unit that outputs the specified position of the endoscope for the lung field in association with the three-dimensional medical image.

[0009] A diagnostic support system according to one aspect of the present disclosure includes an endoscope for the lung field, an automatic operation mechanism that automatically operates the endoscope for the lung field, and a control unit that controls the automatic operation mechanism. The control unit acquires information regarding a respiratory phase of a subject, acquires a three-dimensional medical image obtained by imaging the inside of the subject in synchronization with the respiratory phase, acquires an endoscopic image of the subject from the endoscope for the lung field, specifies the position of the endoscope for the lung field in the three-dimensional medical image synchronized with the respiratory phase, and outputs operation support information regarding the operation of the endoscope for the lung field to the automatic operation mechanism based on the three-dimensional medical image in which the position of the endoscope for the lung field is specified.

Advantages of the Invention

[0010] According to the present disclosure, it is possible to provide a program or the like for specifying the position of an endoscope device for the lung field in the lung corresponding to the respiratory phase of the lung of an observation target.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Modes for Carrying Out the Invention

[0012] (Embodiment 1) Hereinafter, the present invention will be described in detail based on the drawings showing its embodiments. FIG. 1 is a schematic diagram showing an overview of the diagnostic support system S according to Embodiment 1. The diagnostic support system S includes an endoscope apparatus 10 provided with a bronchoscope 40 for the lung field, and an information processing apparatus 6 communicably connected to the endoscope apparatus 10, and outputs support information regarding the diagnosis of the lungs (bronchi) of a subject.

[0013] The endoscope apparatus 10 transmits an image (captured image) captured by the imaging element 445 of the bronchoscope 40 for the lung field to the endoscope processor 20, and the endoscope processor 20 performs various image processes such as gamma correction, white balance correction, shading correction, and geometric correction for reducing image distortion, thereby generating an endoscope image in a state that is easy for the operator to visually observe. The endoscope apparatus 10 outputs (transmits) the generated endoscope image to the information processing apparatus 6. The information processing apparatus 6 that has acquired the endoscope image transmitted from the endoscope apparatus 10 performs various information processes based on these endoscope images and outputs information regarding diagnostic support.

[0014] The endoscope apparatus 10 includes an endoscope processor 20, a bronchoscope 40 for the lung field, and a display device 50. The display device 50 is, for example, a liquid crystal display device or an organic EL (Electro Luminescence) display device.

[0015] The display device 50 is installed at the upper stage of the storage shelf 16 with casters. The endoscope processor 20 is housed in the middle stage of the storage shelf 16. The storage shelf 16 is arranged in the vicinity of an endoscope examination bed (not shown). The storage shelf 16 has a pull-out type shelf equipped with a keyboard 15 connected to the endoscope processor 20.

[0016] The endoscope processor 20 has a substantially rectangular parallelepiped shape and is provided with a touch panel 25 on one surface. A reading unit 28 is disposed below the touch panel 25. The reading unit 28 is a connection interface for reading and writing a portable recording medium, such as a USB connector, an SD (Secure Digital) card slot, or a CD-ROM (Compact Disc Read Only Memory) drive.

[0017] The endoscope for lung field 40 has an insertion portion 44, an operation portion 43, a universal cord 49, and a scope connector 48. A control button 431 is provided on the operation portion 43. The insertion portion 44 is long, and one end thereof is connected to the operation portion 43 via a folding portion 45. The insertion portion 44 has a flexible portion 441, a bending portion 442, and a tip portion 443 in order from the operation portion 43 side. The bending portion 442 bends in response to the operation of a bending knob 433. A physical detection device such as a three-axis acceleration sensor, a gyro sensor, a geomagnetic sensor, a magnetic coil sensor, or an endoscope insertion shape observation device (CoronaNavi) is mounted on the insertion portion 44, and when the endoscope for lung field 40 is inserted into the subject's body, it may be configured to acquire detection results from these physical detection devices.

[0018] The universal cord 49 is long, and a first end thereof is connected to the operation portion 43 and a second end thereof is connected to the scope connector 48, respectively. The universal cord 49 is flexible. The scope connector 48 has a substantially rectangular parallelepiped shape.

[0019] Figure 2 is a block diagram showing a configuration example of the endoscope device 10 included in the diagnostic support system S. The control unit 21 is an arithmetic control device that executes the program of the present embodiment. One or more CPUs (Central Processing Unit), GPUs (Graphics Processing Unit), or multi-core CPUs are used for the control unit 21. The control unit 21 is connected to each hardware part constituting the endoscope processor 20 via a bus.

[0020] The main memory device 22 is a memory device such as, for example, SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory. In the main memory device 22, information necessary during the processing performed by the control unit 21 and the program being executed by the control unit 21 are temporarily stored. The auxiliary storage device 23 is a storage device such as, for example, SRAM, flash memory, or a hard disk, and is a storage device with a larger capacity than the main memory device 22. In the auxiliary storage device 23, for example, acquired captured images and generated endoscope images may be stored as intermediate data.

[0021] The communication unit 24 is a communication module or communication interface for communicating with the information processing device 6 via a network, either wired or wirelessly. For example, it is a short-range wireless communication module such as wifi (registered trademark), Bluetooth (registered trademark), or a wide-area wireless communication module such as 4G, LTE, etc. The touch panel 25 includes a display unit such as a liquid crystal display panel and an input unit laminated on the display unit. The communication unit 24 may communicate with an X-ray CT device 92, an MRI device, an ultrasonic diagnostic device, or a storage device (not shown) that stores data output from these devices.

[0022] The display device I / F 26 is an interface that connects the endoscope processor 20 and the display device 50. The input device I / F 27 is an interface that connects the endoscope processor 20 and an input device such as the keyboard 15.

[0023] The light source 33 is a high-brightness white light source such as, for example, a white LED or a xenon lamp. The light source 33 is connected to the bus via a driver (not shown). The lighting, extinguishing, and brightness change of the light source 33 are controlled by the control unit 21. The illumination light emitted from the light source 33 enters the optical connector 312. The optical connector 312 engages with the scope connector 48 and supplies illumination light to the endoscope for lung field 40.

[0024] The outline of the functions of the endoscope for the lung field 40 connected to the endoscope processor 20 will be described. Inside the scope connector 48, the universal cord 49, the operation unit 43, and the insertion unit 44, a fiber bundle, a cable bundle, an air supply tube, etc. are inserted. The illumination light emitted from the light source 33 is radiated from the illumination window provided at the distal end portion 443 via the optical connector 312 and the fiber bundle. The range illuminated by the illumination light is photographed by the image sensor provided at the distal end portion 443. The photographed image is transmitted from the image sensor to the endoscope processor 20 via the cable bundle and the electrical connector 311. The light source 33 may be an LED provided at the distal end portion 443. In this case, the optical connector 312 and the fiber bundle become unnecessary.

[0025] The control unit 21 of the endoscope processor 20 functions as an image processing unit 211 by executing the program stored in the main storage device 22. The image processing unit 211 performs various image processes such as gamma correction, white balance correction, shading correction, and geometric correction for reducing image distortion on the image (photographed image) output from the endoscope for the lung field 40, and outputs it as an endoscope image.

[0026] Figure 3 is a block diagram showing a configuration example of the information processing apparatus 6 included in the diagnostic support system S. The information processing apparatus 6 includes a control unit 62, a communication unit 61, a storage unit 63, and an input / output I / F 64. The information processing apparatus 6 is, for example, a server apparatus, a personal computer, or the like. The server apparatus includes not only a single server apparatus but also a cloud server apparatus or a virtual server apparatus configured by a plurality of computers. The information processing apparatus 6 may be provided as a cloud server located on an external network accessible from the endoscope processor 20.

[0027] The control unit 62 has an arithmetic processing unit with a timing function such as one or more CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), etc., and reads and executes the program P stored in the storage unit 63 to perform various information processes, control processes, etc. related to the information processing device 6. Alternatively, the control unit 62 may be composed of a quantum computer chip, and the information processing device 6 may be a quantum computer.

[0028] The storage unit 63 includes a volatile storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, and a non-volatile storage area such as EEPROM or a hard disk. The storage unit 63 stores in advance the program P (program product) and data referred to during processing. The program P (program product) stored in the storage unit 63 may store the program P (program product) read from the recordable medium 632 readable by the information processing device 6. Also, the program P may be downloaded from an external computer (not shown) connected to a communication network (not shown) and stored in the storage unit 63. The storage unit 63 stores an endoscope image DB631 (DataBase) described later. Further, the storage unit 63 may store an entity file (instance file of a neural network (NN)) constituting various learning models described later.

[0029] The communication unit 61 is a communication module or communication interface for communicating with the endoscope device 10 by wire or wirelessly, and is, for example, a short-range wireless communication module such as wifi (registered trademark), Bluetooth (registered trademark), or a wide-range wireless communication module such as 4G, 5G, etc. The communication unit 61 may communicate with an X-ray CT device 92, an MRI device (see FIG. 6), an ultrasonic diagnostic device, or a storage device (not shown) that stores data output from these devices.

[0030] The input / output I / F 64 conforms to a communication standard such as USB or DSUB, and is a communication interface for serial communication with an external device connected to the input / output I / F 64. A display unit 7 such as a display and an input unit 8 such as a keyboard are connected to the input / output I / F 64, and the control unit 62 outputs the results of information processing performed based on an execution command or an event input from the input unit 8 to the display unit 7. A respiratory synchronization device 91 (see FIG. 6) may be connected to the input / output I / F 64.

[0031] The respiratory synchronization device 91 is a device that is configured with a marker or the like attached to the abdomen or the like of the subject, and outputs a signal (respiratory synchronization signal) corresponding to the respiratory cycle (respiratory cycle). By receiving the respiratory synchronization signal from the respiratory synchronization device 91 via the input / output I / F 64, the information processing device 6 can acquire information about the subject's respiratory state, such as the phase in the respiratory cycle, the time point of the maximum inhalation state (100% state) or the maximum exhalation state (0% state), and the respiratory waveform.

[0032] FIG. 4 is an explanatory diagram illustrating the data layout of the endoscopic image DB631. The endoscopic image DB631 is stored in the storage unit 63 of the information processing apparatus 6 and is configured by database management software such as an RDBMS (Relational Database Management System) installed in the information processing apparatus 6. Alternatively, the endoscopic image DB631 may be stored in a predetermined storage area accessible from the information processing apparatus 6, such as a storage device communicably connected to the information processing apparatus 6. Alternatively, the endoscopic image DB631 may be stored in the main storage device 22 of the endoscope apparatus 10. That is, the predetermined storage area includes the storage unit 63 of the information processing apparatus 6, the main storage device 22 of the endoscope apparatus 10, and a storage device accessible from the information processing apparatus 6 or the endoscope apparatus 10. The information processing apparatus 6 may acquire the endoscopic image, the examination date and time, and the subject attribute information output by the endoscope processor 20 and register them in the examination result DB631. Alternatively, the endoscopic image, the examination date and time, and the subject attribute information directly output from the endoscope processor 20 may be directly registered in the examination result DB631.

[0033] The endoscopic image DB631 includes, for example, a subject master table and an image table, and the subject master table and the image table are associated with each other by a subject ID, which is an item (metadata) included in both tables.

[0034] The subject master table includes, as management items (metadata), for example, a subject ID, gender, date of birth, and age. In the item (field) of the subject ID, ID information is stored to uniquely identify the subject who has undergone the endoscopic examination. In the items (fields) of gender and date of birth, the biological attributes of gender and date of birth of the subject ID are stored, and in the item (field) of age, the current age calculated from the date of birth is stored. These, gender and age, are managed by the subject master table as the biological information of the subject.

[0035] The image table includes, as management items (metadata), for example, subject ID, examination date and time, endoscopic image, frame number, S coordinate (insertion distance), phase angle, and three-dimensional medical image.

[0036] The item (field) of the subject ID is for associating with the biological attributes of the subject managed in the subject master table, and the ID value of each subject is stored therein. The item (field) of the examination date and time stores the date and time when the subject with the subject ID underwent an endoscopic examination. The item (field) of the endoscopic image stores the endoscopic image of the subject with the subject ID as object data. The endoscopic image may be a still image in, for example, the jpeg format by one frame or a moving image in, for example, the avi format by several frames. The item (field) of the endoscopic image may store information indicating the storage location (file path) of the endoscopic image saved as a file.

[0037] The item (field) of the frame number stores the frame number of the moving image when the endoscopic image is a moving image. By storing the frame number of the moving image, even if the endoscopic image is a moving image, it can be handled in the same way as a still image and can be associated with the position information (coordinates in the body coordinate system) of the three-dimensional medical image described later.

[0038] The item (field) of the S coordinate (insertion distance) stores the insertion distance of the endoscope 40 for the lung field at the time of shooting the endoscopic image stored in the same record as the value of the S coordinate. The derivation of the insertion distance (S coordinate) will be described later.

[0039] In the item (field) of the phase angle, the phase angle (phase angle of the respiratory phase) at the time of shooting the endoscopic image stored in the same record is stored. Although details will be described later, the information processing apparatus 6 (control unit 62) acquires a respiration synchronization signal from the respiration synchronization apparatus 91 in parallel with the acquisition of the endoscopic image, and specifies the phase angle in the respiratory phase (respiratory cycle) based on the acquired respiration synchronization signal. The information processing apparatus 6 (control unit 62) stores the specified phase angle in the item (field) of the phase angle, whereby each endoscopic image and the phase angle are associated and stored.

[0040] In the item (field) of the three-dimensional medical image, a three-dimensional medical image in, for example, DICOM (Digital Imaging and Communications in Medicine) format, which is generated based on data output from means capable of photographing the inside of the body with a three-dimensional image, such as an X-ray CT apparatus 92 (X-ray CT, X-ray cone beam CT) or an MRI apparatus (MRI-CT), or an ultrasonic diagnostic apparatus, is stored as object data. Alternatively, information indicating the storage location (file path) of the three-dimensional medical image stored as a file may be stored.

[0041] The image table may further include items (fields) of the bending history, the viewpoint position, and the viewpoint direction. In this case, in the item (field) of the bending history, for example, the bending history information detected using an endoscopic insertion shape detection device connected to the endoscopic processor 20 is stored. In the item (field) of the viewpoint position, the coordinates in the body of the endoscope for lung field 40 at the time when the endoscopic image is taken, that is, the coordinates in the coordinate system (in-body coordinate system) of the three-dimensional medical image are stored. In the item (field) of the viewpoint direction, the orientation of the endoscope for lung field 40 at the time when the endoscopic image is taken, that is, the rotation angle in the coordinate system (coordinates in the in-body coordinate system) of the three-dimensional medical image is stored.

[0042] FIG. 5 is an explanatory diagram showing the insertion distance (S coordinate value) of the pulmonary endoscope 40. As shown in the figure, the bronchi and other parts of the lungs photographed by the pulmonary endoscope 40 are expressed in three-dimensional shapes in a three-dimensional medical image. A space is formed inside the inner wall of the bronchi and other parts, and this space serves as the insertion path along which the pulmonary endoscope 40 is inserted. The S coordinate, which represents the insertion distance of the pulmonary endoscope 40, is a location inside the insertion path (inside the inner wall of the bronchi and other parts) where the path length of the insertion path is approximately equal to the insertion distance. Therefore, based on the S coordinate, the coordinates of the tip 443 of the pulmonary endoscope 40 located inside the inner wall of the bronchi and other parts can be derived. The insertion distance is measured, for example, by providing a measuring unit composed of a roller encoder or the like in a mouthpiece placed in the mouth of the subject, and measuring the distance the pulmonary endoscope 40 has been inserted into the body. The location of the measuring unit is not limited to the mouthpiece; a measuring unit using a non-contact sensor may be placed in the throat of the subject. The S coordinate may be further corrected using bending history information.

[0043] 6 is a functional block diagram illustrating functional units included in the control unit 62 of the information processing device 6. The control unit 21 of the endoscopic processor 20 (endoscopic device 10) functions as an image processing unit 211 by executing a program stored in the main storage device 22. The control unit 62 of the information processing device 6 functions as an acquisition unit 621, a respiratory phase identification unit 622, a position derivation unit 623, and an operation support information output unit 624 by executing a program P stored in the storage unit 63.

[0044] The image processing unit 211 of the endoscope processor 20 performs various image processes such as gamma correction, white balance correction, and shading correction on the image (captured image) output from the lung field endoscope 40, and outputs it as an endoscope image. The image processing unit 211 outputs (transmits) the generated endoscope image and the inspection date and time based on the imaging time of the endoscope image to the information processing device 6. The image processing unit 211 may further output the subject ID input from the keyboard 15 to the information processing device 6. The image processing unit 211 outputs information regarding the insertion distance (S coordinate) of the lung field endoscope 40 from a sensor disposed in the insertion unit 44 (flexible tube) of the lung field endoscope 40 to measure the surrounding environment of the lung field endoscope 40 to the information processing device 6. The image processing unit 211 may, for example, superimpose the information regarding the insertion distance of the lung field endoscope 40 acquired from the sensor on the endoscope image and display it on the display device.

[0045] Sensors for acquiring the S coordinate, which is the distance at which the lung field endoscope 40 is inserted into the body, include, for example, a temperature sensor, a light sensor, a pressure sensor, a wetness sensor (electrode), and a humidity sensor. For example, when the sensor is a light sensor, the light sensor is disposed inside the insertion unit 44 (flexible tube), but can receive light even when the insertion unit 44 (flexible tube) is inserted into the body. Therefore, it is possible to distinguish that the portion where the light sensor receives more light is outside the body and the portion where it receives less light is inside the body. Then, the control unit 21 of the endoscope processor 20 can derive the S coordinate, which is the distance (length) of the insertion unit 44 (flexible tube) inserted into the body, by specifying the light sensor at the boundary position, which is the body cavity insertion site, based on the signal obtained by the light sensor.

[0046] The control unit 21 of the endoscope processor 20 acquires the bending history information of the endoscope for the lung field 40 inserted into the body, and determines the insertion status of the endoscope for the lung field 40 according to the acquired bending history information. The control unit 21 of the endoscope processor 20 may detect the bending history information, for example, by using an endoscope insertion shape detection device (not shown) connected to the endoscope processor 20. The endoscope insertion shape detection device may be a device in which a plurality of magnetic coils are arranged at predetermined intervals along the longitudinal direction of the insertion portion 44 inside the insertion portion 44 of the endoscope for the lung field 40, as disclosed in, for example, Japanese Patent Application Laid-Open No. 2019-37643. The bending history information indicates physical parameters or information related to bending, such as the bending angle and the bending direction.

[0047] The acquisition unit 621 acquires the subject ID, the examination date and time, the endoscope image, the S coordinate (insertion distance), and the bending history information output by the endoscope processor 20. Based on the acquired subject ID, the acquisition unit 621 acquires the three-dimensional medical image of the subject output from means capable of photographing the inside of the body with a three-dimensional image, such as an X-ray CT device 92, a cone beam CT device, an MRI device, or an ultrasonic diagnostic device, which is communicably connected. When the three-dimensional medical image output from another inspection device capable of photographing the inside of the body with a three-dimensional image, such as the X-ray CT device 92, the cone beam CT device, the MRI device, or the ultrasonic diagnostic device, is already stored in, for example, an external server (not shown), the information processing device 6 may access the external server and acquire the three-dimensional medical image of the subject based on the subject ID output from the endoscope processor 20.

[0048] The three-dimensional medical image is an image represented by volume data composed of tomographic image data output from a device capable of capturing three-dimensional images of the inside of the body, such as an X-ray CT device 92, a cone-beam CT device, an MRI device, or an ultrasound diagnostic device, as well as an image represented by volume data output from a multi-slice X-ray CT device or an X-ray cone-beam CT device using an X-ray flat panel. When an X-ray CT device 92 or a cone-beam CT device is used, for example, dual energy imaging may be performed using an X-ray CT to obtain an image in which the composition (body composition) of each pixel in the three-dimensional medical image can be determined by the effective mass number (effective-Z). When an MRI device is used, an image may be obtained in which information about the composition (body composition) of each pixel in the three-dimensional medical image, such as fat or lactic acid, is added.

[0049] The acquired 3D medical image is an image reconstructed according to the respiratory phase, and is an image generated based on a group of CT data captured at a predetermined phase angle (2π) corresponding to normal end-tidal breathing (exhalation: 0%). Therefore, the 3D medical image corresponding to the respiratory phase is an image showing the state of the lungs at normal end-tidal breathing (phase angle: 2π), for example, and therefore the influence of respiratory movement of the lungs being examined can be reduced.

[0050] The 3D medical image is an image reconstructed (generated) by an examination using the X-ray CT scanner 92 performed prior to the examination using the lung endoscope 40. If a region of interest (RI) such as a tumor is present in the lung being examined, the position of the RI is identified in the internal body coordinate system using the 3D medical image. When identifying the presence or absence of a region of interest such as a tumor based on the 3D medical image, a region of interest model (learning model) may be used that outputs the position and type of the region of interest contained in the 3D medical image when the 3D medical image is input. The region of interest model may be configured, for example, using a neural network with a segmentation function such as R-CNN or YOLO.

[0051] The acquisition unit 621 further acquires a signal (respiratory synchronization signal) corresponding to the respiratory cycle (respiratory cycle) from the respiratory synchronization device 91. The output of the respiratory synchronization signal by the respiratory synchronization device 91, i.e., the detection of the respiratory phase, and the acquisition (imaging) of the endoscopic image are performed simultaneously in parallel. The acquisition unit 621 associates the acquired respiratory synchronization signal with the endoscopic image and outputs them to the respiratory phase identification unit 622.

[0052] The respiratory phase identification unit 622 identifies the phase angle of the respiratory phase at the time the endoscopic image was captured (the time of image capture) in accordance with the respiratory synchronization signal and the endoscopic image output from the acquisition unit 621. The respiratory phase identification unit 622 may generate a periodic respiratory waveform based on the acquired respiratory synchronization signal. The respiratory phase identification unit 622 identifies the phase angle of the respiratory phase (the phase angle at the time the endoscopic image was captured) based on the acquired respiratory synchronization signal and determines whether the phase angle of the endoscopic image currently captured (acquired) matches the phase angle of the 3D medical image. The respiratory phase identification unit 622 may associate the phase angle identified based on the respiratory synchronization signal with the endoscopic image captured at the time the respiratory synchronization signal was acquired and store the associated data in the endoscopic image DB 631 stored in the storage unit 63. By storing each captured endoscopic image (each frame in the case of a video image) in this manner with the phase angle of the respiratory phase at the time the endoscopic image was captured, various operations or processes synchronized with the respiratory phase (respiratory cycle) can be performed on the endoscopic image.

[0053] The respiratory synchronization signal acquired by the acquisition unit 621 corresponds to the endoscopic image on the time axis. Thus, the phase angle of the respiratory phase at the time point when the endoscopic image was acquired (imaging time point) is specified by the respiratory synchronization signal acquired (detected) at that time point (imaging time point). Therefore, the respiratory synchronization signal corresponds to a synchronization signal that synchronizes each of the continuously acquired endoscopic images with the respiratory phase (respiratory cycle). Needless to say, when the endoscopic image is a moving image, the acquisition time point of the endoscopic image is the acquisition time point of the frame (still image) constituting the moving image. In this way, by acquiring the respiratory synchronization signal from the respiratory synchronization device 91 in parallel with (simultaneously with) the examination using the bronchoscope 40 for the lung field (imaging of the endoscopic image), the phase angle of the respiratory phase at the imaging time point of each of the endoscopic images acquired in time series, that is, continuously over time, can be associated. The three-dimensional medical image is reconstructed (generated) based on a predetermined phase angle (for example, the phase angle corresponding to normal end-expiration: 2π). Therefore, by using the phase angle of the three-dimensional medical image and the endoscopic image associated with the same phase angle, the influence of body movement due to breathing can be reduced, and the position of the bronchoscope 40 for the lung field (tip portion 443) in the three-dimensional medical image (body coordinate system) can be accurately specified.

[0054] The position derivation unit 623 derives the position of the tip portion 443 of the bronchoscope 40 for the lung field in the three-dimensional medical image (body coordinate system) that becomes a predetermined phase angle at the respiratory phase, based on the insertion distance (S coordinate) of the bronchoscope 40 for the lung field output from the acquisition unit 621, the bending history, and the like. When deriving the position of the tip portion 443 of the bronchoscope 40 for the lung field, the position derivation unit 623 may use the insertion distance (S coordinate) and the like at the imaging time point of the endoscopic image that has the same phase angle as the phase angle of the three-dimensional medical image to derive the position of the bronchoscope 40 for the lung field (tip portion 443). The S coordinate may be further corrected using the bending history information.

[0055] That the phase angle of the three-dimensional medical image and the phase angle at the time of capturing the endoscopic image are the same is not limited to the case where these phase angles exactly match. For example, when the phase angle at the time of capturing the endoscopic image falls within a predetermined angular range such as ±5° with respect to the phase angle of the three-dimensional medical image, etc., it is intended to be included in the allowable angular range for the operation of the endoscope 40 for the lung field by a doctor or the like.

[0056] The operation support information output unit 624 outputs, as operation support information, information regarding the position of the distal end portion 443 of the endoscope 40 for the lung field and the path from the position of the distal end portion 443 to the position of the region of interest such as a lesion. The operation support information output unit 624 may output and display, in an associated manner, such information regarding the position of the endoscope 40 (distal end portion 443) for the lung field and the path to the region of interest by superimposing or attaching it to the three-dimensional medical image and the endoscopic image.

[0057] When the operation support information output unit 624 outputs a three-dimensional medical image with the endoscope 40 for the lung field (distal end portion 443) superimposed, if the phase angles of the endoscopic image and the three-dimensional medical image match, the position of the endoscope 40 for the lung field (distal end portion 443) superimposed and displayed on the three-dimensional medical image is not affected by body movement due to breathing and has high suitability. If the phase angles of the endoscopic image and the three-dimensional medical image do not match, the position of the endoscope 40 for the lung field (distal end portion 443) superimposed and displayed on the three-dimensional medical image is considered to be affected by body movement due to breathing and is likely to be different from the actual position. The operation support information output unit 624 may make the display form of the endoscope 40 for the lung field (distal end portion 443) superimposed and displayed on the endoscopic image or the three-dimensional medical image different depending on whether the phase angles of the endoscopic image and the three-dimensional medical image match or not. For example, when the phase angles of the endoscopic image and the three-dimensional medical image match, the endoscope 40 for the lung field (distal end portion 443) may be displayed in blue, and when they do not match, the endoscope 40 for the lung field (distal end portion 443) may be displayed in yellow. Each functional unit such as the operation support information output unit 624 may store various data acquired or output in the endoscopic image DB631.

[0058] In the present embodiment and the like, the processing of each functional unit has been described by dividing it into the processing by the control unit 21 of the endoscope processor 20 and the processing by the control unit 62 of the information processing apparatus 6. However, the sharing of these processes is merely an example and is not limited thereto. The control unit 21 of the endoscope processor 20 may function as all the functional units performed by the control unit 62 of the information processing apparatus 6. Alternatively, the control unit 21 of the endoscope processor 20 and the control unit 62 of the information processing apparatus 6 may function as each functional unit in a series of processes in cooperation with each other, for example, by performing inter-process communication.

[0059] FIG. 7 is a flowchart showing an example of a processing procedure by the control unit 62 of the information processing apparatus 6. The control unit 62 of the information processing apparatus 6 starts the processing of the flowchart based on, for example, the input content from the input unit 8 connected to the own apparatus.

[0060] The control unit 62 of the information processing apparatus 6 acquires information such as the inspection date and time, the subject ID, the endoscope image, and the insertion distance output from the endoscope processor 20 (S101). The control unit 62 of the information processing apparatus 6 may further acquire the bending history information of the endoscope for the lung field 40 inserted into the body from the endoscope processor 20 in addition to the insertion distance (S coordinate).

[0061] The control unit 62 of the information processing apparatus 6 acquires a three-dimensional medical image reconstructed based on a predetermined phase angle from the X-ray CT apparatus 92 or the like (S102). The control unit 62 of the information processing apparatus 6 acquires a three-dimensional medical image reconstructed based on a predetermined phase angle, that is, a three-dimensional medical image synchronized with the respiratory phase, from the X-ray CT apparatus 92 or the like. The three-dimensional medical image is an image reconstructed based on the CT image (tomographic image data) of the inspection result of the X-ray CT apparatus 92 or the like performed before the inspection (imaging of the endoscope image) of the lung (bronchus) by the endoscope for the lung field 40 for the same subject. The control unit 62 of the information processing apparatus 6 uses the acquired subject ID to access the X-ray CT apparatus 92 or a data management server of the three-dimensional medical image or the like, and acquires the three-dimensional medical image that is the inspection result of the subject of the endoscope for the lung field 40 by the X-ray CT apparatus 92 or the like received previously.

[0062] In an examination of the lungs using the X-ray CT device 92, respiratory motion in the lungs causes changes in the body motion contained in the captured CT images (tomographic image data). However, the 3D medical images (X-ray CT 3D images) in this embodiment are 3D medical images (3D medical images synchronized with the respiratory phase) reconstructed based on a predetermined phase angle. That is, in parallel with the examination (taking of CT images) of the lungs using the X-ray CT device 92, the respiratory waveform (respiratory phase) is measured or calculated based on a respiratory synchronization signal acquired from the respiratory synchronization device 91, and CT images (tomographic image data) at the predetermined respiratory phase are aggregated and rearranged to reconstruct a 3D medical image at the predetermined respiratory phase. Using 3D medical images reconstructed based on a predetermined phase angle (e.g., a phase angle corresponding to normal end-tidal breathing: 2π) in this manner reduces the effects of respiratory motion.

[0063] The three-dimensional medical images are not limited to those reconstructed based on a single respiratory phase (phase angle), but may be reconstructed based on multiple respiratory phases (phase angles). For example, for CT images (tomographic image data) captured by the X-ray CT device 92, two sets (two phases) of three-dimensional medical images may be reconstructed for each of two respiratory phases, namely, the maximum inhalation state (100% state: π) and the maximum exhalation state (0% state: 2π), in one respiratory cycle. The number of respiratory phases may be two or more. For example, the processing unit angle may be set to π / 6, and 12 sets (12 phases) of three-dimensional medical images may be reconstructed. The three-dimensional medical images reconstructed in this manner based on multiple phases may be segmented into multiple phase units and may be organized as an archive file of three-dimensional medical images associated with each phase angle. The position of the pulmonary endoscope may be identified for each segmented three-dimensional medical image.

[0064] As described above, the three-dimensional medical image is captured before the examination using the lung endoscope 40, and includes the entire lungs of the subject. Therefore, if there is a region of interest (ROI) in the lung, such as a lesion or a lesion candidate, such as a tumor, the region of interest will be included in the three-dimensional medical image. The position of the region of interest included in the three-dimensional medical image (coordinates in the internal body coordinate system) is identified (assigned to the three-dimensional medical image) by, for example, a diagnosis by a doctor or the like. The region of interest may be identified using a learning model (region of interest model) that outputs the region and type of the region of interest when a three-dimensional medical image is input.

[0065] The control unit 62 of the information processing device 6 acquires a signal (respiratory synchronization signal) corresponding to the respiratory cycle (respiratory cycle) from the respiratory synchronization device 91 (S103). The control unit 62 of the information processing device 6 determines a phase angle in the respiratory phase based on the respiratory synchronization signal (S104). The respiratory synchronization device 91 is configured with a marker or the like attached to the abdomen or the like of the subject of the pulmonary endoscope 40, and outputs a signal (respiratory synchronization signal) corresponding to the respiratory cycle (respiratory cycle). The control unit 62 of the information processing device 6 acquires the respiratory synchronization signal from the respiratory synchronization device 91 in parallel with acquiring the endoscopic image, and determines the phase angle in the respiratory phase based on the acquired respiratory synchronization signal. Alternatively, a device that measures body movement from a video image (moving image) of the subject and determines the respiratory phase may be used.

[0066] Since the capturing (acquisition) of endoscopic images and the determination of the phase angle using the respiratory synchronization signal are continuously performed in parallel, the control unit 62 of the information processing device 6 can continuously determine the phase angle at the time of capturing the endoscopic images. The control unit 62 of the information processing device 6 may store the determined phase angle in the storage unit 63 (store it in the endoscopic image DB 631) in association with the endoscopic images. If the endoscopic images are moving images, the control unit 62 may store the phase angle at the time when each frame (still image) constituting the moving image was captured in association with the frame (still image). In this way, by associating each endoscopic image with the phase angle at the time when the endoscopic image was captured, the endoscopic image can be synchronized with the respiratory phase. Furthermore, since the insertion distance (S coordinate) and bending history at the time when the endoscopic image was captured are associated with the endoscopic image, the respiratory phase is also associated with these insertion distance (S coordinate) and bending history.

[0067] The control unit 62 of the information processing device 6 derives the position of the pulmonary endoscope 40 in the three-dimensional medical image (S105). The three-dimensional medical image corresponds to a predetermined phase angle, and the control unit 62 of the information processing device 6 extracts an endoscopic image having the same phase angle as the phase angle of the three-dimensional medical image. This allows the acquired endoscopic image and the three-dimensional medical image to be synchronized (associated by the same phase angle) based on the phase angle. The phase angle of the three-dimensional medical image and the phase angle at the time of capturing the endoscopic image being the same does not necessarily mean that these phase angles are completely the same, but may also mean that the phase angle at the time of capturing the endoscopic image falls within a predetermined angle range, such as ±5°, of the phase angle of the three-dimensional medical image. Because the extracted endoscopic image and the three-dimensional medical image have the same phase angle, the influence of body movement due to breathing between these images is small, thereby improving the accuracy of deriving the position of the pulmonary endoscope 40 (tip portion 443) in the three-dimensional medical image.

[0068] The control unit 62 of the information processing apparatus 6 specifies the position of the tip 443 of the endoscope for lung field in the 3D medical image (body coordinate system) based on the insertion distance (S coordinate) and the bending history at the time of imaging of the extracted endoscope image (endoscope image with the same phase angle as the 3D medical image). By using the insertion distance (S coordinate) and the bending history of the endoscope for lung field 40, the path length and the insertion direction of the insertion path (such as the bronchus of the lung) into which the endoscope for lung field 40 is inserted can be specified, and thereby the coordinates of the tip 443 of the endoscope for lung field located inside the inner wall of the bronchus or the like can be derived. When the control unit 62 of the information processing apparatus 6 derives the position of the tip 443 of the endoscope for lung field 40 in the 3D medical image (body coordinate system) based on the insertion distance (S coordinate) etc. of the endoscope for lung field 40, for example, a virtual endoscope image may be used as disclosed in the publication of International Publication No. WO2021 / 054360.

[0069] The control unit 62 of the information processing apparatus 6 outputs operation support information based on the derived position of the endoscope for lung field 40 and the position of the region of interest included in the 3D medical image (S106). The control unit 62 of the information processing apparatus 6 specifies the path from the position of the tip 443 of the endoscope for lung field 40 at the current time to the position of the region of interest based on the position of the tip 443 of the endoscope for lung field 40 (coordinates in the body coordinate system) and the position of the region of interest (coordinates in the body coordinate system) in the 3D medical image (body coordinate system).

[0070] The bronchus of the lung shown in the 3D medical image is defined as a three-dimensional solid object in the body coordinate system defined in three dimensions, and the control unit 62 of the information processing apparatus 6 specifies the path by calculating a direction vector or the like from the position of the tip 443 of the endoscope for lung field 40 to the position of the region of interest by using the spatial information of the path inside the bronchus. The direction vector includes the insertion direction, the insertion amount, and the insertion speed of the endoscope for lung field 40, and includes information regarding the path to the region of interest such as a lesion.

[0071] When outputting operation support information, the control unit 62 of the information processing apparatus 6 may generate image data in which the operation support information is superimposed on an endoscopic image or a three-dimensional medical image, and output the image data to, for example, the display unit 7. The display unit 7 displays an endoscopic image or a three-dimensional medical image on which operation support information is superimposed based on the image data output from the control unit 62 of the information processing apparatus 6.

[0072] When the three-dimensional medical image is reconstructed (segmented in multiple phase units) based on a plurality of respiratory phases, the control unit 62 of the information processing apparatus 6 may perform a series of processes by switching the three-dimensional medical image according to the respiratory phase specified based on the respiratory synchronization signal acquired from the respiratory synchronization device 91. For example, when the three-dimensional medical image is reconstructed based on 12-phase respiratory phases, the control unit 62 of the information processing apparatus 6 may perform processes based on the three-dimensional medical images corresponding to the individual respiratory phases by performing 12 switches, which is the number of phases of the respiratory phase, in one respiratory cycle. By increasing the number of phases of the respiratory phase in reconstructing the three-dimensional medical image in this way, the synchronization timing (synchronization point) with the endoscopic image can be increased, and the operation support information provided to doctors and the like can be expanded.

[0073] FIG. 8 is an explanatory diagram showing one aspect of the integrated image display screen. As described above, the control unit 62 of the information processing apparatus 6 generates image data including an endoscopic image or a three-dimensional medical image on which operation support information such as the insertion direction of the endoscope 40 for the lung field is superimposed, and outputs the image data to the display unit 7. The integrated image display screen 71 is an example of a display screen constituted by the image data, and the display unit 7 displays the integrated image display screen 71 based on the image data.

[0074] The integrated image display screen 71 includes, for example, a region for displaying bibliographic items such as the subject ID, a region for displaying an endoscopic image, a region for displaying a three-dimensional medical image, a region for displaying a respiratory waveform, and a region for displaying information regarding the currently displayed endoscopic image and the viewpoint position (position of the distal end portion 443) where the endoscopic image is captured.

[0075] The area displaying bibliographic information such as the subject ID displays bibliographic information for data management, such as the subject ID used to identify the 3D medical image corresponding to the endoscopic image, the examination date and time of the pulmonary endoscope 40, and the date the 3D medical image was generated.

[0076] In the area displaying the endoscopic image, the endoscopic image currently being captured by the pulmonary endoscope 40 is displayed in real time. Operational support information such as the insertion direction of the pulmonary endoscope 40 is superimposed on the endoscopic image. Depending on the setting of the display options described below, the endoscopic image may be displayed semi-transparently, and internal body parts located at the back of the internal body wall displayed in the endoscopic image may be displayed by dotted lines or the like. The internal body parts displayed by dotted lines or the like in this semi-transparent display may be, for example, lesion candidate parts extracted based on shape information of the internal body parts identified in the three-dimensional medical image.

[0077] In the area displaying the 3D medical image, internal body parts such as the lungs (bronchi) represented in the 3D medical image are displayed as 3D objects, with operation support information such as the position (viewpoint) of the tip 443 of the pulmonary endoscope 40 and the insertion direction starting from the position (viewpoint) of the tip 443 superimposed and displayed. The 3D object can be rotated by dragging any part of the 3D object. The 3D medical image may be displayed with the position of a lesion candidate extracted based on shape information of the internal body part identified by the 3D medical image highlighted, for example.

[0078] The endoscopic image is displayed in real time according to the imaging time point and changes over time. When the phase angle at the imaging time point of the endoscopic image matches the phase angle used when reconstructing the three-dimensional medical image, these endoscopic images and the three-dimensional medical image have the same phase angle and are synchronized. The control unit 62 of the information processing apparatus 6 may change the display modes, such as the tip 443 of the endoscope 40 for the lung field and the arrow indicating the insertion direction, which are superimposed on the endoscopic image and the three-dimensional medical image, depending on whether these phase angles are the same (synchronized) or different (asynchronous). For example, when these phase angles are the same (synchronized), the tip 443 of the endoscope 40 for the lung field may be superimposed and displayed in blue, and when they are different (asynchronous), it may be superimposed and displayed in yellow. Thereby, it is possible to efficiently notify the operator of the endoscope 40 for the lung field, such as a doctor, whether the phase angle of the endoscopic image being imaged at the current time matches the phase angle of the three-dimensional medical image (whether it is in a synchronized state).

[0079] When the three-dimensional medical image is reconstructed based on a plurality of respiratory phases, the control unit 62 of the information processing apparatus 6 may switch and display the three-dimensional medical image according to the respiratory phase specified based on the respiratory synchronization signal acquired from the respiratory synchronization device 91. Thereby, in one respiratory cycle, the three-dimensional medical image is switched and displayed according to the number of phases of the respiratory phase, and the three-dimensional medical image can be switched and displayed in response to the respiratory cycle of the subject. By switching and displaying the three-dimensional medical image according to the respiratory phase, the synchronization timing (synchronization point) between the three-dimensional medical image and the endoscopic image can be increased, and the operation support information provided to doctors and the like can be expanded (the output frequency can be increased).

[0080] The respiratory waveform display area displays a graph showing a respiratory waveform generated based on a respiratory synchronization signal acquired from a respiratory synchronization device 91. The respiratory synchronization device 91 may be, for example, the RGSC respiratory synchronization system manufactured by Varian Medical Systems, Inc. or the AZ-733VI respiratory synchronization system manufactured by Anzai Medical Co., Ltd. The horizontal axis of the respiratory waveform graph represents time, and the vertical axis represents lung volume (amount of inhaled air). In the respiratory waveform, the first half of the respiratory cycle represents the inhalation period, and the second half represents the exhalation period. The phase angle in the respiratory cycle may be defined as 0 (start of inhalation) to 2π (end of exhalation). Three-dimensional medical images are reconstructed at a predetermined phase angle, and the respiratory waveform graph displays an object such as a line diagram indicating the phase angle of the three-dimensional medical image. This allows doctors and other personnel to efficiently receive information about the relationship between the phase angle used to reconstruct the three-dimensional medical image and the respiratory cycle.

[0081] In the respiratory waveform graph, an object such as an arrow indicating the phase angle of the endoscopic image currently being displayed may be displayed. Since the phase angle at the time of capturing the endoscopic image is associated with the acquired endoscopic image, by displaying the phase angle of the displayed endoscopic image in the respiratory waveform graph, information regarding the relationship between the phase angle of the endoscopic image and the respiratory cycle can be efficiently provided to a physician or other such person. If the endoscopic image is a video, the arrow indicating the phase angle of the endoscopic image will also slide over time (moving to the right along the horizontal axis, which is the time axis) as the video progresses (time passes).

[0082] In the area for displaying the viewpoint position and the like where the endoscopic image was captured, the position (viewpoint position) and the viewpoint direction (rotation angle) of the endoscope for lung field 40 inside the body at the time of capturing the endoscopic image displayed in the area for displaying the endoscopic image are displayed. As described above, the control unit 62 (acquisition unit 621) of the information processing apparatus 6 continuously acquires the endoscopic image, the S coordinate indicating the insertion distance of the endoscope for lung field 40, and the bending history from the endoscope processor 20, and continuously derives the position (viewpoint position) of the distal end portion 443 of the endoscope for lung field 40 based on the acquired S coordinate and the like. Therefore, in the area for displaying the viewpoint position and the like where the endoscopic image was captured, following the operation of the endoscope for lung field 40 by a doctor or the like, the position viewpoint position and the viewpoint direction (coordinates and rotation angle in the body coordinate system) of the distal end portion 443 of the endoscope for lung field 40 are displayed in real time.

[0083] In the area for displaying the information of the currently displayed endoscopic image, for example, information regarding the internal body position (bronchus) or pixel at the image center of the currently displayed endoscopic image is displayed. As described above, in the three-dimensional medical image, the information regarding the effective mass number (effective-Z), which is the substance discrimination information based on X-rays, or the composition (body composition) of each pixel of the three-dimensional medical image such as fat or lactic acid, is included in the information regarding the internal body position (pixel). Therefore, based on the coordinates of the internal body coordinate system indicating the image center of the endoscopic image, the effective mass number (effective-Z) and the information regarding the body composition extracted from the three-dimensional medical image can be displayed in the area. Also, regarding the presence or absence of a lesion at the internal body position included in the currently displayed endoscopic image, by using a region of interest model (learning model) that takes the endoscopic image as input and outputs information regarding the presence or absence of a lesion, it can be displayed in the area. The region of interest model (learning model) that outputs information regarding the presence or absence of a region of interest such as a lesion based on the thus input endoscopic image may use any object detection algorithm having the function of a segmentation network, such as CNN, RCNN (Regions with Convolutional Neural Network), Fast RCNN, Faster-RCNN, SSD (Single Shot Multibox Detector), YOLO (You Only Look Once), etc. Regarding the phase angle of the currently displayed endoscopic image, it is displayed in a graph in the area for displaying the respiratory waveform, but in this area as well, it may be configured to display whether or not the phase angle of the currently displayed endoscopic image matches the phase angle of the three-dimensional medical image.

[0084] The integrated image display screen may have a display option field for setting display options. The display option field has a toggle switch for setting whether to display the endoscopic image semi-transparently. By checking the toggle switch, the internal body wall surface (the inner wall surface of the bronchi) displayed in the endoscopic image is made semi-transparent based on shape data of the internal body part contained in the 3D medical image, and the internal body part located behind the internal body wall surface is displayed using dotted lines, etc. As described above, the internal body part located behind the internal body wall surface may be, for example, a region of interest such as a lesion extracted based on shape information of the internal body part identified in the 3D medical image.

[0085] According to this embodiment, the control unit 62 of the information processing device 6 identifies the position of the pulmonary endoscope 40 in a 3D medical image synchronized with the respiratory phase, thereby enabling the position of the pulmonary endoscope 40 to be identified in the 3D medical image in accordance with the respiratory phase of the lungs being observed. When the lungs are the subject of 3D medical imaging using an X-ray CT scanner 92 or the like, the subject (lungs) undergoes body movement due to respiration. By using 3D medical images synchronized with the respiratory phase to achieve a predetermined phase angle, the influence of body movement due to respiration can be reduced and the position of the pulmonary endoscope 40 in the 3D medical image can be identified, improving the accuracy of the position identification. The position of the pulmonary endoscope 40 is identified using the insertion distance (S coordinate) and bending history of the pulmonary endoscope inserted into the subject's body at the time the endoscopic image was captured. This allows the position on the 3D medical image corresponding to the insertion distance (S coordinate), etc. to be efficiently identified.

[0086] According to the present embodiment, the endoscopic image is acquired as a moving image with a frame rate of, for example, 60 frames, and in these multiple frames, the first frame in which the phase angle in the respiratory phase is, for example, 270° (phase is 0%: exhalation) is specified. According to the synchronization signal (corresponding to a predetermined phase angle) output from the respiration synchronization device 91 that synchronizes the three-dimensional medical image and the endoscopic image, the endoscopic image having the same phase angle as the three-dimensional medical image is specified. Alternatively, based on the relationship between the imaging time point in the specified first frame and the respiratory cycle, a process of assigning a phase angle (phase) to subsequent frames may be performed. The three-dimensional medical image acquired in synchronization with the respiratory phase has a predetermined phase angle (phase), such as 270° (phase is 0%: exhalation). By applying the insertion distance (S coordinate) at the time when a frame (endoscopic image) having the same phase angle (phase) is imaged to the three-dimensional medical image having the predetermined phase angle (phase), the three-dimensional medical image and the endoscopic image are synchronized based on the respiratory phase, and the position of the tip 443 of the endoscope for the lung field 40 in the three-dimensional medical image can be specified.

[0087] According to the present embodiment, according to the present embodiment, the control unit 62 of the information processing device 6 divides the acquired multiple three-dimensional medical images into a plurality of phase units consisting of four divisions, for example, at 90° (inhalation: 50%), 180° (inhalation: 100%), 270° (exhalation: 50%), and 360° (exhalation: 100%), and specifies the position of the endoscope for the lung field 40 for each of the divided three-dimensional medical images. Thereby, in one cycle in the respiratory phase, sampling can be performed according to the number of the divisions, and the synchronization frequency with the endoscopic image for the three-dimensional medical image can be improved. Since the three-dimensional medical images divided into the plurality of phase units are switched and output according to the respiratory phase, information corresponding to the body movement due to the respiration of the subject's lungs can be efficiently provided to the operator of the endoscope for the lung field 40, such as a doctor.

[0088] According to this embodiment, the operation support information includes information regarding the insertion direction, insertion amount, insertion speed, and target point coordinates indicating the insertion destination of the pulmonary endoscope 40 (coordinates of the area of interest such as a lesion), and therefore can provide useful information to doctors and other operators of the pulmonary endoscope 40, thereby contributing to diagnostic support for doctors and other such persons.

[0089] (Embodiment 2) 9 is a flowchart showing an example of a processing procedure by the control unit 62 of the information processing device 6 according to the second embodiment (real-time 3D image). As in the first embodiment, the control unit 62 of the information processing device 6 starts the processing of the flowchart based on, for example, input contents from the input unit 8 connected to the information processing device itself.

[0090] The control unit 62 of the information processing device 6 acquires the examination date and time, the subject ID, and the endoscopic image output from the endoscope processor 20 (S201). The control unit 62 of the information processing device 6 acquires the examination date and time, the subject ID, and the endoscopic image output from the endoscope processor 20, similar to step S101 in embodiment 1. In this embodiment, the control unit 62 of the information processing device 6 does not need to acquire the insertion distance (S coordinate) and the bending history.

[0091] The control unit 62 of the information processing device 6 acquires an X-ray CT three-dimensional image reconstructed based on a predetermined phase angle from the X-ray CT device 92 (S202). In this embodiment, the three-dimensional medical images acquired by the control unit 62 of the information processing device 6 include an X-ray CT three-dimensional image captured by X-ray CT and a real-time three-dimensional image captured by a CT fluoroscopy function described below. The X-ray CT three-dimensional image corresponds to the three-dimensional medical image described in the first embodiment and is an image reconstructed based on a CT image (tomographic image data) obtained from an examination result of the X-ray CT device 92 or the like performed on the same subject before the examination of the lungs (bronchi) using the lung endoscope 40 (capturing of an endoscopic image). The process of S202 is performed in the same manner as in process S102 of the first embodiment, and an X-ray CT three-dimensional image reconstructed based on a predetermined phase angle is acquired from the X-ray CT device 92.

[0092] The control unit 62 of the information processing apparatus 6 acquires a real-time three-dimensional image from the X-ray CT apparatus 92 (S203). The X-ray CT apparatus 92 is, for example, a high-speed X-ray CT apparatus equipped with a CT fluoroscopy function. The control unit 62 of the information processing apparatus 6 acquires a real-time three-dimensional image taken by the CT fluoroscopy function from the X-ray CT apparatus 92. By using the CT fluoroscopy function implemented in the X-ray CT apparatus 92 (high-speed X-ray CT apparatus), a three-dimensional image in real time (real-time) in units of one second or the like can be continuously reconstructed, and a real-time three-dimensional image can be imaged (reconstructed).

[0093] The imaging by the CT fluoroscopy function and the reconstruction of the real-time three-dimensional image are performed in parallel simultaneously with the imaging (acquisition) of the endoscopic image, and the reconstruction of the real-time three-dimensional image and the imaging (acquisition) of the endoscopic image are continuously processed on the same time axis. Therefore, the real-time three-dimensional image will include the endoscope 40 for the lung field inserted into the bronchus of the lung.

[0094] The control unit 62 of the information processing apparatus 6 acquires a signal (respiratory synchronization signal) corresponding to the respiratory cycle from the respiratory synchronization apparatus 91 (S204). The control unit 62 of the information processing apparatus 6 specifies the phase angle in the respiratory phase based on the respiratory synchronization signal (S205). The control unit 62 of the information processing apparatus 6 performs the processes of S204 and S205 in the same manner as the processes of S103 and S104 in Embodiment 1. The control unit 62 of the information processing apparatus 6 may associate the specified phase angle with the endoscopic image and the real-time three-dimensional image.

[0095] The control unit 62 of the information processing apparatus 6 identifies the position of the endoscope for the lung field 40 in the real-time three-dimensional image (S206). The real-time three-dimensional image is also defined by the body coordinate system shown in three dimensions, similar to the X-ray CT three-dimensional image. The real-time three-dimensional image reconstructed simultaneously with the imaging of the endoscope image includes the endoscope for the lung field 40 that captured the endoscope image. The control unit 62 of the information processing apparatus 6 performs edge detection or pattern detection on the real-time three-dimensional image to identify the tip 443 of the endoscope for the lung field 40, and calculates the position (coordinates in the body coordinate system) of the tip 443, thereby identifying the position of the endoscope for the lung field 40 (tip 443) in the real-time three-dimensional image.

[0096] The control unit 62 of the information processing apparatus 6 identifies the position of the endoscope for the lung field 40 in the X-ray CT three-dimensional image by performing a matching process between the X-ray CT three-dimensional image and the real-time three-dimensional image (S207). When performing the matching process between the X-ray CT three-dimensional image and the real-time three-dimensional image, the control unit 62 of the information processing apparatus 6 performs a comparison process using, for example, the cosine similarity on both images, and determines that these images match when the degree of image matching (similarity), which is the comparison result, is equal to or greater than a predetermined value. Alternatively, the matching process between the X-ray CT three-dimensional image and the real-time three-dimensional image may use an image matching model that outputs (estimates) the degree of matching (similarity) of these images to determine that these images match when the degree of matching (similarity) output by the image matching model is equal to or greater than a predetermined value. Alternatively, the matching process between the X-ray CT three-dimensional image and the real-time three-dimensional image may use a matching process between the coordinate systems (body coordinate systems) in these three-dimensional images. By performing the matching process between the coordinate systems, the position (coordinates) of the endoscope for the lung field 40 (tip 443) identified in the real-time three-dimensional image is converted into the position (coordinates) in the coordinate system in the X-ray CT three-dimensional image, and the position of the endoscope for the lung field 40 (tip 443) in the X-ray CT three-dimensional image may be identified.

[0097] The control unit 62 of the information processing apparatus 6 may output by superimposing the position of the endoscope 40 for lung field (tip portion 443) specified in an X-ray CT three-dimensional image having a higher resolution than the real-time three-dimensional image. Since the X-ray CT three-dimensional image is synchronized with the respiratory phase, the influence of body movement due to respiration can be reduced, the position of the endoscope 40 for lung field in the X-ray CT three-dimensional image can be specified, and the accuracy of position specification can be improved.

[0098] The control unit 62 of the information processing apparatus 6 outputs operation support information based on the derived position of the endoscope 40 for lung field and the position of the region of interest included in the three-dimensional medical images (X-ray CT three-dimensional image, real-time three-dimensional image) (S208). The control unit 62 of the information processing apparatus 6 performs the process of S208 in the same manner as the process S106 of Embodiment 1.

[0099] According to the present embodiment, the three-dimensional medical images include an X-ray CT three-dimensional image taken by X-ray CT before the examination by the endoscope 40 for lung field, and a real-time three-dimensional image taken by the CT fluoroscopy function simultaneously with the examination by the endoscope 40 for lung field. For example, by using the CT fluoroscopy function mounted on a high-speed X-ray CT apparatus, three-dimensional images in real time (real-time) in units of 1 second or the like can be continuously reconstructed, and real-time three-dimensional images can be imaged (reconstructed). Since the imaging of the real-time three-dimensional image is performed simultaneously with the examination by the endoscope 40 for lung field, that is, the imaging of the bronchi of the lung by the endoscope 40 for lung field, the tip portion 443 of the endoscope 40 for lung field is included in the real-time three-dimensional image. Thereby, based on the coordinates of the endoscope 40 for lung field (tip portion 443) in the image coordinate system of the real-time three-dimensional image, the position of the endoscope 40 for lung field (tip portion 443) in the lung (bronchi) shown in the real-time three-dimensional image can be specified. When specifying the position of the endoscope 40 for lung field based on the endoscope 40 for lung field included in the real-time three-dimensional image, the position of the endoscope 40 for lung field in the X-ray CT three-dimensional image may be specified by performing coincidence processing between the X-ray CT three-dimensional image and the real-time three-dimensional image.

[0100] The control unit 62 of the information processing apparatus 6 superimposes and outputs the position of the endoscope 40 for the lung field (tip portion 443) specified by the real-time three-dimensional image and the X-ray CT three-dimensional image with a higher resolution than the real-time three-dimensional image. Since the X-ray CT three-dimensional image is synchronized with the respiratory phase, the influence of body movement due to respiration can be reduced, the position of the endoscope 40 for the lung field in the X-ray CT three-dimensional image can be specified, and the accuracy of the position specification can be improved.

[0101] According to the present embodiment, the X-ray CT three-dimensional image is captured before the real-time three-dimensional image and the examination using the endoscope 40 for the lung field, and the entire lung of the subject is included in the X-ray CT three-dimensional image. Therefore, when there is a region of interest (ROI) such as a lesion or a lesion candidate in the lung, the region of interest is included in the X-ray CT three-dimensional image. In the X-ray CT three-dimensional image synchronized with the respiratory phase, since the position (current location) of the endoscope 40 for the lung field (tip portion 443) at the current time (the imaging time of the endoscope) is specified, using the X-ray CT three-dimensional image, the path from the current location to the location where the region of interest is located can be specified, and operation support information for inserting the endoscope 40 for the lung field along the path can be derived.

[0102] (Embodiment 3) FIG. 10 is a schematic diagram showing an outline of the diagnostic support system according to Embodiment 3 (automatic operation mechanism 434). FIG. 11 is a functional block diagram illustrating functional units included in the control unit of the information processing apparatus 6. The endoscope 40 for the lung field included in the diagnostic support system S according to Embodiment 3 includes an automatic operation mechanism 434 that automatically operates the control button 431 and the bending knob 433 based on the operation support information output from the control unit 62 (operation support information output unit 624) of the information processing apparatus 6.

[0103] The automatic operation mechanism 434 is communicably connected to the information processing device 6 and acquires (receives) the operation support information output (transmitted) from the information processing device 6. The automatic operation mechanism 434 includes, for example, a microcomputer (not shown) that generates an on / off signal or a pulse signal of the control button 431 or the curved knob 433 from the acquired operation support information, and a motor and a cam mechanism (not shown) that operate or drive the control button 431 or the curved knob 433 based on these signals output from the microcomputer. Based on the operation support information output from the information processing device 6 in this way, the automatic operation mechanism 434, the control button 431, the curved knob 433, etc. cooperate with each other, and an automatic operation such as automatic insertion of the endoscope for lung field 40 into the subject's body is performed according to the operation support information.

[0104] Since the operation support information includes information regarding the path from the position of the distal end portion 443 of the endoscope for lung field 40 at the current time point to the position of the region of interest such as a lesion, the distal end portion 443 of the endoscope for lung field 40 can reach the position of the region of interest by performing an automatic operation according to the operation support information. Since the operation support information is derived when the phase angles of the endoscopic image and the three-dimensional medical image match, it is possible to reduce the influence of body movement (body movement) due to breathing in the lung and perform an automatic operation such as automatic insertion of the endoscope for lung field 40.

[0105] The operation support information obtained by the automatic operation mechanism 434 from the control unit 62 (operation support information output unit 624) of the information processing device 6 is not limited to the operation support information such as the insertion and bending of the insertion unit 44. For example, the insertion unit 44 of the endoscope 40 for the lung field is provided with an air injection unit (not shown) or a hand unit (not shown), and the operation support information may include information related to operations such as air injection by the air injection unit or excision (sampling) of the lesion site by the hand unit. That is, based on the shape information and distance image information of the internal body part (bronchus) specified in the acquired three-dimensional medical image, the operation support information output unit 624 generates information related to the operation of the air injection unit or the hand unit, includes it in the operation support information, and outputs it to the automatic operation mechanism 434. The automatic operation mechanism 434 may automatically operate the air injection unit or the hand unit based on the acquired operation support information. Note that the information related to the operation of the air injection unit or the hand unit included in the operation support information may be displayed superimposed on the endoscope image or the like on the integrated image display screen 71.

[0106] According to the present embodiment, based on the position of the endoscope 40 for the lung field specified in the three-dimensional medical image synchronized with the respiratory phase, operation support information related to the operation of the endoscope 40 for the lung field can be efficiently output, and the automatic operation mechanism 434 automatically operates the endoscope 40 for the lung field according to the operation support information output from the information processing device 6. Therefore, it is possible to provide a diagnostic support system S that efficiently supports the operation for an operator such as a doctor who operates the endoscope 40 for the lung field.

[0107] The embodiments disclosed this time should be considered as illustrative in all respects and not restrictive. The technical features described in each embodiment can be combined with each other, and the scope of the present invention is intended to include all modifications within the scope of the claims and the scope equivalent to the claims.

Description of Reference Numerals

[0108] S Diagnostic Support System 10 Endoscope Device 15 Keyboard 16 Storage Shelf 20 Endoscope Processor 21 Control Unit 211 Image Processing Unit 22 Main Memory Device 23 Auxiliary Memory Device 24 Communication Unit 25 Touch Panel 26 Display Device I / F 27 Input Device I / F 28 Reading Unit 31 Endoscope Connector 311 Electrical Connector 312 Optical Connector 33 Light Source 40 Endoscope for Lung Field 43 Operation Unit 431 Control Button 433 Bending Knob 434 Automatic Operation Mechanism 44 Insertion Section (Flexible Tube) 441 Flexible Section 442 Bending Section 443 Tip Section 444 Imaging Section 445 Imaging Element 446 Light Source for Photography 45 Stopping Section 48 Scope Connector 49 Universal Code 50 Display Device 6 Information Processing Device 61 Communication Unit 62 Control Unit 621 Acquisition Unit 622 Respiratory Phase Identification Unit 623 Position Derivation Unit 624 Operation Support Information Output Unit 63 Memory Unit 631 Endoscope Image DB 632 Recording Medium P Program (Program Product) 64 Input / Output I / F 7 Display Section 8 Input Section 91 Respiratory Synchronization Device 92 X-ray CT Device

Claims

1. Cause a computer to acquire information regarding the respiratory phase of a subject, acquire a three-dimensional medical image obtained by imaging the inside of the subject's body in synchronization with the respiratory phase so as to have a predetermined phase angle, acquire an endoscopic image of the subject from an endoscope for the lung field, specify the position of the endoscope for the lung field in the three-dimensional medical image synchronized with the respiratory phase, output the specified position of the endoscope for the lung field in association with the three-dimensional medical image, wherein the three-dimensional medical image is reconstructed at a predetermined phase angle in the respiratory phase, output information regarding the correspondence between the phase angle at the time of imaging of the acquired endoscopic image and the phase angle of the three-dimensional medical image A program for executing the process.

2. The process of specifying the position of the endoscope for the lung field includes acquiring information regarding the insertion distance of the endoscope for the lung field inserted into the subject's body at the time of imaging the endoscopic image in association with the respiratory phase, and specifying the position of the endoscope for the lung field in the three-dimensional medical image synchronized with the respiratory phase based on the information regarding the insertion distance associated with the respiratory phase. The program according to claim 1.

3. Divide the acquired plurality of three-dimensional medical images into a plurality of phase units in the respiratory phase, and specify the position of the endoscope for the lung field for each of the divided three-dimensional medical images. The program according to claim 1 or claim 2.

4. Switch and output the three-dimensional medical images divided into the plurality of phase units according to the respiratory phase. The program according to claim 3.

5. The three-dimensional medical image includes a real-time three-dimensional image imaged using a CT fluoroscopy function in parallel with the imaging of the endoscopic image by the endoscope for the lung field, and specify the position of the endoscope for the lung field based on the endoscope for the lung field included in the real-time three-dimensional image. The program according to any one of claims 1 to 4.

6. The three-dimensional medical image includes an X-ray CT three-dimensional image imaged by an X-ray CT device and the real-time three-dimensional image, and by performing a coincidence process between the X-ray CT three-dimensional image and the real-time three-dimensional image, specify the position of the endoscope for the lung field in the X-ray CT three-dimensional image based on the endoscope for the lung field included in the real-time three-dimensional image. The program according to claim 5.

7. Based on the three-dimensional medical image in which the position of the endoscope for the lung field is specified, output operation support information regarding the operation of the endoscope for the lung field The program according to any one of claims 1 to 6

8. The operation support information includes information regarding a path to a region of interest derived based on the three-dimensional medical image The program according to claim 7

9. Output the operation support information to an automatic operation mechanism that performs automatic operation of the endoscope for the lung field The program according to claim 7 or claim 8

10. Obtain information regarding the respiratory phase of the subject by referring to a storage unit, Obtain a three-dimensional medical image obtained by imaging the inside of the subject in synchronization with the respiratory phase so as to have a predetermined phase angle by referring to the storage unit, Obtain an endoscopic image of the subject from the endoscope for the lung field by referring to a storage unit, Specify the position of the endoscope for the lung field in the three-dimensional medical image synchronized with the respiratory phase, Output the specified position of the endoscope for the lung field in association with the three-dimensional medical image, The three-dimensional medical image is reconstructed at a predetermined phase angle in the respiratory phase, Output information regarding the correspondence between the phase angle at the imaging time of the obtained endoscopic image and the phase angle of the three-dimensional medical image An information processing method executed by a computer

11. A phase information acquisition unit that acquires information regarding the respiratory phase of the subject, A three-dimensional medical image acquisition unit that acquires a three-dimensional medical image obtained by imaging the inside of the subject in synchronization with the respiratory phase so as to have a predetermined phase angle, An endoscopic image acquisition unit that acquires an endoscopic image of the subject from the endoscope for the lung field, A position specification unit that specifies the position of the endoscope for the lung field in the three-dimensional medical image synchronized with the respiratory phase, An output unit that outputs the specified position of the endoscope for the lung field in association with the three-dimensional medical image and The three-dimensional medical image is reconstructed at a predetermined phase angle in the respiratory phase, Output information regarding the correspondence between the phase angle at the imaging time of the obtained endoscopic image and the phase angle of the three-dimensional medical image An information processing apparatus

12. An endoscope for the lung field, An automatic operation mechanism that performs automatic operation of the endoscope for the lung field, A control unit that controls the automatic operation mechanism, and The control unit acquires information regarding the respiratory phase of the subject, acquires a three-dimensional medical image obtained by imaging the inside of the subject in synchronization with the respiratory phase so as to have a predetermined phase angle, An endoscopic image of the subject is acquired from the endoscope for the lung field, the position of the endoscope for the lung field in the three-dimensional medical image synchronized with the respiratory phase is specified, based on the three-dimensional medical image in which the position of the endoscope for the lung field is specified, operation support information regarding the operation of the endoscope for the lung field is output to the automatic operation mechanism, the three-dimensional medical image is reconstructed at a predetermined phase angle in the respiratory phase, information regarding the correspondence between the phase angle at the imaging time of the acquired endoscopic image and the phase angle of the three-dimensional medical image is output Diagnosis support system.

Images