Support device, support method, and support program

Abstract

To provide a support device, a support method, and a support program capable of effectively supporting a procedure of inserting a medical device into a tubular structure.SOLUTION: The support device, which supports insertion of a medical device into a tubular structure, acquires a two-dimensional medical image and a three-dimensional path being three-dimensional information of a path which the medical device passes through, derives a first feature amount expressing the relation between a reference point on a three-dimensional space and the three-dimensional path, performs control of displaying a synthetic image obtained by superimposing the three-dimensional path on the two-dimensional medical image, and differentiates a display mode of the three-dimensional path corresponding to the first feature amount in the synthetic image.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an assistance device, an assistance method, and an assistance program. [Background technology]

[0002] Patent Document 1 discloses a technique for displaying a composite image on a display, in which preoperative preparation information is superimposed on a two-dimensional surgical field image captured by an endoscope. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO2023 / 162657A1 Summary of the Invention [Problem to be solved by the invention]

[0004] In the medical field, medical instruments are inserted into tubular structures. For example, there is a procedure called transbronchial lung biopsy, which combines preoperative computed tomography (CT) images, bronchoscopy, and intraoperative X-ray images to make a definitive diagnosis of lung cancer. In this procedure, in peripheral areas where a bronchoscope cannot be inserted, the practitioner inserts a treatment tool while referring to two-dimensional images obtained by X-ray fluoroscopy to perform a biopsy. This procedure is relatively difficult because it is difficult to determine the depth position of the lesion in the two-dimensional image and lesions with relatively low cell density are difficult to see. Therefore, there is a need for effective support for the procedure of inserting medical instruments into tubular structures.

[0005] The present disclosure has been made in consideration of the above circumstances, and aims to provide an assistance device, an assistance method, and an assistance program that can effectively assist the procedure of inserting a medical instrument into a tubular structure. [Means for solving the problem]

[0006] A first aspect of the support device is an support device that includes at least one processor and supports the insertion of a medical instrument into a tubular structure. The processor acquires a two-dimensional medical image and a three-dimensional path that is three-dimensional information about the path the medical instrument takes, derives a first feature that represents the relationship between a reference point in three-dimensional space and the three-dimensional path, controls the display of a composite image in which the three-dimensional path is superimposed on the two-dimensional medical image, and varies the display mode of the three-dimensional path in the composite image depending on the first feature.

[0007] In the second aspect of the support device, in the support device of the first aspect, the processor acquires the three-dimensional position of the region of interest, derives a second feature that represents the relationship between the reference point and the three-dimensional position of the region of interest, controls the display of a composite image in which the region of interest is superimposed on a two-dimensional medical image, and varies the display mode of the region of interest in the composite image depending on the second feature.

[0008] The support device of the third aspect is the support device of the first or second aspect, in which the processor acquires a gaze direction for projecting a three-dimensional route onto a two-dimensional plane, and the first feature is a feature representing the relative positional relationship between the reference point and the gaze direction of the three-dimensional route.

[0009] A fourth aspect of the support device is the support device of the second aspect, in which the processor acquires a gaze direction for projecting a three-dimensional path onto a two-dimensional plane, and the second feature is a feature representing the relative positional relationship of the gaze direction between the reference point and the three-dimensional position of the area of ​​interest.

[0010] The support device of a fifth aspect is the support device of any one of the first to fourth aspects, wherein the reference point is a point on a three-dimensional path.

[0011] The assistance device of a sixth aspect is the assistance device of the fifth aspect, wherein the reference point is the position of the medical instrument on the three-dimensional path.

[0012] The support device of a seventh aspect is the support device of the sixth aspect, wherein the medical instrument is an endoscope equipped with a treatment tool, and the reference point is the position of the treatment tool on the three-dimensional path.

[0013] An eighth aspect of the support device is the support device of the sixth aspect, wherein the medical instrument is an endoscope equipped with a camera, and the reference point is the position of the camera on the three-dimensional path.

[0014] The support device of a ninth aspect is the support device of the first aspect, wherein the first feature amount is a distance in three-dimensional space between the reference point and the three-dimensional path.

[0015] The assistance device of a tenth aspect is the assistance device of the second aspect, wherein the reference point is a three-dimensional position of the region of interest.

[0016] The support device of the eleventh aspect is a support device of any one of the first to tenth aspects, in which the processor acquires a salient route on a three-dimensional route and, in controlling the display of a composite image, controls the salient route to be highlighted compared to routes other than the salient route on the three-dimensional route.

[0017] The assistance device of a twelfth aspect is the assistance device of the eleventh aspect, wherein the salient route is a route connecting a reference point and a point on the three-dimensional route whose distance from the region of interest is equal to or less than a threshold.

[0018] The support device of a thirteenth aspect is the support device of the eleventh aspect, wherein the remarkable route is a route registered in advance as preoperative information.

[0019] The assistance device of a fourteenth aspect is the assistance device of any one of the first to thirteenth aspects, wherein the path through which the medical instrument passes is the bronchial cavity.

[0020] A support device according to a fifteenth aspect is the support device according to any one of the first to fourteenth aspects, wherein the two-dimensional medical image is an image obtained by radiography.

[0021] The support device of the 16th aspect is the support device of any one of the 1st to 14th aspects, wherein the three-dimensional path is the bronchial lumen in a three-dimensional medical image, and the two-dimensional medical image is a virtual radiological image generated from the three-dimensional medical image.

[0022] The support device of the 17th aspect is a support device of any one of the first to sixteenth aspects, in which the processor acquires a three-dimensional medical image including a path taken by a medical instrument, deforms the three-dimensional medical image by aligning the three-dimensional medical image with a two-dimensional medical image, and acquires the three-dimensional path based on the deformed three-dimensional medical image.

[0023] In an 18th aspect of the assistance method, a processor of an assistance device that has at least one processor and assists in the insertion of a medical instrument into a tubular structure acquires a two-dimensional medical image and a three-dimensional path that is three-dimensional information about the path taken by the medical instrument, derives a first feature that represents the relationship between a reference point in three-dimensional space and the three-dimensional path, controls the display of a composite image in which the three-dimensional path is superimposed on the two-dimensional medical image, and executes processing to vary the display manner of the three-dimensional path in the composite image depending on the first feature.

[0024] The assistance program of a 19th aspect includes at least one processor, and causes a processor of an assistance device that assists in the insertion of a medical instrument into a tubular structure to acquire a two-dimensional medical image and a three-dimensional path that is three-dimensional information about the path taken by the medical instrument, derive a first feature that represents the relationship between a reference point in three-dimensional space and the three-dimensional path, control the display of a composite image in which the three-dimensional path is superimposed on the two-dimensional medical image, and execute processing that changes the display manner of the three-dimensional path in the composite image depending on the first feature. [Effects of the Invention]

[0025] According to the present disclosure, it is possible to effectively assist the procedure of inserting a medical instrument into a tubular structure. [Brief explanation of the drawings]

[0026] [Figure 1]FIG. 1 is a diagram illustrating an example of a medical information system. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the support device. [Figure 3] FIG. 2 is a block diagram showing an example of a functional configuration of the support device. [Figure 4] 10A and 10B are diagrams illustrating an example of a perspective image and a composite image. [Figure 5] FIG. 10 is a diagram showing an example of a lesion area and a portion of a route of interest in a composite image. [Figure 6] FIG. 10 is a diagram showing an example of a portion of a salient path in a composite image. [Figure 7] 10 is a flowchart illustrating an example of an insertion support process. [Figure 8] 10A and 10B are diagrams showing an example of a lesion area and a portion of a route of interest in a composite image according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0027] Hereinafter, examples of embodiments for carrying out the technology of the present disclosure will be described in detail with reference to the drawings.

[0028] First, the configuration of a medical information system 10 will be described with reference to Fig. 1. As shown in Fig. 1, the medical information system 10 includes a support device 1, a three-dimensional image capturing device 2, a fluoroscopic image capturing device 3, and an image storage server 4. The support device 1, the three-dimensional image capturing device 2, the fluoroscopic image capturing device 3, and the image storage server 4 are each connected via a network 5 in a state where they can communicate with each other.

[0029] The three-dimensional imaging device 2 captures an image of a diagnostic target region of the subject H, thereby generating a three-dimensional medical image representing the region. Examples of the three-dimensional imaging device 2 include a CT device, an MRI (Magnetic Resonance Imaging) device, and a PET (Positron Emission Tomography) device. The three-dimensional medical image, consisting of a plurality of tomographic images, generated by the three-dimensional imaging device 2 is transmitted to and stored in the image storage server 4. In this embodiment, the target region of the subject H is the lungs, and the three-dimensional imaging device 2 is a CT device. That is, the three-dimensional medical image according to this embodiment is a CT image. In this embodiment, it is assumed that, before a treatment is performed on the subject H, the three-dimensional imaging device 2 captures an image of the subject H's chest, thereby obtaining a three-dimensional medical image including the subject H's chest.

[0030] The fluoroscopic imaging device 3 has a C-arm 3A, an X-ray source 3B, and an X-ray detector 3C. The X-ray source 3B and the X-ray detector 3C are attached to both ends of the C-arm 3A, respectively. In the fluoroscopic imaging device 3, the C-arm 3A is configured to be rotatable and movable so that the subject H can be imaged from any direction. During a treatment on the subject H, the fluoroscopic imaging device 3 performs fluoroscopic imaging by continuously irradiating the subject H with X-rays, as an example of radiation, at a predetermined frame rate and sequentially detecting the X-rays that have passed through the subject H with the X-ray detector 3C, thereby sequentially acquiring radiographic images of the subject H. In the following description, the radiographic images of each sequentially acquired frame will be referred to as a fluoroscopic image. The fluoroscopic image acquired by radiographic imaging using the fluoroscopic imaging device 3 is an example of a two-dimensional medical image according to the disclosed technology.

[0031] The image storage server 4 is a computer that stores and manages various data, and is equipped with a large-capacity external storage device and database management software. The image storage server 4 communicates with other devices via a wired or wireless network 5, sending and receiving image data and the like. Specifically, the image storage server 4 acquires various data via the network 5, including image data representing 3D medical images acquired by the 3D imaging device 2 and fluoroscopic images acquired by the fluoroscopic imaging device 3, and stores and manages the data on a recording medium such as a large-capacity external storage device. The storage format of the image data and communication between devices via the network 5 are based on protocols such as DICOM (Digital Imaging and Communication in Medicine).

[0032] In this embodiment, a case will be described in which a biopsy procedure is performed by excising a portion of a lesion, such as a pulmonary nodule, present in the lungs of the subject H while performing fluoroscopic imaging of the subject H to examine the presence of the disease in detail. For this purpose, the fluoroscopic imaging device 3 is disposed in a treatment room for performing the biopsy. An ultrasound endoscope device 6 is also installed in the treatment room. The ultrasound endoscope device 6 includes an endoscope 6A having an ultrasound probe and treatment tools, such as forceps, attached to its tip. In this embodiment, an operator, such as a doctor, inserts the endoscope 6A into the bronchi of the subject H to perform a biopsy of the lesion. During the biopsy, the fluoroscopic imaging device 3 captures fluoroscopic images of the subject H and displays the captured fluoroscopic images in real time. The operator checks the position of the tip of the endoscope 6A within the subject H on the fluoroscopic images and moves the tip of the endoscope 6A to the location of the target lesion. The endoscope 6A is an example of a medical instrument according to the disclosed technology. The bronchial lumen is an example of a path through which the endoscope 6A according to the disclosed technology passes.

[0033] Since lung lesions such as pulmonary nodules occur outside the bronchi, not inside, the operator moves the tip of the endoscope 6A to the target position, then performs a procedure to extract part of the lesion using a treatment tool such as forceps while confirming the location of the lesion in an ultrasound image obtained by photographing the outside of the bronchi with the ultrasound probe.

[0034] In this embodiment, the three-dimensional medical image and the fluoroscopic image are obtained by capturing an imaging range including a region to be diagnosed of the same subject H. That is, at least a part of the imaging ranges of the three-dimensional imaging device 2 and the fluoroscopic imaging device 3 overlap. Therefore, the three-dimensional medical image obtained by the three-dimensional imaging device 2 also includes the bronchial lumen, which is the path through which the endoscope 6A passes.

[0035] Next, with reference to FIG. 2, a hardware configuration of the support device 1 according to this embodiment will be described. The support device 1 is a device that supports the insertion of an endoscope 6A into a bronchus, which is an example of a tubular structure. As shown in FIG. 2, the support device 1 includes a CPU (Central Processing Unit) 20, a memory 21 as a temporary storage area, and a non-volatile storage unit 22. The support device 1 also includes a display 23 such as a liquid crystal display, an input device 24 such as a keyboard and a mouse, and a network I / F (Interface) 25 connected to a network 5. The input device 24 may be a touch panel integrated with the display 23. The CPU 20, the memory 21, the storage unit 22, the display 23, the input device 24, and the network I / F 25 are connected to a bus 27. The CPU 20 is an example of a processor according to the disclosed technology. An example of the support device 1 is a computer such as a personal computer or a server computer.

[0036] The storage unit 22 is realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage unit 22, which serves as a storage medium, stores an assistance program 30. The CPU 20 reads the assistance program 30 from the storage unit 22, expands it in the memory 21, and executes the expanded assistance program 30.

[0037] The storage unit 22 also stores three-dimensional medical images 32 and three-dimensional path data 34. The three-dimensional medical images 32 include a group of tomographic images obtained by capturing images of a subject H who is the target of biopsy. The three-dimensional medical images 32 are acquired from the image storage server 4 via the network 5 prior to the biopsy. The three-dimensional path data 34 will be described in detail later.

[0038] Next, the functional configuration of the support device 1 will be described with reference to Fig. 3. As shown in Fig. 3, the support device 1 includes an acquisition unit 40, a transformation unit 42, a first generation unit 44, a derivation unit 46, an identification unit 48, a second generation unit 50, and a display control unit 52. When the CPU 20 executes the support program 30, the support device 1 functions as the acquisition unit 40, the transformation unit 42, the first generation unit 44, the derivation unit 46, the identification unit 48, the second generation unit 50, and the display control unit 52.

[0039] The acquisition unit 40 acquires the three-dimensional medical image 32 from the storage unit 22. The acquisition unit 40 also sequentially acquires, via the network I / F 25, fluoroscopic images captured by the fluoroscopic image capturing device 3 at a predetermined frame rate.

[0040] The deformation unit 42 deforms the 3D medical image 32 by aligning the 3D medical image 32 acquired by the acquisition unit 40 with the fluoroscopic image. For example, the deformation unit 42 derives deformation parameters for affine transformation so that the positions of parts depicted in the 3D medical image 32 and the fluoroscopic image, such as the bronchi and pulmonary artery, are aligned. The deformation unit 42 then deforms the 3D medical image 32 using the derived deformation parameters. Note that the deformation unit 42 may also align the 3D medical image 32 with the fluoroscopic image using a known method other than affine transformation, such as rigid transformation or nonlinear non-rigid transformation. The deformation unit 42 may also align the 3D medical image 32 with the fluoroscopic image using a trained model obtained by machine learning, such as deep learning.

[0041] The first generating unit 44 acquires a three-dimensional path, which is three-dimensional information about the bronchial lumen as an example of a path taken by the endoscope 6A, based on the three-dimensional medical image 32 after deformation by the deformation unit 42. Specifically, the first generating unit 44 generates three-dimensional path data 34, which is a collection of voxel data, by performing 3D (three-dimensional) modeling that numerically describes the three-dimensional shape of the body of the subject H, based on the three-dimensional medical image 32 after deformation. Voxel data is a unit of pixel in three-dimensional space, and has three-dimensional coordinate information and pixel values. In this embodiment, the imaging range of the three-dimensional medical image 32 includes the bronchi. That is, the three-dimensional path data 34 includes a three-dimensional path, which is three-dimensional information about the bronchial lumen. This three-dimensional path is a collection of voxel data corresponding to the bronchial lumen.

[0042] In this embodiment, a lesion region to be treated, such as a pulmonary nodule, is specified in advance as the region of interest. The 3D medical image 32 also includes the lesion region. Therefore, the 3D path data 34 also includes the 3D position of the lesion region. Note that the lesion is not limited to a pulmonary nodule, but may be a tumor, etc. The region of interest is not limited to a lesion region, but may be an anatomical region. An anatomical region refers to, for example, a specific region including the anatomical structure of a living body. An anatomical region includes, for example, bones, muscles, organs, blood vessels, etc.

[0043] The derivation unit 46 acquires a three-dimensional route from the three-dimensional route data 34. The derivation unit 46 also derives a feature amount (hereinafter referred to as a "first feature amount") that represents the relationship between a reference point in the three-dimensional space and the three-dimensional route. A specific example of the process of deriving the first feature amount by the derivation unit 46 will be described below.

[0044] The derivation unit 46 acquires a line of sight direction for projecting the three-dimensional path onto a two-dimensional plane. This line of sight direction is used when the second generation unit 50, which will be described later, generates a composite image. The derivation unit 46 also sets a reference point on the three-dimensional path. For example, the derivation unit 46 sets the position of the treatment tool of the endoscope 6A on the three-dimensional path as the reference point. For example, the operator specifies the position of the treatment tool on the fluoroscopic image displayed on the display 23 via the input device 24. The derivation unit 46 back-projects the position of the treatment tool on the specified fluoroscopic image to set it as the reference point on the three-dimensional path.

[0045] The derivation unit 46 also derives, as the first feature quantity, a feature quantity representing the relative positional relationship between the reference point and the three-dimensional route in the line of sight direction. As described above, the three-dimensional route is a collection of voxel data, and the voxel data has three-dimensional coordinate information. The derivation unit 46 derives, as the first feature quantity, the distance between the reference point and the three-dimensional route along the line of sight direction, based on the line of sight direction, the three-dimensional coordinates of the reference point, and the three-dimensional coordinate information of each piece of voxel data constituting the three-dimensional route. In this embodiment, the derivation unit 46 assigns a negative sign to the distance between the reference point and the three-dimensional route that is closer to the viewpoint than the reference point, i.e., closer to the viewpoint than the reference point when viewed from the viewpoint. The derivation unit 46 also assigns a positive sign to the distance between the reference point and the three-dimensional route that is farther from the viewpoint than the viewpoint than the reference point, i.e., closer to the viewpoint than the reference point when viewed from the viewpoint. The derivation unit 46 may treat the distance as an absolute value and add information to the absolute value representing the distance indicating whether the distance is closer to the viewpoint position than the reference point or farther from the viewpoint position than the reference point.

[0046] The derivation unit 46 also acquires the three-dimensional position of the lesion area from the three-dimensional path data 34. The derivation unit 46 also derives a feature amount (hereinafter referred to as a "second feature amount") that represents the relationship between the reference point and the three-dimensional position of the lesion area. Similar to the first feature amount, the derivation unit 46 derives a feature amount that represents the relative positional relationship of the gaze direction with respect to the three-dimensional position of the lesion area as the second feature amount. That is, the derivation unit 46 derives the distance along the gaze direction between the reference point and the three-dimensional position of the lesion area as the second feature amount based on the gaze direction, the three-dimensional coordinates of the reference point, and the three-dimensional position of the lesion area.

[0047] The identification unit 48 acquires a noteworthy path on the three-dimensional path from the three-dimensional path data 34. Specifically, the identification unit 48 derives the distance between the three-dimensional path and the lesion area. This distance may be a distance in the three-dimensional space represented by the three-dimensional path data 34, or a distance along the line of sight. The identification unit 48 then identifies a path connecting a point on the three-dimensional path where the derived distance is equal to or less than a threshold and a reference point as a noteworthy path, and acquires the noteworthy path from the three-dimensional path data 34. In other words, the noteworthy path corresponds to a path toward the lesion area on the three-dimensional path. The threshold in this case is set in advance depending on, for example, the type of lesion and the type of organ that includes the path through which the medical instrument passes.

[0048] The second generating unit 50 generates a composite image by superimposing the three-dimensional path on the perspective image. A specific example of the process of generating a composite image by the second generating unit 50 will be described below.

[0049] First, the second generation unit 50 sets a viewpoint position when projecting the three-dimensional path onto a two-dimensional plane. This two-dimensional plane is a virtually set two-dimensional projection surface. Since the three-dimensional path is superimposed on the fluoroscopic image acquired by the X-ray detector 3C, the two-dimensional plane corresponds to the X-ray detection surface of the X-ray detector 3C. For example, the second generation unit 50 sets a position corresponding to the position of the X-ray source 3B with respect to the two-dimensional plane as the viewpoint position based on the relative positional relationship between the X-ray detector 3C and the X-ray source 3B. The direction from the viewpoint position to a point on the two-dimensional plane becomes the line of sight for projecting the three-dimensional path onto the two-dimensional plane.

[0050] The second generating unit 50 generates a composite image in which the three-dimensional path is superimposed on the perspective image by projecting the three-dimensional path onto a two-dimensional plane using a known method such as perspective projection and performing registration using known regions such as the bronchi, pulmonary artery, and lesion region. At this time, the second generating unit 50 also superimposes the lesion region included in the three-dimensional path data 34 on the perspective image, just like the three-dimensional path.

[0051] The second generating unit 50 assigns, to the composite image, color information representing a color different from the binary color used in the perspective image to the route of interest on the three-dimensional route. Specifically, the second generating unit 50 assigns, to the route of interest, color information representing a color according to the first feature amount derived by the derivation unit 46. Furthermore, in the composite image, the second generating unit 50 assigns, to the lesion area, color information representing a color according to the second feature amount derived by the derivation unit 46, similar to the route of interest.

[0052] The display control unit 52 controls the display of the composite image generated by the second generation unit 50 on the display 23. FIG. 4 shows an example of a perspective image and a composite image. As shown in FIG. 4, in the composite image, a color corresponding to a first feature amount is assigned to the salient route superimposed on the perspective image, and a color corresponding to a second feature amount is assigned to the lesion area. The thick solid line portion in the composite image of FIG. 4 represents the salient route. The thick dashed line portion in the composite image of FIG. 4 represents a route other than the salient route on the three-dimensional route. That is, in controlling the display of the composite image, the display control unit 52 controls the display of the salient route to be highlighted compared to routes other than the salient route on the three-dimensional route.

[0053] FIG. 5 shows an example of an enlarged view of the target route and the lesion area in the composite image. As shown in FIG. 5, in the composite image, each pixel representing the target route is assigned a color corresponding to a first feature value that indicates the relative positional relationship in the line of sight between the point on the target route corresponding to that pixel and the reference point. That is, the display control unit 52 changes the display mode of the three-dimensional route in the composite image according to the first feature value. Therefore, by viewing the color of the target route on the three-dimensional route, the operator can grasp information about the depth direction of the path through which the endoscope 6A passes. Furthermore, because the reference point is set at the position of the treatment tool attached to the tip of the endoscope 6A, the operator can grasp information about the depth direction relative to the treatment tool.

[0054] 5, in the composite image, each pixel representing the lesion area is assigned a color corresponding to a second feature value that represents the relative positional relationship in the line of sight between the point in the lesion area corresponding to that pixel and the reference point. That is, display controller 52 changes the display mode of the lesion area in the composite image according to the second feature value. Therefore, by viewing the color of the lesion area, the operator can grasp information about the depth direction of the lesion area to be treated.

[0055] As shown in Fig. 5, the display control unit 52 may perform control to display a legend of colors according to the first feature amount and the second feature amount on the same screen as the composite image. Also, as shown in Fig. 6, color information according to the second feature amount does not need to be assigned to the lesion area.

[0056] Next, the operation of the support device 1 will be described with reference to Fig. 7. The CPU 20 executes the support program 30, thereby executing the insertion support process shown in Fig. 7. The insertion support process is executed, for example, when the operator specifies the position of a treatment tool on a fluoroscopic image.

[0057] 7, the acquisition unit 40 acquires the three-dimensional medical image 32 from the storage unit 22. In step S12, the fluoroscopic image captured by the fluoroscopic image capturing device 3 at a predetermined frame rate is acquired via the network I / F 25. In step S14, the deformation unit 42 deforms the three-dimensional medical image 32 by aligning the three-dimensional medical image 32 acquired in step S10 with the fluoroscopic image acquired in step S12, as described above.

[0058] In step S16, the first generation unit 44 generates three-dimensional route data 34 based on the three-dimensional medical image 32 after deformation by the processing of step S14, as described above. In step S18, the derivation unit 46 acquires a three-dimensional route from the three-dimensional route data 34 generated in step S16. Furthermore, the derivation unit 46 derives a first feature amount representing the relationship between a reference point in three-dimensional space and the three-dimensional route, as described above.

[0059] In step S20, the derivation unit 46 acquires the three-dimensional position of the lesion area from the three-dimensional route data 34 generated in step S16. As described above, the derivation unit 46 also derives a second feature amount representing the relationship between the reference point and the three-dimensional position of the lesion area. In step S22, the identification unit 48 acquires a salient route on the three-dimensional route from the three-dimensional route data 34 generated in step S16.

[0060] In step S24, the second generation unit 50 generates a composite image by superimposing the three-dimensional path and the lesion area on the fluoroscopic image, as described above. In the composite image, the second generation unit 50 assigns color information representing a color according to the first feature amount derived in step S18 to the path of interest acquired in step S22. In addition, the second generation unit 50 assigns color information representing a color according to the second feature amount derived in step S20 to the lesion area in the composite image. In step S26, the display control unit 52 controls the display 23 to display the composite image generated in step S24. When the processing of step S26 ends, the insertion support processing ends.

[0061] As described above, according to this embodiment, it is possible to effectively assist the procedure of inserting a medical instrument into a tubular structure.

[0062] In the above embodiment, the derivation unit 46 sets the position of the treatment tool of the endoscope 6A on the three-dimensional path as the reference point. However, the disclosed technology is not limited to this. For example, the derivation unit 46 may set the position of the camera of the endoscope 6A on the three-dimensional path as the reference point. Furthermore, for example, the derivation unit 46 may set the three-dimensional position of a region of interest that is not located on the three-dimensional path as the reference point. FIG. 8 shows an example of a composite image when a predetermined point within a lesion region is set as the reference point as an example of a region of interest. As shown in FIG. 8, in this embodiment, the operator can determine whether each position on the target path is close to or far from the lesion region.

[0063] In the above embodiment, a feature representing the relative positional relationship between the reference point and the three-dimensional route in terms of the line of sight direction is used as the first feature, but the disclosed technology is not limited to this. For example, the distance in three-dimensional space between the reference point and the three-dimensional route may be used as the first feature. In this embodiment, the derivation unit 46 derives the distance in three-dimensional space between the reference point and the three-dimensional route based on the three-dimensional coordinates of the reference point and the three-dimensional coordinates of the three-dimensional route.

[0064] In the above embodiment, a case has been described in which a route connecting a reference point and a point on a three-dimensional route whose distance from the region of interest is equal to or less than a threshold has been applied as a route of interest, but the disclosed technology is not limited to this. For example, a route registered in advance as preoperative information may be applied as a route of interest.

[0065] In the above embodiment, a case has been described in which a fluoroscopic image captured by the fluoroscopic imaging device 3, i.e., a radiological image obtained by actually irradiating radiation, is used as the two-dimensional medical image, but the disclosed technology is not limited to this. For example, a virtual radiological image generated from a three-dimensional medical image may be used as the two-dimensional medical image.

[0066] In the above embodiment, the display control unit 52 changes the display mode by changing the colors of the three-dimensional route and the region of interest, but the disclosed technology is not limited to this. For example, the display control unit 52 may change the display mode by changing the transparency, border thickness, border type, etc. of the three-dimensional route and the region of interest. When changing the transparency, an example is a mode in which the transparency increases with increasing distance from the reference point.

[0067] In the above embodiment, the display control unit 52 controls the display of the prominent route in the composite image by adding a color to the prominent route, thereby highlighting the prominent route. However, the disclosed technology is not limited to this. For example, the display control unit 52 may control the prominent route to be highlighted by changing the transparency, the thickness and type of the border of the prominent route.

[0068] In the above embodiment, a case has been described in which a bronchi is used as the tubular structure and a bronchial lumen is used as the path through which the medical instrument passes, but the disclosed technology is not limited to this. For example, a blood vessel may be used as the tubular structure and a blood vessel lumen may be used as the path through which the medical instrument passes.

[0069] Furthermore, in the above embodiment, for example, the following various processors can be used as the hardware structure of a processing unit that executes various processes such as each functional unit of the support device 1. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to execute specific processes, such as a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0070] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0071] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by computers such as client and server. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0072] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0073] In the above embodiment, the assistance program 30 is pre-stored (installed) in the storage unit 22, but the present invention is not limited to this. The assistance program 30 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The assistance program 30 may also be downloaded from an external device via a network. [Explanation of symbols]

[0074] 1 Support equipment 2. 3D imaging device 3. Fluoroscopic imaging device 3A C-arm 3B X-ray source 3C X-ray detector 4. Image storage server 5. Network 6. Endoscopic ultrasound equipment 6A Endoscope 10 Medical Information Systems 20 CPU 21 Memory 22 Memory section 23 Display 24 Input Devices 25 Network I / F 27 Bus 30 Support Programs 32 3D Medical Imaging 34 3D route data 40 Acquisition Department 42 Deformed part 44 1st generation part 46 Derivation part 48 Specific part 50 Generation 2 52 represents the Ministry of Control H was found

Claims

1. 1. An assist device for assisting insertion of a medical device into a tubular structure, the assist device comprising at least one processor, the assist device comprising: The processor: Acquiring a two-dimensional medical image and a three-dimensional path that is three-dimensional information of a path along which the medical instrument passes; deriving a first feature amount representing a relationship between a reference point in a three-dimensional space and the three-dimensional path; Controlling the display of a composite image in which the three-dimensional path is superimposed on the two-dimensional medical image, and varying the display mode of the three-dimensional path in the composite image according to the first feature amount. Support equipment.

2. The processor: Obtaining the three-dimensional position of the region of interest; deriving a second feature amount representing a relationship between the reference point and a three-dimensional position of the region of interest; Controlling the display of the composite image in which the region of interest is superimposed on the two-dimensional medical image, and varying the display mode of the region of interest in the composite image according to the second feature amount. The support device according to claim 1 .

3. The processor: obtaining a line of sight direction for projecting the three-dimensional path onto a two-dimensional plane; The first feature amount is a feature amount that represents a relative positional relationship between the reference point and the three-dimensional route in the line-of-sight direction. The support device according to claim 1 .

4. The processor: obtaining a line of sight direction for projecting the three-dimensional path onto a two-dimensional plane; The second feature amount is a feature amount that represents a relative positional relationship between the reference point and the three-dimensional position of the region of interest in the line of sight direction. The support device according to claim 2 .

5. The reference point is a point on the three-dimensional path. The support device according to any one of claims 1 to 4.

6. The reference point is the position of the medical instrument on the three-dimensional path. The support device according to claim 5.

7. the medical instrument is an endoscope equipped with a treatment tool, The reference point is the position of the treatment tool on the three-dimensional path. The support device according to claim 6.

8. the medical instrument is an endoscope equipped with a camera; The reference point is the position of the camera on the three-dimensional path. The support device according to claim 6.

9. The first feature amount is a distance in three-dimensional space between the reference point and the three-dimensional path. The support device according to claim 1 .

10. The reference point is the three-dimensional position of the region of interest. The support device according to claim 2 .

11. The processor: Acquire a noteworthy route on the three-dimensional route; In the control of displaying the composite image, the prominent route is highlighted in comparison with routes other than the prominent route on the three-dimensional route. The support device according to any one of claims 1 to 4.

12. The salient route is a route connecting the reference point and a point on the three-dimensional route whose distance from the region of interest is equal to or less than a threshold. The assistance device according to claim 11.

13. The noted route is a route that is registered in advance as preoperative information. The assistance device according to claim 11.

14. The path through which the medical device passes is the bronchial lumen. The support device according to any one of claims 1 to 4.

15. The two-dimensional medical image is an image obtained by radiography. The support device according to any one of claims 1 to 4.

16. the three-dimensional path is a bronchial lumen in a three-dimensional medical image; The two-dimensional medical image is a virtual radiological image generated from the three-dimensional medical image. The support device according to any one of claims 1 to 4.

17. The processor: acquiring a three-dimensional medical image including a path taken by the medical instrument; deforming the three-dimensional medical image by aligning the three-dimensional medical image with the two-dimensional medical image; Acquiring the three-dimensional path based on the three-dimensional medical image after deformation The support device according to any one of claims 1 to 4.

18. 1. An assist device for assisting insertion of a medical instrument into a tubular structure, the assist device comprising: at least one processor; Acquiring a two-dimensional medical image and a three-dimensional path that is three-dimensional information of a path along which the medical instrument passes; deriving a first feature amount representing a relationship between a reference point in a three-dimensional space and the three-dimensional path; Controlling the display of a composite image in which the three-dimensional path is superimposed on the two-dimensional medical image, and varying the display mode of the three-dimensional path in the composite image according to the first feature amount. A method to assist in carrying out the process.

19. An assist device for assisting insertion of a medical instrument into a tubular structure, the assist device comprising: Acquiring a two-dimensional medical image and a three-dimensional path that is three-dimensional information of a path along which the medical instrument passes; deriving a first feature amount representing a relationship between a reference point in a three-dimensional space and the three-dimensional path; Controlling the display of a composite image in which the three-dimensional path is superimposed on the two-dimensional medical image, and varying the display mode of the three-dimensional path in the composite image according to the first feature amount. A support program for executing the process.

Images