Image processing apparatus and image processing method

Abstract

To provide an image processor capable of more accurately controlling the display of an image instructed by a user.SOLUTION: An image processing apparatus includes an image acquisition unit that acquires a fluoroscopic image of an inside of a subject, a position detection unit 25 that acquires information indicating a position of a catheter inserted into the subject and represented in the fluoroscopic image, a line-of-sight recognition unit 30 that acquires information indicating a position of a viewpoint of a user with respect to the fluoroscopic image, and a system control unit 21 that controls display of the image on the basis of the information indicating the position of the catheter and the information indicating the position of the viewpoint.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method for controlling the display of an image of the inside of a subject. [Background technology]

[0002] X-ray imaging devices that capture images of the inside of a subject have a function to capture fluoroscopic images to assist in inserting an insert such as a catheter into the desired location. An operator such as a doctor guides the inserted catheter to the desired location while checking the fluoroscopic image displayed on the monitor, and then performs treatment at that location. In doing so, the operator may wish to move the position of the image displayed on the monitor or zoom in or out.

[0003] Patent Document 1 discloses a technique for moving a bed device or an enlarged image so that the tip position of a catheter is positioned at a predetermined position on a preset image. Patent Document 2 discloses a technique for gradually or continuously changing the display magnification of a displayed image according to the operator's line of sight, gestures, etc. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-150206 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-115965 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology of Patent Document 1 does not allow the operator to move the image to any position he or she desires. Also, the change in magnification described in Patent Document 2 may take time to reach the magnification desired by the user.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device that can more accurately control the display of an image designated by a user.

[0007] However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problems. Problems corresponding to the effects of the configurations shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the image processing device of the present invention has an image acquisition unit that acquires an image of the inside of a subject, a position acquisition unit that acquires information indicating the position of an insert inserted inside the subject and represented in the image, a viewpoint acquisition unit that acquires information indicating the position of the user's viewpoint relative to the image, and a control unit that controls the display of the image based on the information indicating the position of the insert and the information indicating the position of the viewpoint. [Effects of the Invention]

[0009] According to the present invention, it is possible to more accurately control the display of an image designated by a user. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the configuration of an X-ray imaging apparatus according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing a processing procedure of the X-ray imaging apparatus according to the first embodiment. [Figure 3] 3A and 3B are diagrams showing examples of display by the X-ray imaging apparatus according to the first embodiment. [Figure 4] FIG. 10 is a diagram showing a processing procedure of the X-ray imaging apparatus according to the second embodiment. [Figure 5] 10A and 10B are diagrams showing examples of display by the X-ray imaging apparatus according to the second embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of the configuration of an X-ray imaging apparatus according to a third embodiment. [Figure 7]FIG. 10 is a diagram showing a processing procedure of an X-ray imaging apparatus according to a third embodiment. [Figure 8] 10A and 10B are diagrams showing examples of display by the X-ray imaging apparatus according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described in detail.

[0012] (First embodiment) The configuration of an X-ray imaging device 1 according to the first embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of the X-ray imaging device 1 according to the first embodiment. As shown in FIG. 1, the X-ray imaging device 1 according to the first embodiment includes an imaging unit 100 and an image processing device 110. The imaging unit 100 includes a high-voltage generator 11, an X-ray tube 12, an X-ray diaphragm device 13, a tabletop 14, a C-arm 15, and an X-ray detector 16. The imaging unit 100 also includes a C-arm rotation / movement mechanism 17, a tabletop movement mechanism 18, a C-arm / tabletop mechanism control unit 19, and an diaphragm control unit 20. The imaging unit 100 also includes an image data generation unit 22 and an image data storage unit 23. The image processing device 110 also includes an image processing unit 24, a position detection unit 25, a display unit 27, a system control unit 21, and an input unit 26. The image processing device 110 includes a computer having a processor, a memory, and a communication device. The image processing device 110 functions as a system control unit 21, an image processing unit 24, a position detection unit 25, and a line of sight recognition unit 30 by the processor executing a program stored in a recording medium such as a memory (not shown).

[0013] The high voltage generator 11 generates a high voltage under the control of the system control unit 21 and supplies the generated high voltage to the X-ray tube 12. The X-ray tube 12 generates X-rays using the high voltage supplied from the high voltage generator 11. The X-ray diaphragm device 13, under the control of the diaphragm control unit 20, narrows down the X-rays generated by the X-ray tube 12 so that a region of interest of the subject P is selectively irradiated. For example, the X-ray diaphragm device 13 has four slidable diaphragm blades. Under the control of the diaphragm control unit 20, the X-ray diaphragm device 13 slides these diaphragm blades to narrow down the X-rays generated by the X-ray tube 12 and irradiate the subject P. The top board 14 is a bed on which the subject P rests, and is placed on a couch (not shown). Note that the subject P is not included in the imaging unit 100.

[0014] The X-ray detector 16 detects X-rays that have passed through the subject P. For example, the X-ray detector 16 has detection elements arranged in a matrix. Each detection element converts the X-rays that have passed through the subject P into an electrical signal, accumulates the electrical signal, and transmits the accumulated electrical signal to the image data generator 22.

[0015] The C-arm 15 holds the X-ray tube 12, the X-ray diaphragm device 13, and the X-ray detector 16. The X-ray tube 12, the X-ray diaphragm device 13, and the X-ray detector 16 are arranged by the C-arm 15 so as to face each other with the subject P in between.

[0016] The C-arm rotation / movement mechanism 17 is a mechanism for rotating and moving the C-arm 15, and the top movement mechanism 18 is a mechanism for moving the top 14. Under the control of the system control unit 21, the C-arm / top mechanism control unit 19 controls the C-arm rotation / movement mechanism 17 and the top movement mechanism 18 to adjust the rotation and movement of the C-arm 15 and the movement of the top 14. Under the control of the system control unit 21, the iris control unit 20 adjusts the opening of the iris blades of the X-ray iris device 13 to control the irradiation range of the X-rays irradiated onto the subject P.

[0017] The image data generating unit 22 generates image data using the electrical signals converted from the X-rays by the X-ray detector 16, and stores the generated image data in the image data storage unit 23. For example, the image data generating unit 22 performs current-voltage conversion, A (Analog) / D (Digital) conversion, and parallel-serial conversion on the electrical signals received from the X-ray detector 16 to generate image data.

[0018] The image data storage unit 23 stores the image data generated by the image data generation unit 22.

[0019] The image processing unit 24 performs various image processing on the image data stored in the image data storage unit 23. For example, the image processing unit 24 generates a photographed image (radiographic image) which is an X-ray transmission image as an image of the inside of the subject, using photographed data collected by irradiating the subject P with X-rays. The image processing unit 24 also generates a fluoroscopic image (radiographic image) as an image of the inside of the subject, using fluoroscopic data collected at a lower dose than when collecting the photographed data. Note that the fluoroscopic image is not necessarily stored by the image data storage unit 23. In addition, in this embodiment, the photographed image and the fluoroscopic image may be collectively referred to as an image. The image processing unit 24 also functions as an image acquisition unit that acquires an image of the inside of the subject P.

[0020] The position detection unit 25 is operated by the system control unit 21. The position detection unit 25 detects the position of a predetermined portion of a device, such as a catheter, that is inserted and moves within the subject, based on the fluoroscopic images generated by the image processing unit 24. The position detection unit 25 then converts the position into position coordinates on the display screen and sends the converted position coordinates to the system control unit 21. In other words, the position detection unit 25 functions as a position acquisition unit that acquires information indicating the position of an insert, such as a catheter, inserted inside the subject and displayed in an image (fluoroscopic image). Generally, a catheter can be distinguished from the subject due to its X-ray transparency and shape. Therefore, the position of the catheter tip is automatically determined by image recognition, and its position in the image coordinates is detected. Alternatively, an easily identifiable marker may be attached to the catheter tip in advance. In this embodiment, the detection target is the tip position of the catheter inserted into the subject, but any position on the insert may be predetermined.

[0021] The input unit 26 receives various instructions from an operator (user) such as a doctor or technician who operates the X-ray diagnostic apparatus 1. For example, the input unit 26 includes a mouse, a keyboard, buttons, a trackball, a joystick, a foot switch, a microphone, an acceleration sensor, etc. The input unit 26 transfers instructions received from the operator (user) to the system control unit 21. In other words, the input unit may be configured to receive instructions not only through physical input operations from the user, but also through voice or gestures.

[0022] The input unit 26 has a plurality of buttons to which various functions (e.g., image playback, enlargement, 3D image rotation, fluoroscopy, photography, image saving, etc.) are assigned, and accepts various instructions when the operator presses each button, for example.

[0023] The input unit 26 includes a foot switch placed at the operator's feet, receives instructions from the operator, and transfers them to the system control unit 21. The input unit 26 also includes a microphone that receives voice information uttered by the operator, and transfers the voice information to the system control unit 21. The input unit 26 also includes an acceleration sensor that detects body movements of the operator, such as the head and feet, and transfers the detected acceleration to the system control unit 21. At least one of the button, foot switch, microphone, and acceleration sensor is used together with line-of-sight information when controlling fluoroscopic images, and details will be described later.

[0024] The display unit 27 displays a GUI (Graphical User Interface) for receiving instructions from an operator and images such as radiographic images and fluoroscopic images. For example, the display unit 27 has a plurality of monitors, each of which displays a radiographic image, a fluoroscopic image, etc. Furthermore, for example, each of the plurality of monitors of the display unit 27 is set as a monitor dedicated to fluoroscopic images or a monitor dedicated to radiographic images.

[0025] The system control unit 21 controls the overall operation of the X-ray diagnostic apparatus 1. For example, the system control unit 21 controls the high-voltage generator 11 according to instructions from the operator transferred from the input unit 26, and adjusts the voltage supplied to the X-ray tube 12, thereby controlling the amount of X-rays irradiated to the subject P and ON / OFF. Also, for example, the system control unit 21 controls the C-arm / top mechanism control unit 19 according to instructions from the operator, and adjusts the rotation and movement of the C-arm 15 and the movement of the top 14. Also, for example, the system control unit 21 controls the iris control unit 20 according to instructions from the operator, and adjusts the opening of the iris blades of the X-ray iris device 13, thereby controlling the irradiation range of the X-rays irradiated to the subject P.

[0026] Furthermore, the system control unit 21 controls the image data generation process by the image data generation unit 22, the image processing or analysis process by the image processing unit 24, etc., in accordance with instructions from the operator. The system control unit 21 also controls the detection of a predetermined position of the device by the position detection unit 25. The system control unit 21 also controls the monitor of the display unit 27 to display a GUI for receiving instructions from the operator and images (e.g., photographed images and fluoroscopic images) stored in the image data storage unit 23. The system control unit 21 also controls the image processing unit 24 in accordance with line-of-sight information from the line-of-sight recognition unit 30 and position information of the catheter tip from the position detection unit 25, and performs display changes such as switching between various modes and moving and enlarging fluoroscopic images. In other words, the system control unit 21 also functions as a control unit that controls the display of images.

[0027] The gaze recognition unit 30 acquires gaze information of an operator such as a doctor or engineer who operates the X-ray diagnostic apparatus 1. Specifically, the gaze recognition unit 30 acquires information indicating where the operator's gaze is directed on the X-ray diagnostic apparatus 1. For example, the gaze recognition unit 30 may be a device having a camera or the like for capturing an image of the operator, or the function of the gaze recognition unit 30 may be a function within an image processing device that receives a signal from a device having a camera or the like. The gaze recognition unit 30 uses the intersection of the plane of the display surface of the monitor of the display unit 27 and the gaze as a viewpoint, converts the intersection into point coordinates in an arbitrary two-dimensional coordinate system stretched on the display surface, and transmits information indicating the position of the viewpoint (viewpoint position) to the system control unit 21.

[0028] For example, the gaze recognition unit 30 detects the doctor's eye position and gaze (gaze direction) from information acquired by a camera, and detects the doctor's viewpoint (point of gaze) on the monitor of the display unit 27. For example, a two-dimensional (x-axis, y-axis) coordinate system is pre-defined on the display surface of the monitor. The gaze recognition unit 30 detects the doctor's gaze from information acquired from the camera and calculates the viewpoint, which is the intersection of the detected gaze and the display surface of the monitor. Then, the gaze recognition unit 30 refers to the two-dimensional coordinate information on the display surface of the monitor, calculates the coordinates of the intersection on the coordinate system defined by the x-axis and y-axis, and transmits this information (viewpoint position) indicating the viewpoint position to the system control unit 21. Note that any conventional method can be used to detect the gaze, such as the limbus tracking method (scleral reflex method), which measures eye movement by utilizing the difference in light reflectance between the sclera (white of the eye) and the cornea (black of the eye). The gaze recognition unit 30 may be, for example, a device installed near the monitor, or a wearable terminal device such as glasses. In this way, the line-of-sight recognition unit 30 functions as a viewpoint acquisition unit that acquires information (viewpoint position) indicating the position of the user's viewpoint relative to the image.

[0029] Fig. 2 is a flowchart executed by the system control unit 21 of the first embodiment. Fig. 3 is an explanatory diagram of a fluoroscopic image displayed on a monitor, executed by the system control unit 21 of the first embodiment. The operation of the system control unit 21 of the first embodiment will be described below with reference to the flowchart of Fig. 2 and the explanatory diagram of Fig. 3.

[0030] In Fig. 3, the fluoroscopic image displayed on the monitor shows blood vessels and a catheter inserted into the subject. The viewpoint of the operator watching the monitor is indicated by a circular icon E. The operator can switch between various modes (to be described later) and change the display magnification of the fluoroscopic image by moving the line of sight to any position on the fluoroscopic image and giving instructions to a predetermined input unit. The modes in the first embodiment are three: a fluoroscopic image movement mode, an enlargement mode, and a reduction mode.

[0031] Figure 3(a) shows the display on the monitor when the operator is viewing a fluoroscopic image. Catheter position C indicates the tip of the catheter. Display information I indicates the selected mode and the display magnification of the fluoroscopic image.

[0032] In step S201, the system control unit 21 determines whether or not an operator has given an instruction via the input unit 26. Here, the operator can give an input instruction by any one of the following methods: pressing a button, stepping on a foot switch, speaking a predetermined word into a microphone, or moving the body near where the acceleration sensor is attached. The system control unit 21 may measure the time the viewpoint position remains stationary and determine that an operator has given an instruction if the viewpoint position remains stationary for a predetermined period of time. If an operator has given an instruction, the system control unit 21 proceeds to step S202; if an operator has not given an instruction, the system control unit 21 repeats the processing of step S201.

[0033] In step S202, the position detection unit 25 detects the catheter position, which is the tip of the catheter, at the time when the operator gives an instruction based on the fluoroscopic image, converts it into coordinates on the display screen, and transmits it to the system control unit 21.

[0034] In step S203, the line-of-sight recognition unit 30 acquires the line-of-sight information of the operator at the time when the operator gives an instruction, converts the line-of-sight position, which is information on the position of the operator's line of sight, into coordinates on the display surface, and transmits it to the system control unit 21.

[0035] In this embodiment, in order to reduce the calculation load on the system control unit 21, the catheter position is detected in step S202 and the viewpoint position is detected in step S203 after an instruction from the operator is received in step S201. If the calculation processing speed of the system control unit 21 is high, the catheter position and the viewpoint position may be detected constantly to determine whether an instruction from the operator has been received.

[0036] In step S204, the system control unit 21 calculates the distance between the coordinates of the catheter position on the display surface and the coordinates of the viewpoint position, and determines whether or not it is equal to or less than a predetermined distance (equal to or less than a threshold). Here, the predetermined distance as the threshold is not particularly limited, but may be, for example, the radius of the circular icon E indicating the viewpoint position. If the coordinates of the catheter position and the coordinates of the viewpoint position are equal to or less than the predetermined distance (equal to or less than the threshold), the system control unit 21 proceeds to step S205, and if they exceed the predetermined distance, the system control unit 21 proceeds to step S210.

[0037] In step S205, the system control unit 21 sequentially switches between the three modes of the first embodiment, namely, the movement mode, the enlargement mode, and the reduction mode of the fluoroscopic image. Fig. 3(b) is a diagram showing the state after the mode switching. As shown in Fig. 3(b), the coordinates of the catheter position C and the coordinates of the viewpoint position E are approximately the same, and the display information I switches from the reduction mode in Fig. 3(a) to the movement mode.

[0038] By adopting such a configuration, the operator can easily switch the display change mode of the fluoroscopic image without taking his / her eyes off the fluoroscopic image, as compared to when a mode switching icon is provided outside the fluoroscopic image.

[0039] In addition, when the detection of the catheter position and the viewpoint position is constantly being performed as described above, before the mode is switched, it may be possible to display a message indicating that the mode will be switched when the coordinates of the catheter position and the coordinates of the viewpoint position become equal to or smaller than a predetermined distance (threshold value). For example, around the circular icon E indicating the viewpoint position, the characters for the mode to be switched to, namely, the move mode, enlarge mode, or reduce mode, may be displayed.

[0040] In step S210, the system control unit 21 determines whether the selected mode is the movement mode. If it is the movement mode, the process proceeds to step S211, and if it is not the movement mode, the process proceeds to step S220.

[0041] In step S211, the system control unit 21 calculates a vector from the coordinates of the catheter position on the display surface to the coordinates of the viewpoint position, and moves the fluoroscopic image in accordance with this vector. In other words, the system control unit 21 moves the position shown in the fluoroscopic image based on the position of the inserted object (catheter) and the viewpoint position (the distance and direction of the line connecting the viewpoint position). FIG. 3(c) is a diagram at the time when the operator gives an instruction to the input unit 26. As shown in FIG. 3(c), the system control unit 21 calculates a vector V with the viewpoint position E as its start point and the catheter position C as its end point. FIG. 3(d) is a diagram in which the fluoroscopic image has been moved in accordance with the vector V, and the blood vessel and catheter have moved from the display surface. Note that the icon E is in the same position as in FIG. 3(c), since it is the operator's viewpoint position. In this way, the operator can arbitrarily determine the position to which the fluoroscopic image will move without taking his or her eyes off the fluoroscopic image.

[0042] As shown in FIG. 3(c), when the displayed fluoroscopic image is enlarged at a predetermined magnification rate, the position of the enlarged fluoroscopic image is moved. On the other hand, when an unenlarged fluoroscopic image, which is an image of the same range as the field of view of the X-ray detector 16, is displayed on the display screen, the tabletop 14, C-arm 15, etc. are moved. Furthermore, when the displayed fluoroscopic image is enlarged at a predetermined magnification rate but the fluoroscopic image after movement exceeds the field of view of the X-ray detector 16, the tabletop 14, C-arm 15, etc. may be moved. Furthermore, if the catheter extends beyond the displayed fluoroscopic image after the fluoroscopic image is moved and is not displayed on the screen, there is a risk that subsequent catheter operation will be hindered. As shown in FIG. 3(c), the system control unit 21 calculates a distance Ld between the edge of the display screen, which is in the same direction as the vector V, and the coordinates of the catheter position. If the distance Ld is smaller than the distance between the coordinates of the catheter position and the coordinates of the viewpoint position, the fluoroscopic image may be moved by the distance Ld.

[0043] In step S220, the system control unit 21 determines whether the selected mode is the enlargement mode. If it is the enlargement mode, the process proceeds to step S221. If it is not the enlargement mode, the process proceeds to step S222, where the reduction mode is selected.

[0044] In step S221, the system control unit 21 calculates the distance between the coordinates of the catheter position on the display screen and the coordinates of the viewpoint position, and enlarges the fluoroscopic image. FIG. 3(e) shows the enlargement mode, illustrating the state at the time when the operator issues an instruction to the input unit 26. The system control unit 21 calculates the distance L1 between the catheter position C and the viewpoint position E, as shown in FIG. 3(e). The system control unit 21 calculates the ratio Le / L1 of the calculated distance L1 to a predetermined value Le, and determines the enlarged display magnification by multiplying the ratio by the current display magnification as the enlargement factor. For example, if the ratio Le / L1 of the distance L1 to the predetermined value Le in FIG. 3(e) is 2, the enlarged display magnification will be 150% × 2 = 300%. FIG. 3(f) shows the display magnification changed based on the ratio Le / L1, and the fluoroscopic image is enlarged to 300%.

[0045] In this way, the operator can arbitrarily determine the magnification ratio without moving his / her line of sight from the fluoroscopic image. Furthermore, by determining the magnification ratio from the current display magnification based on the ratio Le / L1 between the distance L1 and the predetermined value Le, the smaller the distance L1, the larger the magnification ratio becomes, and vice versa. This allows the operator to increase the magnification ratio by positioning his / her line of sight closer to the catheter when he / she wishes to greatly enlarge the area around the catheter he / she is focusing on. Furthermore, when he / she wishes to slightly enlarge the entire fluoroscopic image, he / she can decrease the magnification ratio by positioning his / her line of sight further away from the catheter. Note that the fluoroscopic image is not enlarged when the ratio Le / L1 between the distance L1 between the catheter position and the viewpoint position and the predetermined distance Le is equal to or less than 1.

[0046] The catheter position C in Fig. 3(f) after magnification is the same as in Fig. 3(e) before magnification. This allows the operator to fix the position of the catheter while manipulating it, and to work efficiently without having to search for the catheter after magnification.

[0047] The predetermined value Le may be arbitrarily set by the operator as an initial value. For example, if a large magnification is desired even when the line of sight is far away from the catheter position, the predetermined value Le is set to a large value.

[0048] Furthermore, when the catheter position and the viewpoint position are constantly detected as described above, the display magnification, which will change depending on the viewpoint position, may be displayed before the fluoroscopic image is enlarged. For example, a number representing the planned display magnification, calculated by multiplying the current display magnification by the ratio Le / L1 of the distance L1 and a predetermined value Le, is displayed around the circular icon E indicating the viewpoint position. In this way, the operator can know the intended display magnification in advance and can determine the display magnification while adjusting the line of sight.

[0049] In step S222, the system control unit 21 calculates the distance between the coordinates of the catheter position on the display screen and the coordinates of the viewpoint position, and reduces the fluoroscopic image. FIG. 3(g) is a diagram illustrating the reduction mode, showing the state at the time when the operator issues an instruction to the input unit 26. The system control unit 21 calculates the distance L2 between the catheter position C and the viewpoint position E, as shown in FIG. 3(g). The system control unit 21 calculates the ratio Lm / L2 of the calculated distance L2 to a predetermined value Lm, and determines the display magnification after reduction by multiplying the current display magnification by this ratio as the reduction rate. For example, if the ratio Lm / L2 of the distance L2 to the predetermined value Lm in FIG. 3(g) is 2 / 3, the display magnification after reduction is 150% × 2 / 3 = 100%. FIG. 3(h) is a diagram showing the display magnification changed based on the ratio Lm / L2, where the fluoroscopic image is reduced to 100%.

[0050] In this way, the operator can arbitrarily determine the reduction ratio without moving his / her line of sight from the fluoroscopic image. Furthermore, by determining the reduction ratio from the current display magnification based on the ratio Lm / L2 between the distance L2 and the predetermined value Lm, the reduction ratio decreases as the distance L2 decreases, and increases as the distance L2 increases. This allows the operator to set his / her line of sight closer to the catheter when he / she wants to reduce the area around the catheter he / she is focusing on, thereby achieving a small reduction ratio. Furthermore, when he / she wants to reduce the area around the catheter in order to view the entire fluoroscopic image, he / she can set his / her line of sight further away from the catheter, thereby achieving a large reduction ratio. Note that the fluoroscopic image is not reduced when the ratio Lm / L2 between the distance L2 between the catheter position and the viewpoint position and the predetermined value Lm is 1 or greater.

[0051] The catheter position C in Figure 3(h) after reduction is the same as that in Figure 3(g) before reduction. This allows the operator to fix the position of the catheter while manipulating it, and to work efficiently without having to search for the catheter after reduction.

[0052] The predetermined value Lm may be set arbitrarily by the operator as an initial value. For example, if a large reduction ratio is desired even when the line of sight is far away from the catheter position, the predetermined value Lm is set to a small value.

[0053] Furthermore, when the catheter position and the viewpoint position are constantly detected as described above, the display magnification that will change depending on the viewpoint position may be displayed before the fluoroscopic image is reduced. For example, a number representing the intended display magnification obtained by multiplying the current display magnification by the ratio Lm / L2 of the distance L2 and a predetermined value Lm is displayed around the circular icon E indicating the viewpoint position. In this way, the operator can know the intended display magnification in advance and can determine the display magnification while adjusting the position of the line of sight.

[0054] In step S230, the system control unit 21 ends this process if the operator inputs an instruction to end this process or if the conditions for ending this process are satisfied. On the other hand, if the instruction to end this process has not been input and the conditions for ending this process have not been satisfied, the system control unit 21 proceeds to step S201.

[0055] According to the first embodiment, the operator can move, enlarge, or reduce the fluoroscopic image in accordance with the direction and distance from the viewpoint to the catheter position without moving his or her line of sight from the fluoroscopic image, which allows for more accurate control of the image display as instructed by the user.

[0056] (Second embodiment) In the first embodiment, a case where three modes, namely, movement of a fluoroscopic image, enlargement, and reduction, are switched between is described. In the second embodiment, a case where two modes, namely, movement of a fluoroscopic image and enlargement / reduction, which can execute either enlargement or reduction, are switched between is described. Note that, compared to the first embodiment, the X-ray imaging apparatus according to the second embodiment has processing content that is partially different from the processing content described in the flowchart of FIG. 2. The following description will focus on the parts where processing is different, and the same components as those in the first embodiment will be assigned the same reference numerals and detailed description will be omitted.

[0057] Fig. 4 is a flowchart executed by the system control unit 21 of the second embodiment, and differs in the processes corresponding to S220, S221, and S222 in Fig. 2. Fig. 5 is an explanatory diagram of a fluoroscopic image displayed on a monitor, executed by the system control unit 21 in the second embodiment. The operation of the system control unit 21 of the second embodiment will be described below with reference to the flowchart of Fig. 4 and the explanatory diagram of Fig. 5.

[0058] 5, the fluoroscopic image displayed on the monitor shows blood vessels and a catheter inserted into the subject. The operator can change the display magnification of the fluoroscopic image, which will be described later, by moving the viewpoint to any position on the fluoroscopic image and giving instructions to a predetermined input unit 26. As described above, the second embodiment has two modes: a fluoroscopic image movement mode and an enlargement / reduction mode.

[0059] In step S320, the system control unit 21 calculates the direction of catheter travel based on the coordinates of the catheter position continuously transmitted from the position detection unit 25. FIG. 5(a) is a diagram illustrating the enlargement / reduction mode, showing the state at the time when the operator issues an instruction to the input unit 26. The system control unit 21 calculates the angle θ between the direction of catheter travel D at the time when the operator issues the instruction and a line connecting the coordinates of the catheter position C and the coordinates of the viewpoint position E. The system control unit 21 determines whether the angle θ is equal to or smaller than a predetermined angle. If it is equal to or smaller than the predetermined angle, the process proceeds to step S321, where the image is enlarged. If it is not equal to or smaller than the predetermined angle, the process proceeds to step S322, where the image is reduced. Note that the predetermined angle is preferably 90 degrees, which is either in the same direction as or opposite to the direction of catheter travel, so that the operator can easily determine the viewpoint when enlarging or reducing the fluoroscopic image. However, the operator may be allowed to set any angle.

[0060] In step S321, the system control unit 21 calculates the distance between the catheter position coordinates on the display screen and the line-of-sight coordinates, and enlarges the fluoroscopic image. The system control unit 21 calculates the distance L3 between the catheter position C and the viewpoint position E, as shown in FIG. 5(a). The system control unit 21 calculates the ratio Le / L3 of the calculated distance L3 to a predetermined value Le, and determines the enlarged display magnification by multiplying the ratio by the current display magnification. For example, if the ratio Le / L3 of the distance L3 to the predetermined value Le in FIG. 5(a) is 2, the enlarged display magnification is 150% × 2 = 300%. FIG. 5(b) shows the display magnification changed based on the ratio Le / L1, and the fluoroscopic image is enlarged to 300%. Note that the catheter position C in the enlarged image in FIG. 5(b) is in the same position as in FIG. 5(a) before enlargement, as in the first embodiment.

[0061] In step S322, the system control unit 21 calculates the distance between the coordinates of the catheter position and the coordinates of the viewpoint position on the display screen and reduces the fluoroscopic image. FIG. 5(c) is a diagram illustrating the scaling mode, showing the state at the time when the operator issues an instruction to the input unit 26, but is shown as an example different from FIG. 5(a). As shown in FIG. 5(c), the angle θ between the line connecting the catheter's direction of travel D, the coordinates of the catheter position C, and the coordinates of the viewpoint position E at the time when the operator issues the instruction exceeds 90 degrees. The system control unit 21 calculates the distance L4 between the catheter position C and the viewpoint position E as shown in FIG. 5(c). The system control unit 21 calculates the ratio Lm / L4 of the calculated distance L4 to a predetermined value Lm, and multiplies the ratio by the current display magnification as a reduction ratio to determine the display magnification after reduction. For example, if the ratio Lm / L4 of the distance L4 to the predetermined value Lm in FIG. 5(c) is 2 / 3, the display magnification after reduction is 150% × 2 / 3 = 100%. Fig. 5(d) shows the fluoroscopic image after changing the display magnification based on the ratio Lm / L2, and the fluoroscopic image is reduced to 100%. Note that the catheter position C in Fig. 5(d) after reduction is at the same position as in Fig. 5(c) before enlargement, as in the first embodiment.

[0062] Before enlarging or reducing a perspective image, the display magnification that will change depending on the viewpoint position may be displayed. For example, a number representing the planned enlargement display magnification, calculated by multiplying the current display magnification by the ratio Le / L3 of the distance L3 to a predetermined value Le, or a number representing the planned reduction display magnification, calculated by multiplying the current display magnification by the ratio Lm / L4 of the distance L4 to a predetermined value Lm, may be displayed around a circular icon E indicating the viewpoint position. In this way, the operator can know in advance whether the viewpoint position is enlarged or reduced and the intended display magnification, and can determine the display magnification while adjusting the viewpoint position.

[0063] According to the second embodiment, the operator can select one mode without taking his / her line of sight off the fluoroscopic image, thereby enlarging or reducing the fluoroscopic image according to the direction of catheter movement and the viewpoint position.

[0064] (Third embodiment) In the second embodiment, a case where a fluoroscopic image is enlarged or reduced in size in accordance with the catheter traveling direction and viewpoint position, and the distance between the catheter position and the viewpoint position, is described. In this embodiment, a case where a fluoroscopic image is enlarged or reduced in size in accordance with the catheter traveling direction and viewpoint position, and the catheter position, viewpoint position, and position of a specific region (target position) is described. In the following description, the same components as those in the first and second embodiments are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0065] FIG. 6 is a diagram showing an example of the configuration of an X-ray diagnostic apparatus 400 according to the third embodiment. The X-ray diagnostic apparatus 400 according to the third embodiment includes a target position identifying unit 40. The target position identifying unit 40 detects a lesion from a fluoroscopic image generated by the image processing unit 24 using a known computer-aided diagnosis (CAD) algorithm. Then, the target position identifying unit 40 identifies the center of gravity or the like of the detected lesion as a target position. A detection method using CAD may include a method using a machine learning model that has been trained to detect lesions. The lesion detection method by the target position identifying unit 40 is not limited to this, and any method such as template matching may be used. The target position identifying unit 40 may also identify a position specified by the operator using the input unit 26 in a displayed fluoroscopic image as the target position. The target position identifying unit 40 may also be referred to as a target acquiring unit that acquires information indicating a target position (the position of a specific part) within a subject.

[0066] Fig. 7 is a flowchart executed by the system control unit 21 of the third embodiment. Steps S502 to S511 and S530 in the flowchart of Fig. 7 are similar to the processes of steps S201 to S211 and S230 in Fig. 4, so their explanation will be omitted, and the processes of steps S501 and S520 to S522 will be explained. Fig. 8 is an explanatory diagram of a fluoroscopic image displayed on a monitor, executed by the system control unit 21 of the third embodiment. The operation of the system control unit 21 of the third embodiment will be explained below with reference to the flowchart of Fig. 7 and the explanatory diagram of Fig. 8.

[0067] 8, the fluoroscopic image displayed on the monitor shows the blood vessels and the catheter inserted into the subject. The lesion detected by the target position specifying unit 40 is shown as a figure G that matches its shape, and may be distinguished from the blood vessels by adding a line or color.

[0068] The operator can change the display magnification of the fluoroscopic image, which will be described later, by moving the line of sight to any position on the fluoroscopic image and giving an instruction to a predetermined input unit 26. As in the second embodiment, the third embodiment also has two modes: a fluoroscopic image movement mode and an enlargement / reduction mode.

[0069] In step S501, the target position specifying unit 40 detects a lesion from the fluoroscopic image, converts the center of gravity position of the lesion or the like into coordinates on the display surface as a target position, and transmits them to the system control unit 21 and the image processing unit 24. The image processing unit 24 applies a line or color to the detected lesion and displays it on the display unit 27.

[0070] In step S520, the system control unit 21 calculates the angle of the line connecting the coordinates of the viewpoint position and the coordinates of the target position from the coordinates of the catheter position on the display screen. FIG. 8(a) is a diagram showing the state in the enlargement / reduction mode at the time when the operator has given an instruction to the input unit 26. The system control unit 21 calculates the angle θ of the line connecting the coordinates of the viewpoint position E and the coordinates of the target position G from the coordinates of the catheter position C at the time when the operator gave the instruction. The system control unit 21 determines whether the angle θ is equal to or less than a predetermined angle. If it is equal to or less than the predetermined angle, the process proceeds to step S521 for enlargement. If it is not equal to or less than the predetermined angle, the process proceeds to step S522 for reduction. Note that the predetermined angle is preferably 90 degrees, which is either in the same direction or the opposite direction from the catheter position to the target position, so that the operator can easily determine the position to place their gaze when enlarging or reducing the fluoroscopic image. Alternatively, the operator may set any angle.

[0071] In step S521, the system control unit 21 enlarges the fluoroscopic image based on three positions on the display screen: the coordinates of the catheter position, the coordinates of the viewpoint position, and the coordinates of the target position. The system control unit 21 calculates the length of each side of a triangle connecting the three points of the catheter position C, the viewpoint position E, and the target position G, and the position of the center of gravity X, as shown in Figure 8(a). The system control unit 21 calculates the ratio Le / L5 of the length L5 of the longest side of the calculated three sides to a predetermined value Le, and multiplies this ratio by the current display magnification as the magnification factor to determine the display magnification after enlargement.

[0072] For example, in FIG. 8(a), if side a connecting catheter position C and viewpoint position E is the longest and the ratio Le / L5 of the length L5 of side a to the predetermined value Le is 1.5, the display magnification after enlargement will be 150% × 1.5 = 225%. FIG. 8(b) shows the display magnification changed based on the ratio Le / a, and the fluoroscopic image is enlarged to 225%. The fluoroscopic image is also moved so that the calculated center of gravity is at the center of the display screen. The center of gravity X in FIG. 8(a) is located at the center of the display screen in the enlarged view of FIG. 8(b). The centers of gravity of the three points—the catheter, the lesion, and the viewpoint position—that the operator always focuses on are positioned near the center of the display screen after enlargement, and the aforementioned three points are moved near the center of the display screen, improving the operator's visibility. However, if the catheter or the lesion extends beyond the fluoroscopic image and is not displayed on the screen after enlargement, a warning may be displayed to prompt the operator to change the viewpoint position without enlarging. Further, as in the first and second embodiments, the enlargement may be performed while the catheter position C is fixed so as not to change before and after the enlargement.

[0073] In step S522, the system control unit 21 reduces the fluoroscopic image based on three positions on the display screen: the catheter position coordinates, the viewpoint position coordinates, and the target coordinates. Fig. 8(c) is a diagram at the time when the operator has given instructions to the input unit 26, and is shown as an example different from Fig. 8(a). The system control unit 21 calculates the lengths of each side and the center of gravity of the triangle connecting the three points of the catheter position C, the viewpoint position E, and the target position G, as shown in Fig. 8(c). The system control unit 21 calculates the ratio Lm / L6 of the length L6 of the longest side of the calculated three sides to a predetermined value Lm, and determines the display magnification after reduction by multiplying this ratio by the current display magnification as an enlargement factor.

[0074] For example, in FIG. 8(c), if side b connecting the catheter position and the line of sight position is the longest and the ratio Lm / L6 of the length L6 of side b to the predetermined value Lm is 2 / 3, the display magnification after reduction will be 150% × 2 / 3 = 100%. FIG. 8(d) shows the display magnification changed based on the ratio Lm / L6, and the fluoroscopic image is reduced to 100%. The fluoroscopic image is also moved so that the calculated center of gravity X is at the center of the display surface. The center of gravity X in FIG. 8(c) is located at the center of the display surface in FIG. 8(d) after reduction. The centers of gravity of the three points—the catheter, the lesion, and the gaze point—that the operator always focuses on are positioned near the center of the display surface after reduction, and the three points are moved near the center of the display surface, improving the operator's visibility. Note that, as in the first and second embodiments, the catheter position C may be fixed and not change before and after reduction when performing the enlargement.

[0075] According to the third embodiment, the operator can select one mode without taking his / her line of sight from the fluoroscopic image, thereby enlarging or reducing the fluoroscopic image according to the catheter position, viewpoint position, and target position.

[0076] Although each embodiment has been described above, the image processing device of the present invention is not limited to these embodiments, and the present invention also includes an image processing device that combines the functions of each embodiment or that omits non-essential elements.

[0077] Furthermore, in the above embodiment, an example has been described in which the imaging unit 100 and the image processing device 110 are separate, but the image processing device 110 may be integrated with the imaging unit 100 .

[0078] In addition, in each embodiment, a catheter is cited as an example of an inserted object to be detected by the position detection unit 25, but the present invention is not limited to this, and the object to be detected may be a balloon catheter, a stent, a guide wire, or the like, which is a device that moves inside the subject. In other words, the inserted object may be an instrument used for treatment or examination such as surgery, which is inserted into the body of the subject.

[0079] Furthermore, in the figures used to explain each embodiment, a screen on which only a fluoroscopic image is displayed is shown for the sake of simplicity of explanation, but information other than the fluoroscopic image, such as an image from an external input video device or a GUI other than the GUI for changing the magnification ratio, may also be displayed on this screen. [Explanation of symbols]

[0080] 21 System control unit 24 Image processing section 25 Position detection unit 30 Gaze recognition section 110 Image processing device

Claims

1. an image acquisition unit that acquires an image of the inside of a subject; a position acquisition unit that acquires information indicating a position of an insert that is inserted inside the subject and is displayed in the image; a viewpoint acquisition unit that acquires information indicating a position of a viewpoint of a user relative to the image; and a control unit that controls display of the image based on information indicating the position of the insert and information indicating the position of the viewpoint.

2. The image processing device described in claim 1, further comprising an input unit into which user instructions are input, and wherein when the instruction is input into the input unit, the control unit controls the display of the image based on information indicating the position of the insert and information indicating the position of the viewpoint.

3. The image processing device described in claim 2, characterized in that the control unit has a first mode for moving the position shown in the image and a second mode for changing the display magnification of the image, and switches the mode to be executed when the distance between the position of the insert and the position of the viewpoint when the instruction is input to the input unit is less than a threshold.

4. The image processing device according to claim 1 , wherein the control unit moves the position shown in the image based on the distance and direction of a line segment connecting the position of the insert and the position of the viewpoint.

5. The image processing device according to claim 1 , wherein the control unit changes the display magnification of the image based on the distance between the position of the insert and the position of the viewpoint.

6. The image processing device described in claim 1, characterized in that the control unit calculates the direction of travel of the insert based on the position of the insert, and changes the display magnification of the image based on the calculated direction of travel and the position of the viewpoint.

7. The image processing device described in claim 6, characterized in that the control unit increases the display magnification of the image when the angle between the line connecting the position of the insert and the position of the viewpoint and the direction of travel is less than a predetermined angle, and decreases the display magnification of the image when the angle is greater than the predetermined angle.

8. 2. The image processing device according to claim 1, further comprising a target acquisition unit that acquires information indicating a target position within the subject, wherein the control unit further controls display of the image based on the information indicating the target position.

9. The image processing device described in claim 8, characterized in that the control unit increases the display magnification of the image when the angle between the line connecting the position of the insert and the position of the viewpoint and the line connecting the target position and the position of the viewpoint is less than a predetermined angle, and decreases the display magnification of the image when the angle is smaller than the predetermined angle.

10. 2. The image processing apparatus according to claim 1, wherein a predetermined position on the insert is used as the position of the insert.

11. 3. The image processing apparatus according to claim 2, wherein the input unit receives instructions from a user using any one of a physical input operation, a voice, and a gesture.

12. 11. The image processing apparatus according to claim 1, wherein the image is a radiographic image.

13. 13. The image processing apparatus according to claim 2, wherein the insert is a tool used for treating or examining the subject.

14. The image processing device of claim 13 , wherein the insert comprises at least one of a catheter, a balloon catheter, a stent, and a guidewire.

15. acquiring an image of the interior of the object; obtaining information indicating the position of an insert inserted inside the subject and represented in the image; obtaining information indicating a position of a user's viewpoint relative to the image; and controlling the display of the image based on the information indicating the position of the insert and the information indicating the position of the viewpoint.

16. The image processing device The function of acquiring images of the interior of the subject; obtaining information indicating the position of an insert inserted inside the subject and represented in the image; a function of acquiring information indicating the position of a user's viewpoint relative to the image; a function of controlling display of the image based on information indicating the position of the insert and information indicating the position of the viewpoint; A program that executes the following.

17. A recording medium on which the program according to claim 16 is recorded.

Images