Image processing device, method and program

Abstract

Image processing device containing a parameter calculation unit that calculates a parameter value relating to an endoscope operation for advancing an endoscope along a predetermined course or path within a luminal structure from a three-dimensional image that specifies the luminal structure. wherein an advance and retraction process in an axial direction, a rotation process about an axis and a process of bending a distal end section are only possible within a plane that contains an axis of the distal end section with the endoscope, and The parameter calculation unit defines a clockwise or counterclockwise direction around the axis as positive, the other direction as negative, and calculates a rotation angle of the endoscope due to the rotation process at any position on the course as the parameter value with respect to the rotation process, wherein the parameter calculation unit calculates the rotation angle of the endoscope due to the rotation process at the arbitrary position such that a sum of a cumulative total angle of rotation angles of the endoscope due to a rotation process performed until the distal end section arrives at the arbitrary position after the distal end section has been inserted into the luminal structure, and the rotation angle of the endoscope due to the rotation process at the arbitrary position falls within a predetermined angular range.

Description

1. Field of the invention

[0001] The present invention relates to an image processing apparatus, a method and a program in which a parameter value related to an endoscope operation for advancing an endoscope along a predetermined course in a luminal structure is calculated from a three-dimensional image indicating the luminal structure. 2. Description of the state of the art

[0002] In recent years, a technology for performing observation or treatment of a tubular structure such as a patient's colon or bronchi using an endoscope has attracted attention. JP2009-254837A discloses a technology for assisting an operator's operation of an endoscope by presenting various types of information for guiding a distal end portion of the endoscope to a desired position in a luminal structure. Further, JP2009-511155A discloses a technology for calculating upward, downward, rightward, and leftward bending angles of a distal end portion at each position on a course for advancing an endoscope along a desired course in a luminal structure in a case where an operation of bending the distal end portion in an up-down direction and a left-right direction is possible in the endoscope.

[0003] US 2011 / 0234780 A1 describes a medical device comprising an endoscopic insertion section provided with an image acquisition unit and a channel; a treatment unit provided with a sensor and a bending section and guided through a channel, wherein a distal end section of the treatment unit can protrude from a distal insertion section of the endoscopic insertion section, wherein the sensor is arranged in the distal end section and the bending section can bend the distal end section; a storage unit capable of storing three-dimensional image data; a target position setting unit adapted to set the target position based on the three-dimensional image data;a virtual endoscopic image generation unit adapted to generate a virtual endoscopic image using a line-of-sight parameter including a position, a direction, and a roll angle of the distal end portion detected by the sensor based on the three-dimensional image data; and an image processing unit adapted to perform an overlay process to thereby display operational information used to insert the distal end portion into the target position in overlay on the virtual endoscopic image.

[0004] US 8 454 497 B2 describes an endoscope device comprising an insertion section having a bending section provided at a distal end side thereof; a bending drive section that drives the bending of the bending section; an insertion shape detection section that detects an insertion shape of the distal end side of the insertion section as insertion shape information; a curved state detection section that detects a curved state based on the insertion shape information as curve information; a plane calculation section that performs calculation to estimate a plane for driving the bending of the bending section from the detected curve information; and a bending control section that drives the bending of the bending drive section along the estimated plane. Summary of the invention

[0005] Furthermore, in an endoscope such as a bronchial endoscope, in which a distal end portion bending operation is possible only within one plane, the endoscope cannot be advanced in a desired direction only by a distal end portion bending operation. In this case, it is necessary to align the one plane in which the distal end portion bending operation is possible with the desired direction by rotating the endoscope around an axis before the bending operation. In this case, the rotation direction (clockwise or counterclockwise) of the endoscope is usually selected to be a rotation direction in which the rotation required for one rotation operation is smaller. For example, in a case where the endoscope is rotated 50° clockwise, it is not a problem to advance the above-described endoscope even with a counterclockwise rotation of 130°.However, normally the rotation of 50° clockwise, in which the amount of rotation is smaller, is chosen.

[0006] However, with this method, rotation in one direction may continue in multiple rotations, and the endoscope may rotate in one direction beyond the range of motion of a wrist. In this case, a problem arises in that an operator must re-grip the endoscope, resulting in poor operability.

[0007] An object of the present invention is to provide an image processing apparatus, a method, and a program that reduce the difficulty of an operator in re-gripping an endoscope and improve operability in a case where the endoscope is an endoscope in which an operation of bending a distal end portion is possible only within a plane, in view of the above circumstances.

[0008] An image processing device of the present invention is an image processing device including a parameter calculation unit that calculates a parameter value related to an endoscope operation for advancing an endoscope along a predetermined course within a luminal structure from a three-dimensional image indicating the luminal structure, wherein an advancing and retracting operation in an axial direction, a rotating operation about an axis, and an operation of bending a distal end portion are possible only within a plane including an axis of the distal end portion in the endoscope, and the parameter calculation unit sets a clockwise direction or a counterclockwise direction around the axis as positive,sets the other direction as negative and calculates a rotation angle of the endoscope due to the rotation process at any position on the course as a parameter value related to the rotation process, wherein the parameter calculation unit calculates the rotation angle of the endoscope due to the rotation process at a position of a target such that a sum of a cumulative total angle of rotation angles of the endoscope due to a rotation process performed until the distal end portion arrives at the position of the target after the distal end portion has been inserted into the luminal structure and the rotation angle of the endoscope due to the rotation process at the position of the target falls within a predetermined angle range.

[0009] Here, the clockwise direction and the counterclockwise direction around the axis mean a clockwise direction and a counterclockwise direction when viewed from a base end side of an endoscope.

[0010] In the image processing apparatus of the present invention, the angle range is from -90° to +90°.

[0011] In the image processing apparatus of the present invention, the luminal structure may be a luminal organ of a human body, the angle range may be from -Ta° to +Tb°, where Ta and Tb are positive values, where Ta and Tb are positive values ​​of more than 90°, and when calculating the rotation angle of the endoscope due to the rotation operation at any position on the course as a parameter value related to the rotation operation, the parameter calculation unit may calculate a rotation angle at which a direction from a dorsal side to a ventral side of a human body or a direction from the ventral side to the dorsal side is closer to an upward direction of an image acquired by the endoscope after the rotation operation as a parameter value related to the rotation operation in a case where two rotation angles of the endoscope due to the rotation operation at the position of the target exist,that the sum falls within the angular range.,

[0012] Here, the upward direction of the image means an upward direction (bottom to top direction) in an up and down direction (vertical direction) of the image.

[0013] In the image processing apparatus of the present invention, the luminal structure may be a luminal organ of a human body, the angle range may be from -Ta° to +Tb°, where Ta and Tb are positive values, where Ta and Tb are positive values ​​of more than 90°, and in calculating the rotation angle of the endoscope due to the rotation operation at an arbitrary position on the course as a parameter value related to the rotation operation, the parameter calculation unit may calculate a rotation angle at which a direction from an inside to an outside of the human body is closer to an upward direction of an image taken by the endoscope after the rotation operation in a case where two rotation angles of the endoscope due to the rotation operation at the position of the target exist so that the sum falls within the angle range.

[0014] In the image processing apparatus of the present invention, the luminal structure may be a luminal organ of a human body, the angle range may be from -Ta° to +Tb°, where Ta and Tb are positive values, where Ta and Tb are positive values ​​of more than 90°, and when calculating the rotation angle of the endoscope due to the rotation operation at any position on the course as a parameter value related to the rotation operation, the parameter calculation unit may calculate a rotation angle at which a sum of absolute values ​​of the rotation angles at positions where respective rotation operations are performed from the position of a target to positions where one or more previous rotation operations are performed is smaller than a parameter value related to the rotation operation in a case where there are two rotation angles of the endoscope due to the rotation operation at the position of the target,that the sum described above falls within the angular range.,

[0015] The image processing apparatus of the present invention may include an image generation unit that applies the parameter value calculated by the parameter calculation unit to generate a virtual endoscopic image from the three-dimensional image.

[0016] The image processing apparatus of the present invention may include: a display unit that displays an image; and a display control unit that displays the parameter value corresponding to the position of the endoscope, calculated by the parameter calculation unit, on the display unit according to the position of the endoscope on the course when the endoscope is advanced along the course in the luminal structure.

[0017] An image processing method of the present invention is an image processing method for calculating a parameter value related to an endoscope operation for advancing an endoscope along a predetermined course within a luminal structure from a three-dimensional image indicating the luminal structure, wherein an advancing and retreating operation in an axial direction, a rotating operation about an axis, and an operation of bending a distal end portion are possible only within a plane including an axis of the distal end portion in the endoscope.be permitted, and a calculation process sets a clockwise direction or a counterclockwise direction around the axis as positive, sets the other direction as negative, and calculates a rotation angle of the endoscope due to the rotation operation at any position on the course as a parameter value related to the rotation operation, wherein the rotation angle of the endoscope due to the rotation operation at a position of a target is calculated such that a sum of a cumulative total angle of rotation angles of the endoscope due to a rotation operation performed until the distal end portion arrives at the position of the target after the distal end portion is inserted into the luminal structure, and the rotation angle of the endoscope due to the rotation operation at the position of the target falls within a predetermined angle range.

[0018] An image processing program of the present invention is an image processing program that causes a computer to execute a process of calculating a parameter value related to an endoscope operation for advancing an endoscope along a predetermined course within a luminal structure from a three-dimensional image indicating the luminal structure, wherein an advancing and retreating operation in an axial direction, a rotating operation around an axis, and an operation of bending a distal end portion are possible only within a plane including an axis of the distal end portion in the endoscope, and a calculation process includes: setting a clockwise direction or a counterclockwise direction around the axis as positive,Setting the other direction as negative and calculating a rotation angle of the endoscope due to the rotation process at any position on the course as a parameter value related to the rotation process, wherein the rotation angle of the endoscope due to the rotation process at the position of a target is calculated such that a sum of a cumulative total angle of rotation angles of the endoscope due to a rotation process performed until the distal end portion arrives at the position of the target after the distal end portion has been inserted into the luminal structure, and the rotation angle of the endoscope due to the rotation process at the position of the target falls within a predetermined angle range.

[0019] In the image processing method and program of the present invention, the angle range may be from -90° to +90°.

[0020] The image processing program of the present invention typically includes multiple program modules, and each process is implemented by one or more program modules. A group of program modules is recorded on a recording medium such as a CD-ROM or DVD, or recorded in a downloadable state in a memory connected to the server computer or a network storage device and made available to a user.

[0021] In the image processing apparatus, method, and program of the present invention, assuming that an advancing and retreating operation in an axial direction, a rotating operation around an axis, and an operation of bending a distal end portion are possible only within a plane including an axis of the distal end portion in the endoscope, the parameter value related to the endoscope operation for advancing the endoscope along a predetermined course in the luminal structure is calculated from the three-dimensional image indicating the luminal structure.Specifically, when a clockwise direction or a counterclockwise direction around the axis is set as positive, the other direction is set as negative, and a rotation angle of the endoscope due to the rotation operation at any position on the course is calculated as a parameter value related to the rotation operation, the rotation angle of the endoscope due to the rotation operation at the position of the target is calculated so that a sum of a cumulative total angle of rotation angles of the endoscope due to a rotation operation performed until the distal end portion arrives at the position of the target after the distal end portion is inserted into the luminal structure and the rotation angle of the endoscope due to the rotation operation at the position of the target falls within the predetermined angle range.Thus, a suitable angle range, taking into account the range of motion of a wrist or the like, is determined in advance so that the rotation of the endoscope due to the rotation process occurs only within the angle range. It is possible to reduce the operator's difficulty in re-gripping the endoscope and improve operability. Furthermore, it is possible to reduce patient burden by shortening examination time thanks to the improved operability. Short description of the drawings Fig. 1 is a diagram illustrating a schematic configuration of an image processing system including an embodiment of the present invention. Fig. 2 is a block diagram illustrating functions used in an image processing apparatus in Fig. 1 are provided. Fig. 3 is a diagram illustrating a method for calculating a bending angle in a bending operation of an endoscope. Fig. Figure 4 is a diagram illustrating a plane passing through positions of a bending action at three points in a course of a bronchi. Fig. 5 is a diagram illustrating a plane in which an operation of bending a distal end portion of an endoscope is possible. Fig. 6 is a diagram illustrating a method for calculating a rotation angle in a rotation operation of the endoscope. Fig. 7 is a diagram (part 1) illustrating a method of determining a rotation angle based on a direction of an image. Fig. 8 is a diagram (part 2) illustrating a method of determining a rotation angle based on a direction of an image. Fig. Fig. 9 is a flowchart illustrating a flow of a process performed by the image processing apparatus of Fig. 1 is carried out. Description of the preferred embodiments

[0022] Hereinafter, an image processing system including an embodiment of an image processing apparatus, a method and a program of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing a schematic configuration of an image processing system. As shown in Fig. 1, in this system, an endoscope device 1, a 3D image recording device 2, an image storage server 3 and an image processing device 4 are communicatively connected to one another via the network 9.

[0023] The endoscope device 1 includes an endoscope 11 that is inserted into a bronchi (a luminal structure) of a person and images the inside of the bronchi, a processor device 12 that generates an image of the inside of the bronchi (hereinafter referred to as "endoscopic image") based on an imaging signal obtained by imaging in the endoscope 11, and a position detection device 13 that detects a position of a distal end of the endoscope 11 in the bronchi.

[0024] The endoscope 11 is designed so that an advance and retraction operation in an axial direction, a rotation operation about an axis, and a bending operation of a distal end portion are possible only within a plane including an axis of the distal end portion. The endoscope 11 includes an operation unit for operating an operation of the endoscope on the base end side and has a mechanism for bending the distal end portion within a predetermined angular range within a plane including an axis of the distal end portion according to an operation input from the operation unit.

[0025] The processor device 12 converts the image signal obtained by imaging in the endoscope 11 into a digital image signal, performs image quality correction through digital signal processing such as white balance adjustment and shading correction, and generates the endoscopic image. The generated endoscopic image is sent to the image processing device 4 via the image storage server 3 or directly.

[0026] The position detection device 13 detects a characteristic shape of a distal end of the endoscope using, for example, an echo device with a detection range in a three-dimensional coordinate system with a specific position of the person as a reference point, to detect the position of the distal end of the endoscope in the bronchi. Information about the detected position is sent to the image processing device 4.

[0027] The 3D image pickup device 2 is a device that images an inspection target part of the person to generate a three-dimensional image representing the inspection target part. Specifically, the 3D image pickup device 2 is a device for computed tomography (CT), magnetic resonance imaging (MRI), or the like. The generated three-dimensional image is sent to the image storage server 3 and stored. In this embodiment, the 3D image pickup device 2 generates a three-dimensional image in which a chest area including the bronchi is captured.

[0028] The image storage server 3 is a computer that stores and manages data and includes an external storage device and database management software. The image storage server 3 acquires image data such as the endoscopic image acquired by the endoscope device 1 and the three-dimensional image generated by the 3D image acquisition device 2 via a network, stores the image data in a recording medium such as an external storage device, and manages the image data.

[0029] The image processing device 4 is an image processing device in which an image processing program of the present invention is installed in a computer (including a smartphone or a tablet computer). The image processing device 4 includes a device body 5 having a central processing unit (CPU) and the like, an input unit 6 that receives input from a user, and a display unit 7 that performs display. The input unit 6 is a mouse, a keyboard, a touch panel, or the like. The display unit 7 is a liquid crystal display, a touch panel, a touch screen, or the like.

[0030] The device body 5 includes a CPU 5a, a memory 5b, and a hard disk drive (HDD) 5c. The CPU 5a, the memory 5b, and the HDD 5c are connected to each other by a bus line. The HDD 5c stores the image processing program of the present invention and the data to which the program refers. The CPU 5a executes various processes using the memory 5b as the primary storage area according to the program stored in the HDD 5c.

[0031] Fig. 2 is a functional block diagram illustrating functions of the image processing device 4. As shown in Fig. 2, the device body 5 of the image processing device 4 functions as an image acquisition unit 51, a parameter calculation unit 52, an image generation unit 53, a position information acquisition unit 54, and a display control unit 55 by the CPU 5a executing information processing according to the program stored in the HDD 5c.

[0032] The image acquisition unit 51 acquires the image data such as the endoscopic image generated by the endoscope device 1 and the three-dimensional image generated by the 3D image capturing device 2 directly from the respective devices or via the image storage server 3. The image acquired by the image acquisition unit 51 is stored in the HDD 5c.

[0033] The parameter calculation unit 52 calculates a parameter value related to an endoscope operation for advancing the endoscope along a predetermined course in the bronchi from the three-dimensional image acquired by the image acquisition unit 51. In this case, it is assumed that an advancing and retracting operation in an axial direction, a rotation operation about an axis, and an operation of bending a distal end portion are possible only within a plane containing an axis of the distal end portion in an endoscope.

[0034] The parameter calculation unit 52 first extracts a 3D graph structure of a bronchial region included in the three-dimensional image through image analysis and automatically or manually sets a course C along which the endoscope is inserted into the extracted graph structure. Based on information about the set course C, the parameter calculation unit 52 calculates parameter values ​​related to the advancing and retracting action, the rotating action, and the bending action of the endoscope.

[0035] Specifically, the parameter calculation unit 52 calculates an insertion length of the endoscope along the set course C as a parameter of the advance and retreat operation. Further, the parameter calculation unit 52 calculates a bending angle of the endoscope, which corresponds to an angle of bending at any position where the endoscope is bent by a branch or the like on the course C, as a parameter of the bending operation at the position. More specifically, the parameter calculation unit 52 calculates an angle θ(i) formed by two vectors including a vector V(i-1) from a position P(i-1) of a previous bending operation on the course C to a position P(i) on the course C and a vector V(i) from the position P(i) to a position P(i+1) of a next bending operation on the course C, as the angle of bending of the endoscope at the position P(i), as shown in Fig. 3 is shown.

[0036] Further, the parameter calculation unit 52 calculates the angle of rotation in the rotation process of the endoscope for further advancing the endoscope to an arbitrary position P(i) (or a position before) at which the endoscope is bent due to a branch or the like on the course C, forward along the course C as a parameter of the rotation process at the position. Specifically, first, a plane S(i) passing through three points of a position P(i-1) of an immediately previous bending process on the course C, a current position P(i), and a position P(i+1) of the next bending process, as shown in Fig. 4. As shown in Fig. As shown in Figure 5, in a state of the endoscope before the rotation process, a plane S is obtained which passes through three points of a position Q1, which is a pivot point of the bending of the distal end portion, and the positions Q2 and Q3 of the distal end of the endoscope, which are obtained when the distal end portion is bent by any two different bending angles. In Fig. 5, a direction in which the operation of bending the distal end portion can be performed is indicated by an arrow B. Likewise, in Fig. 6 a direction in which the operation of bending the distal end portion can be performed is indicated by an arrow B.

[0037] Furthermore, the parameter calculation unit 52 calculates as in Fig. 6 shows a rotation angle around the axis of the endoscope, which is a rotation angle to cause the plane S to overlap the plane S(i), and the sum of which with the cumulative total angle, the rotation angle of the endoscope due to the rotation process performed until the distal end portion of the endoscope arrives at the current position P(i) after the distal end portion of the endoscope is inserted into the bronchi, wherein the clockwise rotation around an axis of the endoscope is positive and counterclockwise rotation is negative, falls within a predetermined angle range Ra (e.g., -90° to +90°), as a parameter of the rotation process at the position P(i).For example, in a case where the predetermined angle range Ra ranges from -90° to +90° and a cumulative total angle of the rotation angle of the endoscope due to the rotation operation performed until the endoscope arrives at the position P(i) is +45°, and the rotation angle of the endoscope to cause the plane S to overlap with the plane S(i) is +50° (clockwise) or -130° (counterclockwise), -130°, that is, a rotation angle whose sum with the cumulative total angle falls within the range of 90° to +90°, is calculated as a parameter of the rotation operation at the position P(i).

[0038] Although a case where the angular range Ra is set from -90° to +90° has been described in the above description, the angular range Ra can be arbitrarily set automatically or manually, and can be changed as needed. An example of the settable angular range Ra may include -Ta° to +Tb° (e.g., -120° to +120°), where Ta and Tb are, for example, positive values ​​greater than 90. Here, when setting such an angular range, a sum of any clockwise and counterclockwise rotation angles at which the plane S can be caused to overlap with the plane S(i) and a cumulative total angle of the rotation angles of the endoscope due to the rotation operation performed until the distal end portion of the endoscope arrives at the current position P(i) may fall within the angular range Ra.For example, in a case where the predetermined angle range Ra ranges from -120° to +120°, a cumulative total angle of the rotation angles of the endoscope due to the rotation operation performed up to the position P(i) is +45°, and the rotation angle of the endoscope capable of causing the plane S to overlap with the plane S(i) is +50° (clockwise) or -130° (counterclockwise), a sum of any rotation angles and the cumulative total angle falls within a range of -120° to +120°.

[0039] In such a case, the parameter calculation unit 52 may calculate one of two rotation angles, clockwise and counterclockwise, as parameter values ​​related to the rotation operation using any of the first to third methods described below.

[0040] As a first method, the parameter calculation unit 52 may calculate the rotation angle at which a direction from a dorsal side to a ventral side of a human body (or a direction from the ventral side to the dorsal side) according to the rotation angle among the two clockwise and counterclockwise rotation angles is closer to an upward direction of an image captured by the endoscope after the rotation process, as a parameter value related to the rotation process.

[0041] In this case, there are various methods for specifying the rotation angle at which a direction from the dorsal side to the ventral side of the human body (or a direction from the ventral side to the dorsal side) is closer to the upward direction of the image among the two rotation angles. For example, a method for acquiring a direction vector on a three-dimensional image corresponding to an upward direction of the image acquired by the endoscope after the rotation process according to the rotation angle includes, with respect to each of the two rotation angles, calculating an inner product of the direction vector on the three-dimensional image and a direction vector directed from the dorsal side to the ventral side of the human body in the three-dimensional image, and specifying the rotation angle at which a value obtained by the calculation is small.Further, there is a method of obtaining the angle β in the upward direction Eup of the image taken by the endoscope immediately after insertion into the bronchi with respect to the direction Bup from the dorsal side to the ventral side of the human body in the three-dimensional image, and specifying a rotation angle at which a sum of the angle β and a cumulative total angle of the rotation angles of the endoscope due to the rotation operation performed until the distal end portion of the endoscope arrives at the current position P(i) after the distal end portion of the endoscope is inserted into the bronchi falls within an angle range Rup of (-90-β)° to (+90-β)°.

[0042] For example, as in Fig. 7, in a case where the predetermined angle range Ra ranges from -120° to +120° and the second angle range Rup ranges from -90° to +90° due to the coincidence of the upward direction Eup of the image taken by the endoscope in a state immediately after insertion into the bronchi with the direction Bup from the dorsal side to the ventral side of the human body in the three-dimensional image, the plane S overlaps with the plane S(i), and in a case where there are two rotation angles of the endoscope whose sum with the cumulative total angle falls within a range of -120° to +120°, the rotation angle among the two rotation angles at which a sum with the cumulative total angle falls within the range of -90° to +90° is calculated as the parameter value related to the rotation process.

[0043] Further, in a case where the predetermined angle range Ra ranges from -120° to +120° and the second angle range Rup ranges from -70° to +110°, since the angle β of an upward direction Eup of the image taken by the endoscope at an insertion start position with respect to a direction Bup from the dorsal side to the ventral side of the human body in the three-dimensional image is -20° as shown in Fig. 8, the plane S can be caused to overlap with the plane S(i), and when there are two rotation angles of the endoscope whose sum with the cumulative total angle falls within the range of -120° to +120°, the rotation angle among the two rotation angles at which the sum with the cumulative total angle falls within the range of -70° to +110° is calculated as the parameter value related to the rotation process.

[0044] As the second method, the parameter calculation unit 52 may calculate the rotation angle at which a direction directed from an inside to an outside of a human body corresponding to the rotation angle among the two clockwise and counterclockwise rotation angles is closer to an upward direction of an image taken by the endoscope after a rotation operation, as a parameter value for the rotation operation.In this case, a method of specifying the rotation angle among the two rotation angles in which a direction directed from the inside to the outside of the human body is closer to the upward direction of the image includes, for example, a method including: first setting a reference point at a position in a three-dimensional image corresponding to a center (or a barycenter) of a human body portion included in the three-dimensional image, and then specifying a rotation angle in which the set reference point is present in a lower portion among the two divided portions in a case where, for each rotation angle, the three-dimensional image acquired by the endoscope after the rotation operation is divided into two in a vertical direction according to the rotation angle.

[0045] As the third method, the parameter calculation unit 52 may calculate a rotation angle at which a sum of absolute values ​​of the rotation angles at positions where respective rotation operations are performed from one position thereof to positions where one or more previous rotation operations are performed is smaller among the two clockwise and counterclockwise rotation angles as a parameter value for the rotation operation.

[0046] The image generation unit 53 generates a virtual endoscopic image obtained by virtually imaging an inner wall of the bronchi included in the three-dimensional image through a virtual endoscope whose position and posture are defined by parameter values ​​of the endoscope operation calculated by the parameter calculation unit 52 from the three-dimensional image. A known method such as a volume rendering method or a surface rendering method can be used to generate the virtual endoscopic image. According to the image generation unit 53, it is possible to obtain not only the virtual endoscopic image of a still image at each position on the course C but also a virtual endoscopic image of a moving image acquired according to a movement of the endoscope along the course C.

[0047] The position information acquisition unit 54 acquires position information of the distal end of the endoscope detected by the position detection device 13.

[0048] The display control unit 55 displays navigation information for guiding the endoscope to a target position along a predetermined course in the bronchi on the display unit 7. Specifically, the display control unit 55 may display a parameter value of the endoscope operation calculated by the parameter calculation unit 52, the virtual endoscopic image of the moving image or the still image generated by the image generation unit 53, and the like as navigation information on the display unit 7. For example, the display control unit 55 may display the endoscopic image acquired in real time by the image acquisition unit 51 of the endoscope device 1 on the display unit 7, and may also display parameter values ​​of an endoscope operation at a position of the distal end of the endoscope 11 acquired by the position information acquisition unit 54.Furthermore, the display control unit 55 may display the virtual endoscopic image for navigating an endoscopic procedure at the position of the distal end of the endoscope 11, obtained by the position information acquisition unit 54, or a next endoscopic procedure on the display unit 7, instead of displaying the endoscopic image and the parameter values ​​of the endoscopic procedure, or in addition to displaying them. Of course, the parameter values ​​of the endoscopic procedure may be displayed together on the virtual endoscopic image display.

[0049] Next, an example of a flow of a process performed by the image processing apparatus 4 will be described with reference to a Fig.9. First, the image acquisition unit 51 acquires the three-dimensional image obtained by imaging the chest area including the bronchi from the image storage server 3 (S1). Then, the parameter calculation unit 52 calculates the parameter value related to the endoscope operation for advancing the endoscope along a predetermined course in the bronchi from the three-dimensional image (S2). Specifically, the parameter calculation unit 52 first extracts a three-dimensional graph structure of a bronchial area included in the three-dimensional image through image analysis, sets the course C along which the endoscope is inserted in the extracted graph structure, and calculates parameter values ​​for the advancing and retracting operation, the rotating operation, and the bending operation of the endoscope based on information about the set course C.Specifically, the parameter calculation unit 52 calculates a rotation angle around the axis of the endoscope to cause the plane S on which an endoscope bending operation can be performed to overlap with the plane S(i) passing through positions of three points on the course C including an arbitrary position P(i), which is a rotation angle whose sum with a cumulative total angle makes the rotation angle of the endoscope due to the rotation operation performed until the distal end portion of the endoscope arrives at the current position P(i) after the distal end portion of the endoscope is inserted into the bronchi fall within a predetermined angle range Ra (e.g., -90° to +90°), as a parameter of the rotation operation of the endoscope at the arbitrary position P(i) where the endoscope is bent by a branch or the like on the course C.

[0050] Then, the position information acquisition unit 54 acquires the position information of the distal end of the endoscope detected by the position detection device 13 (S3), and the display control unit 55 acquires the parameter value of the endoscope operation corresponding to the position of the distal end of the endoscope 11 obtained by the position information acquisition unit 54 from the parameter values ​​related to the endoscope operation calculated in step S2 (S4). Furthermore, the image generation unit 53 generates a virtual endoscopic image for navigating the endoscope operation according to the position of the distal end of the endoscope 11 acquired by the position information acquisition unit 54 from the three-dimensional image (S5). The display control unit 55 displays the information about the parameter value acquired in step S4 or the virtual endoscopic image acquired in step S5 on the display unit 7 (S6), and ends the process.

[0051] With the above configuration, in the image processing system of this embodiment, assuming that the endoscope is an endoscope in which an advancing and retracting operation in an axial direction, a rotating operation about an axis, and an operation of bending a distal end portion are possible only within a plane including an axis of the distal end portion, the parameter calculating unit 52 calculates the parameter value related to the endoscope operation for advancing the endoscope along a predetermined course in the bronchi from the three-dimensional image representing the bronchi.Specifically, when a clockwise direction or a counterclockwise direction around the axis is set as positive, the other direction is set as negative, and a rotation angle of the endoscope due to the rotation operation at any position on the course is calculated as a parameter value related to the rotation operation, the rotation angle of the endoscope due to the rotation operation at the position of the target is calculated so that a sum of a cumulative total angle of rotation angles of the endoscope due to a rotation operation performed until the distal end portion arrives at the position of the target after the distal end portion is inserted into the bronchi and the rotation angle of the endoscope due to the rotation operation at the position of the target falls within the predetermined angle range Ra.Thus, an appropriate angular range Ra for which a movement range of a wrist or the like is taken into account is determined in advance so that the rotation of the endoscope due to the rotation operation is performed only within the angular range Ra, and it is possible to reduce the difficulty of an operator in re-gripping an endoscope and improve operability.

[0052] Particularly, in the image processing system of this embodiment, in a case where the angle range Ra is from -90° to +90°, it is possible to reduce the operator's difficulty in re-gripping an endoscope and improve operability.

[0053] Although the case where the image processing device 4 includes the image generation unit 53, the position information acquisition unit 54, and the display control unit 55 has been described in the above embodiment, configurations thereof may be omitted, wherein the parameter value related to the endoscope operation is calculated by the parameter calculation unit 52 from the three-dimensional image acquired by the image acquisition unit 51, and the calculated parameter value may be provided to an external device.

[0054] In the above embodiment, the case where the image processing apparatus, method, and program of the present invention are applied to the operation of the endoscope in the bronchi has been described, but the present invention is not limited thereto, and the present invention can also be applied to a case where the endoscope is operated in another tubular structure having a branched structure such as a vessel. List of reference symbols 1 endoscope device 2 3D image recording device 3 image storage servers 4 Image processing device 5 fixture body 5a CPU 5b Memory 5c HDD 6 Input unit 7 Display unit 9 Network 11 Endoscope 12 Processor device 13 Position detection device 51 Image acquisition unit 52 Parameter calculation unit 53 Image generation unit 54 Position information acquisition unit 55 Display control unit

Claims

[1] An image processing apparatus including a parameter calculation unit that calculates a parameter value related to an endoscope operation for advancing an endoscope along a predetermined course within a luminal structure from a three-dimensional image indicating the luminal structure, wherein an advancing and retracting operation in an axial direction, a rotating operation about an axis, and an operation of bending a distal end portion are only possible within a plane containing an axis of the distal end portion with the endoscope, and the parameter calculation unit sets a clockwise direction or a counterclockwise direction around the axis as positive, sets the other direction as negative, and calculates a rotation angle of the endoscope due to the rotation operation at an arbitrary position on the course as the parameter value related to the rotation operation, wherein the parameter calculation unit calculates the rotation angle of the endoscope due to the rotation operation at the arbitrary position such that a sum of a cumulative total angle of rotation angles of the endoscope due to a rotation operation performed until the distal end portion arrives at the arbitrary position after the distal end portion is inserted into the luminal structure and the rotation angle of the endoscope due to the rotation operation at the arbitrary position falls within a predetermined angle range. [2] The image processing apparatus according to claim 1, wherein the angle range is from -90° to +90°. [3] Image processing device according to claim 1, wherein the luminal structure is a luminal organ of a human body, the angle range is from -Ta° to +Tb°, where Ta and Tb are positive values, where Ta and Tb are positive values ​​greater than 90, and the parameter calculation unit, in calculating the rotation angle of the endoscope due to the rotation operation at the arbitrary position on the course, calculates, as a parameter value related to the rotation operation, a rotation angle in which a direction from a dorsal side to a ventral side of a human body is closer to an upward direction of an image taken by the endoscope after the rotation operation, as a parameter value related to the rotation operation in a case where there are two rotation angles of the endoscope due to the rotation operation at the arbitrary position, so that the sum falls within the angle range. [4] Image processing apparatus according to claim 1, wherein the luminal structure is a luminal organ of a human body, the angle range is from -Ta° to +Tb°, where Ta and Tb are positive values, where Ta and Tb are positive values ​​greater than 90, and the parameter calculation unit, in calculating the rotation angle of the endoscope due to the rotation operation at the arbitrary position on the course, calculates, as a parameter value related to the rotation operation, a rotation angle in which a direction from an inside to an outside of the human body is closer to an upward direction of an image taken by the endoscope after the rotation operation, as a parameter value related to the rotation operation in a case where there are two rotation angles of the endoscope due to the rotation operation at the arbitrary position, so that the sum falls within the angle range. [5] Image processing apparatus according to claim 1, where the angle range is from -Ta° to +Tb°, where Ta and Tb are positive values, where Ta and Tb are positive values ​​greater than 90, and the parameter calculation unit, when calculating the rotation angle of the endoscope due to the rotation operation at an arbitrary position on the course, calculates, as a parameter value related to the rotation operation, a rotation angle at which a sum of absolute values ​​of the rotation angles at positions where respective rotation operations are performed from the arbitrary position to positions where one or more previous rotation operations are performed is smaller than a parameter value related to the rotation operation in a case where there are two rotation angles of the endoscope due to the rotation operation at the arbitrary position, so that the above-described sum falls within the angle range. [6] An image processing apparatus according to any one of claims 1 to 5, comprising: an image generation unit that applies the parameter value calculated by the parameter calculation unit to generate a virtual endoscopic image from the three-dimensional image. [7] An image processing apparatus according to any one of claims 1 to 6, comprising: a display unit that displays an image; and a display control unit that displays the parameter value corresponding to the arbitrary position of the endoscope and calculated by the parameter calculation unit on the display unit according to the arbitrary position of the endoscope on the course when the endoscope is advanced along the course in the luminal structure. [8] Image processing method that includes: Calculating a parameter value relating to an endoscope operation for advancing an endoscope along a predetermined course within a luminal structure from a three-dimensional image indicating the luminal structure, wherein an advance and retraction operation in an axial direction, a rotation operation about an axis and an operation of bending a distal end portion are only possible or permitted with the endoscope within a plane containing an axis of the distal end portion, and a process for calculating the parameter value, which includes: setting a clockwise direction or a counterclockwise direction around the axis as positive, setting the other direction as negative, and calculating a rotation angle of the endoscope due to the rotation operation at an arbitrary position on the course as the parameter value related to the rotation operation, wherein the rotation angle of the endoscope due to the rotation operation at the arbitrary position is calculated such that a sum of a cumulative total angle of rotation angles of the endoscope due to a rotation operation performed until the distal end portion arrives at the arbitrary position after the distal end portion is inserted into the luminal structure and the rotation angle of the endoscope due to the rotation operation at the arbitrary position falls within a predetermined angle range. [9] An image processing method according to claim 8, wherein the angle range is from -90° to +90°. [10] An image processing program for causing a computer to perform a process of calculating a parameter value relating to an endoscope operation for advancing an endoscope along a predetermined course within a luminal structure from a three-dimensional image indicating the luminal structure, wherein an advancing and retracting operation in an axial direction, a rotating operation about an axis, and an operation of bending a distal end portion are only possible within a plane containing an axis of the distal end portion with the endoscope, and a process for calculating the parameter value contains: Setting one of a clockwise direction and a counterclockwise direction around the axis as positive, setting the other direction as negative, and calculating a rotation angle of the endoscope due to the rotation operation at an arbitrary position on the course as the parameter value related to the rotation operation, wherein the rotation angle of the endoscope due to the rotation operation at the arbitrary position is calculated such that a sum of a cumulative total angle of rotation angles of the endoscope due to a rotation operation performed until the distal end portion arrives at the arbitrary position after the distal end portion is inserted into the luminal structure and the rotation angle of the endoscope due to the rotation operation at the arbitrary position falls within a predetermined angle range. [11] Image processing program according to claim 10, wherein the angle range is from -90° to +90°.

Images