Image processing device, manipulator system, and operation method of image processing device

Abstract

An image processing device (200) comprises an acquisition unit (220) and a processing unit (210). The acquisition unit acquires shape information pertaining to the shape of a manipulator that is capable of changing the shape. On the basis of the shape information, the processing unit displays, on a display unit (20), a CG image of a manipulator model viewed from a virtual camera disposed in an object space. The processing unit performs rotation processing in which the line of sight of the virtual camera is rotated by a rotation angle according to the shape.

Description

[Technical field]

[0001] The present invention relates to an image processing device and a manipulator system. and an operation method of an image processing device etc. [Background technology]

[0002] Patent Document 1 discloses a navigation technology that displays a virtual 3D model of the bending part of an endoscope together with an endoscopic image. When an angle operation is performed on the operation unit, the bending angle of the bending part in the 3D model changes based on the operation input. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-529882 Summary of the Invention [Problem to be solved by the invention]

[0004] When a surgeon adjusts the angle while viewing a 3D model, it can be difficult to perform intuitive operation. For example, when the bending angle is large, the 3D model is displayed so that the tip of the endoscope faces forward, and the imaging direction of the endoscopic image and the 3D model do not match, making intuitive operation difficult. [Means for solving the problem]

[0005] One aspect of the present disclosure includes an acquisition unit that acquires shape information regarding the shape of a shape-changeable manipulator, and a processing unit that displays on a display a CG image of a manipulator model viewed from a virtual camera in object space based on the shape information, wherein the processing unit relates to an image processing device that performs rotation processing to rotate the line of sight of the virtual camera by a rotation angle corresponding to the shape.

[0006] Another aspect of the present disclosure relates to a manipulator system including a manipulator having an imaging unit at its tip, an image processing device, and a display, wherein the image processing device displays, on the display, a CG image of a manipulator model in the line of sight direction of a virtual camera that follows the imaging direction of the imaging unit.

[0007] Yet another aspect of the present disclosure relates to an image processing method for acquiring shape information regarding a shape of a shape-changing manipulator, performing a rotation process for rotating the line of sight of a virtual camera in an object space by a rotation angle corresponding to the shape based on the shape information, and displaying on a display a CG image of a manipulator model viewed from the virtual camera in the object space.

[0008] Yet another aspect of the present disclosure relates to a display method for displaying a CG image of a manipulator model, the display method displaying on a display the CG image of the manipulator model in the line of sight direction of a virtual camera that follows the imaging direction of an imaging unit provided at the tip of the manipulator. [Brief description of the drawings]

[0009] [Figure 1] An example of how a 3D model is displayed when the base end of the bending part of the endoscope is fixed. [Diagram 2] 1 shows an example of the configuration of an image processing device and an endoscope system. [Diagram 3] FIG. 2 is a diagram for explaining an endoscope in a real space. [Figure 4] FIG. 1 is an explanatory diagram of a 3D model constructed by the processing unit. [Diagram 5] 4 shows the relationship between the curvature angle and the rotation angle of the line of sight in the first embodiment. [Figure 6] 4 is a display example in the first embodiment. [Figure 7] 4 is a flowchart of a process according to the first embodiment. [Figure 8] 13 shows the relationship between the curvature angle and the rotation angle of the line of sight in the second embodiment. [Figure 9]13 is a display example of a CG image in the second embodiment. [Figure 10] 11 is a flowchart of a process according to a second embodiment. [Figure 11] 13 shows the relationship between the curvature angle and the rotation angle of the line of sight in the third embodiment. [Figure 12] 13 is a display example of a CG image in the third embodiment. [Figure 13] 13 is a flowchart of a process according to a third embodiment. [Figure 14] 13 is a flowchart when the fifth embodiment is combined with the second embodiment. [Figure 15] 13 is a flowchart when the fifth embodiment is combined with the third embodiment. [Figure 16] 13 shows a configuration example of a bending portion in the sixth embodiment. [Figure 17] An example of the two-stage curvature control mode. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present embodiment will be described below. Note that the present embodiment described below does not unduly limit the contents described in the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential components of the present disclosure.

[0011] For example, an example of displaying a 3D model of an endoscope in an endoscope system will be described below, but the method of this embodiment can be applied to the case of displaying a CG image of a manipulator model in a manipulator system. The manipulator model is a three-dimensional model of a manipulator whose shape changes with operation. The manipulator has an imaging unit installed thereon or holds an imaging unit, and the imaging direction of the imaging unit changes with the change in the shape of the manipulator. The CG image is an image generated from the three-dimensional model of the manipulator. Although the CG image is two-dimensional information, it is also called a 3D model or a 3D image in the sense that it is an image of a three-dimensional model.

[0012] 1. Image processing device and endoscope system Figure 1 shows an example of a 3D model of an endoscope with the base end of the bending section fixed. An endoscope image is displayed on the left side of the screen, and a CG image of the endoscope is displayed on the right side of the screen. CG stands for Computer Graphics.

[0013] In Fig. 1, Xr, Yr, and Zr indicate a coordinate system fixed to the tip of the endoscope in the real space where the actual endoscope exists, and Zr corresponds to the imaging direction. x, y, and z are world coordinate systems fixed to the object space in which the 3D model is constructed. In the object space, the base end of the bending part is fixed to face the z direction, and the imaging direction changes depending on the bending angle.

[0014] Display image IMA1 shows an example in which the bending portion is straight and not curved. In the endoscopic image, the screen depth direction always coincides with the imaging direction. When the bending portion is straight, the imaging direction also coincides with the screen depth direction in the 3D model, and the imaging direction in the endoscopic image coincides with the imaging direction in the 3D model. In this case, an intuitive bending operation is possible while viewing the 3D model. That is, the desired bending direction and the operation direction intuitively coincide with each other.

[0015] Display image IMA2 shows an example in which the bending portion has made a J-turn to the left, i.e., the tip of the endoscope faces the front of the screen. Since the imaging direction in the 3D model faces the front of the screen, the imaging direction in the endoscopic image and the imaging direction in the 3D model are opposite. In this case, it is difficult to intuitively match the desired bending direction with the operation direction, making it difficult to intuitively perform bending operation while viewing the 3D model. For example, if it is desired to bend further, the tip of the endoscope is moved to the right in the 3D model, but the operation results in a left bending, and the moving direction of the tip on the 3D model does not match the operation direction.

[0016] For example, in an endoscope that is bent by a manual dial operation, the surgeon can estimate the bending angle to some extent by force feedback such as resistance from the dial accompanying the pulling of the wire. However, in an electric endoscope in which the bending drive is motorized, the wire is pulled via a driving unit such as a motor, so that force feedback is difficult to obtain with the operation unit held by the surgeon. In addition, in an electric endoscope, bending can be performed using an operation unit separated from the driving unit such as a touch pad or a remote controller, in which case force feedback is even more difficult to obtain. Therefore, it is desirable to assist the surgeon in understanding by navigation that displays the bending state in a 3D model so that the surgeon can easily grasp the bending angle. In this case, there is a problem that the operation direction of the touch pad, etc. and the movement of the 3D model are difficult to intuitively match.

[0017] 2 shows an example of the configuration of an image processing device and an endoscope system according to this embodiment. The endoscope system 300 includes an endoscope 10, a display 20, a control device 100, an image processing device 200, and an operation unit 400. Note that, although an electric endoscope will be described here as an example, the technique described below is applicable not only to electric endoscopes but also to various endoscopes having bending sections. Also, as described above, in this embodiment, the endoscope system 300 corresponds to a manipulator system, and the endoscope 10 corresponds to a manipulator.

[0018] The endoscope 10, also called a scope, is inserted into the body and captures images of the inside of the body. In this embodiment, a flexible endoscope including a flexible insertion portion is assumed. The endoscope 10 includes a bending portion 40 and an imaging portion 30. The bending portion 40 is provided at the tip of the endoscope 10 and is configured to be bendable in response to a bending operation. The bending portion 40 is also called a bending device or bending mechanism. The imaging portion 30 is an imaging device including an imaging element and an objective lens, etc., and is provided at the tip side of the bending portion 40. The imaging portion 30 captures images of the inside of the body and transmits the image signals to the control device 100. The endoscope 10 may further include an illumination lens, a forceps port, a suction port, etc.

[0019] The control device 100 controls the endoscope 10 and drives it to bend. The control device 100 is also called a controller, a drive device, a processor unit, etc. The control device 100 includes a motor 101, a drive circuit 102, a sensor 103, and an image interface 104.

[0020] The drive circuit 102 is a circuit that drives the motor 101, and drives the motor 101 in response to an operation input from the operation unit 400. The motor 101 bends the bending portion 40 by pulling a wire connected to the bending portion 40 of the endoscope 10. As a result, the bending portion 40 is bent at a bending angle in response to the operation input. The sensor 103 is a sensor that detects the bending angle of the bending portion 40, and transmits the detection signal to the image processing device 200. The sensor 103 is, for example, a rotary encoder that detects the amount of rotation of the motor 101, or an optical sensor that detects the amount of travel of the wire. The sensor 103 transmits the detected bending angle to the image processing device 200.

[0021] It should be noted that the sensor 103 only needs to detect a physical quantity capable of estimating the bending angle, and does not need to detect the bending angle itself. Although an example in which a sensor is provided inside the control device 100 is shown in Fig. 2, any other configuration may be used as long as it is capable of detecting the bending angle of the bending portion 40. For example, the bending angle may be acquired from an operation input of a bending operation to the operation unit 400, or the shape of the bending portion 40 may be observed from the outside using a UPD, which is an endoscope shape observation device, and the bending angle may be acquired from the observed shape.

[0022] The operation unit 400 accepts an operation input for operating the endoscope 10, and transmits information of the operation input to the control device 100 by wireless communication. The operation unit 400 can accept at least a bending operation. The operation unit 400 is, for example, a touch pad, and recognizes a touch or swipe operation on the touch pad as a bending operation. The operation unit 400 may include a button or dial for the bending operation, and may communicate with the control device 100 by wire rather than wirelessly.

[0023] The image interface 104 receives an image signal from the imaging unit 30 of the endoscope 10, generates an endoscopic image by image processing the image signal, and transmits image data of the endoscopic image to the image processing device 200. The image processing is, for example, development processing, gradation correction processing, white balance processing, etc. Note that the image interface 104 may be configured to transfer the image signal to the image processing device 200, and the image processing device 200 may construct an endoscopic image from the image signal.

[0024] The image processing device 200 generates a 3D model of the endoscope 10 based on the bending angle detected by the sensor 103, and displays the CG image and the endoscopic image on the display 20. The image processing device 200 is an information processing device such as a PC or a server. The image processing device 200 includes an acquisition unit 220 and a processing unit 210. Various types of hardware are assumed for the acquisition unit 220, and the acquisition unit 220 may be, for example, a communication interface provided in the information processing device, or a data interface provided in the hardware constituting the processing unit 210. The processing unit 210 is also called a processor, processing device, processing circuit, or the like. Various types of hardware are assumed for the processing unit 210, and the processing unit 210 may be, for example, a CPU, a GPU, a microcomputer, or the like, or an ASIC, an FPGA, or the like, or a combination thereof. The image processing device 200 may include a memory (not shown), and the memory may store a program describing the processing performed by the processing unit 210, and the processing unit 210 may execute the program to realize the processing performed by the processing unit 210. Although FIG. 2 shows an example in which the image processing device 200 is provided as an external device of the control device 100, the image processing device 200 may be built into the control device 100.

[0025] The acquisition unit 220 receives image data of the endoscope image and information on the bending angle from the control device 100. The processing unit 210 generates a 3D model of the endoscope 10 in the object space, and displays the CG image of the endoscope 10 generated based on the 3D model and the endoscope image on the display 20. The endoscope image is displayed in a first display area of ​​the display 20, and the CG image is displayed in a second display area of ​​the display 20. Alternatively, a first display and a second display may be provided, and the endoscope image may be displayed on the first display, and the CG image may be displayed on the second display. The processing unit 210 bends the bending portion of the 3D model at the bending angle detected by the sensor 103. The surgeon can grasp the shape of the tip of the endoscope inserted into the body by looking at the CG image.

[0026] A method for constructing a 3D model and a CG image will be described below. First, the endoscope 10 in the real space will be described with reference to FIG.

[0027] Although not shown in FIG. 3, the base of the flexible section is connected to the control device 100. The base end of the bending section 40 is connected to the tip of the flexible section, and the rigid section is provided at the tip of the bending section 40. The imaging section 30 is built into the rigid section. The bending section 40 has a large number of joints connected in series, and can be bent in the up-down and left-right directions. The bending section 40 has a long and thin cylindrical shape when not bent, and bending refers to a state in which the axis of the cylinder is curved and bent up-down and left-right. It is assumed that the XYZ coordinates are fixed to the tip of the flexible section, that is, the base end of the bending section 40. The Z axis is parallel to the axial direction NDr of the tip of the flexible section. At this time, the up-down direction of the bending is bending around the X axis, and the left-right direction of the bending is bending around the Y axis. In addition, when the axial direction of the tip of the bending section 40, that is, the imaging direction, is IDr, the angle formed between the axial direction NDr of the tip of the flexible section and the imaging direction IDr is the bending angle θr.

[0028] 3 shows the bending angle around the Y axis, that is, in the left-right direction. However, since the bending section 40 is operated independently in the up-down direction and the left-right direction, there are bending angles around the X axis and bending angles around the Y axis. In the following, the content described by indicating the bending angle as θr can be applied to bending operations around either the X axis or the Y axis. When describing them with an explicit distinction, the bending angle around the X axis is described as θx, and the bending angle around the Y axis is described as θy.

[0029] FIG. 4 shows an explanatory diagram of the 3D model constructed by the processing unit 210. Assume that xyz coordinates are fixed in the object space in which the 3D model 50 is constructed. The xyz coordinates are also called world coordinates. When the axial direction of the tip of the flexible section is NDm, the 3D model 50 of the endoscope is fixed in the object space so that NDm and the z direction are parallel. The fixed section is, for example, the base of the flexible section or the proximal end of the bending section. When the axial direction of the tip of the bending section, that is, the imaging direction, is IDm, the angle formed between the axial direction NDm of the tip of the flexible section and the imaging direction IDm is the bending angle θm. The 3D model 50 is, for example, a model composed of primitive surfaces such as polygons, or a wireframe model.

[0030] The sensor 103 detects the bending angle θr of the endoscope 10 in the real space, and the processing unit 210 constructs the 3D model 50 by setting the bending angle θr as the bending angle θm in the 3D model 50. The processing unit 210 sets the line of sight direction SDc of the virtual camera 52 based on the bending angle θm, and sets the position and attitude of the virtual camera 52 so that the line of sight direction SDc faces the gaze point RC. The gaze point RC is set, for example, at the tip of the bending part, but is not limited thereto, and may be set within the bending part, such as near the tip of the bending part. The processing unit 210 places the virtual camera 52 so as to look down on the gaze point RC from slightly above the 3D model 50 (in the +y direction).

[0031] The processing unit 210 changes the line of sight direction SDc by changing the position and attitude of the virtual camera 52 according to the curvature angle θm. At this time, the line of sight direction SDc of the virtual camera 52 rotates around the gaze point RC. Specifically, in the case of curvature around the y-axis, the line of sight direction SDc rotates around a rotation axis that passes through the gaze point RC and is parallel to the y-axis. Note that hereinafter, the gaze point RC is also referred to as the rotation center of the line of sight direction SDc. The position and attitude of the virtual camera 52 are also referred to as the viewpoint, and changing the position and attitude of the virtual camera 52 according to the curvature angle θm is also expressed as rotating the viewpoint.

[0032] The angle between the line of sight direction SDc of the virtual camera 52 and the axial direction NDm of the tip of the flexible section is defined as θc. The line of sight direction SDc is expressed by this angle θc. In the case of curvature around the y-axis, the angle between the line of sight direction SDc and the axial direction NDm on the zx plane is θc. In other words, the rotation angle of the line of sight direction SDc around the y-axis is θc. In the case of curvature around the x-axis, the rotation angle of the line of sight direction SDc around the x-axis is θc.

[0033] The processing unit 210 controls the rotation angle of the line of sight direction SDc around the x-axis according to the curvature angle θx around the x-axis, and controls the rotation angle of the line of sight direction SDc around the y-axis according to the curvature angle θy around the y-axis. Note that the content described below using the angle θc can be applied to both viewpoint rotation around the x-axis and around the y-axis. When explicitly distinguishing between them, the angle of rotation around the x-axis is described as θcx, and the angle of rotation around the y-axis is described as θcy.

[0034] The processing unit 210 generates a CG image of the endoscope 3D model seen from the virtual camera 52, and displays the CG image on the display 20 together with the endoscope image. As a result, the viewpoint of the virtual camera 52 changes according to the bending angle of the bending portion 40, and the imaging direction in the endoscope image and the imaging direction in the 3D model on the screen can be made to approximately match. By making the imaging directions approximately match, the surgeon can easily grasp the operation direction of the bending operation. As described later in the first to sixth embodiments, the imaging direction in the endoscope image and the imaging direction in the 3D model do not necessarily have to match, and it is sufficient that they are similar in direction to the extent that the bending operation direction can be intuitively grasped. The processing unit 210 may display a CG image of the internal body 3D model together with the CG image of the endoscope 3D model. The internal body 3D model is constructed from a tomographic image captured in advance by, for example, CT or MRI, and constructed in the object space together with the endoscope 3D model.

[0035] In the above embodiment, the image processing device 200 includes an acquisition unit 220 and a processing unit 210. The acquisition unit 220 acquires shape information related to the shape of a manipulator whose shape can be changed. The processing unit 210 displays a CG image of a manipulator model viewed from a virtual camera 52 in an object space on the display 20 based on the shape information. The processing unit 210 performs a rotation process for rotating the line of sight direction SDc of the virtual camera 52 by a rotation angle θc according to the shape.

[0036] According to this embodiment, the imaging direction of the imaging unit installed in the manipulator or held by the manipulator changes according to the shape of the manipulator. At this time, the viewpoint of the virtual camera 52 changes according to the shape of the manipulator, so that the imaging direction in the captured image and the imaging direction in the CG image of the manipulator model can be made to approximately coincide on the screen.

[0037] In this embodiment, the manipulator includes a bending portion 40. The acquisition unit 220 acquires information on the bending angle θr of the bending portion 40 as shape information. The processing unit 210 performs a rotation process to rotate the line of sight direction SDc of the virtual camera 52 by a rotation angle θc according to the bending angle θr.

[0038] According to this embodiment, the imaging direction IDr of the imaging unit 30 installed at the tip of the bending portion 40 changes according to the bending angle θr. At this time, the viewpoint of the virtual camera 52 changes according to the bending angle θr, so that the imaging direction IDr in the endoscopic image and the imaging direction IDm in the CG image of the endoscopic 3D model can be made to approximately coincide on the screen. This makes it easier for the surgeon to intuitively grasp the operation direction of the bending operation when viewing the 3D model.

[0039] In this embodiment, the rotation process is a process of rotating the line of sight SDc of the virtual camera 52 around a set point set on the manipulator model. More specifically, in the case of rotation around the x-axis, the rotation process is a process of rotating around a rotation axis that passes through the set point and is parallel to the x-axis, and in the case of rotation around the y-axis, the rotation process is a process of rotating around a rotation axis that passes through the set point and is parallel to the y-axis.

[0040] In this embodiment, the gaze point RC corresponds to the set point. As described above, the gaze point RC may be set within the curved portion of the 3D model. That is, the set point is not limited to the tip of the manipulator model, but may be set on the manipulator model. For example, the set point may be set at an arbitrary position in a part whose shape changes due to an operation, or at a base end or the like where a part whose shape changes is connected to a part whose shape does not change.

[0041] Moreover, in this embodiment, the manipulator system includes a manipulator having an imaging unit at its tip, an image processing device 200, and a display 20. The image processing device 200 displays, on the display 20, a CG image of a manipulator model in a line of sight direction SDc of a virtual camera 52 that follows the imaging direction IDr of the imaging unit.

[0042] According to this embodiment, the line of sight direction SDc of the virtual camera 52 follows the imaging direction IDr, and the CG image of the manipulator model is displayed so that the imaging direction in the CG image of the manipulator model approximately matches the imaging direction in the captured image. Note that the line of sight direction follows the imaging direction means that the line of sight direction changes in the same direction as the imaging direction changes, and the line of sight direction does not necessarily match the imaging direction. For example, in the second and third embodiments described later, the line of sight direction does not necessarily match the imaging direction, but such cases are also included in "tracking."

[0043] 2. First embodiment Fig. 5 shows the relationship between the curvature angle θm and the rotation angle θc of the line of sight direction SDc in the first embodiment. Fig. 6 shows a display example in the first embodiment. Note that, in the following, for angles around the y-axis, a counterclockwise angle seen from the +y direction is considered to be a positive angle, and for angles around the x-axis, a counterclockwise angle seen from the +x direction is considered to be a positive angle. In terms of curvature, left curvature and upward curvature are curvatures with positive angles. However, the definition of the angle is not limited to this.

[0044] As shown in FIG. 5, the processing unit 210 sets θc=θm in the entire range of -180 degrees≦θm≦180 degrees. That is, the processing unit 210 matches the line of sight SDc of the virtual camera 52 with the bending angle θm of the 3D model 50 of the endoscope. As a result, as shown in FIG. 6, the imaging direction IDm in the CG image is always the screen depth direction, and always matches the imaging direction IDr in the endoscope image. The display image IMB1 is a display example when the bending portion 40 is not bent, and the display image IMB2 is a display example when the bending portion 40 makes a J-turn to the left. The virtual camera 52 is arranged so that the line of sight SDc matches the imaging direction IDm, and the imaging direction IDm in the CG image is the screen depth direction.

[0045] 7 is a flowchart of the process in the first embodiment. In step S1, the processing unit 210 estimates the bending angle θr of the endoscope 10 from the wire pulling amount detected by the sensor 103. Specifically, a model that associates the wire pulling amount with the bending angle θr is described using a formula or a lookup table, etc., and the wire pulling amount is converted into the bending angle θr using the model. Note that the control device 100 may perform the process of step S1. Also, the physical amount for estimating the bending angle θr is not limited to the wire pulling amount, and may be, for example, an operation knob rotation amount, a motor rotation amount, or a shape measurement result by a UPD.

[0046] In step S2, the processing unit 210 estimates the shape of the curved portion from the curved angle θr. That is, the processing unit 210 sets the curved angle of the 3D model to θm=θr, and reproduces the shape of the curved portion when curved at the curved angle θm in the 3D model. Specifically, the processing unit 210 generates a 3D model at the curved angle θm by simulating the shape of the curved portion at the curved angle θm based on design information such as the joints of the curved portion and the joints connecting the joints.

[0047] In step S3, the processing unit 210 sets the viewpoint of the virtual camera 52 at a position that has a specified positional relationship with respect to the tip of the curved portion of the 3D model. Specifically, the processing unit 210 sets the gaze point RC of the virtual camera 52 at the tip or base end of the curved portion, sets the angle of the line of sight direction SDc to θc = θm, and installs the virtual camera 52 in the object space based on these parameters.

[0048] In step S4, the processing unit 210 generates a CG image of the endoscope 3D model as seen by the virtual camera 52. In step S5, the processing unit 210 displays the generated CG image on the display 20 together with the endoscope image.

[0049] In the above embodiment, the processing unit 210 changes the rotation angle θc of the line of sight direction SDc in synchronization with the curvature angle θr. Synchronizing the rotation angle θc with the curvature angle θr means matching the rotation angle θc in real time with the change in the curvature angle θr.

[0050] According to this embodiment, the viewpoint of the virtual camera 52 rotates so that the line of sight direction SDc faces in the same direction as the bending angle θr, so that the imaging direction IDr in the endoscopic image and the imaging direction IDm in the CG image of the endoscopic 3D model can be made to match on the screen. As a result, regardless of the bending angle, the operation direction of the bending operation matches the moving direction of the tip of the bending portion in the 3D model, so that the surgeon can intuitively grasp the operation direction.

[0051] 3. Second embodiment Fig. 8 shows the relationship between the curvature angle θm and the rotation angle θc of the line of sight direction SDc in the second embodiment. Fig. 9 shows an example of a display of a CG image in the second embodiment.

[0052] As shown in FIG. 8, the processing unit 210 sets θc=0 degrees in the range of |θm|≦α degrees, sets θc=-β degrees in the range of -180≦θm<-α degrees, and sets θc=β degrees in the range of α<θm≦180 degrees. 0<α<180, 0<β<180. If 2α=β, the viewpoint switches when the imaging direction comes to the center between the line of sight before rotation and the line of sight after rotation, so it is easy to grasp the switching of the viewpoint. However, it is sufficient that α≦β. FIG. 9 shows a display example when α=90 degrees and β=180 degrees. When α=90 degrees, the CG image is reversed left and right when the imaging direction becomes 90 degrees. The imaging direction changes between the depth direction of the screen and the horizontal direction, and does not face the front side of the horizontal direction of the screen. As a result, the difference between the imaging direction in the endoscopic image and the imaging direction in the CG image becomes 90 degrees or less, and intuitive bending operation is possible.

[0053] 10 is a flowchart of the process in the second embodiment. In step S21, the processing unit 210 estimates the bending angle θr of the endoscope 10 from the wire pulling amount detected by the sensor 103.

[0054] In step S22, the processing unit 210 sets the viewpoint of the virtual camera 52 at a predetermined position behind the endoscope. Specifically, the processing unit 210 sets the gaze point RC at the base end of the bending portion, and sets the angle of the line of sight SDc to θc = 0 degrees.

[0055] In step S23, the processing unit 210 judges whether or not the curvature angle around the x-axis is |θx|≧α. If |θx|<α, the process proceeds to step S25. If |θx|≧α, in step S24, the processing unit 210 rotates the viewpoint of the virtual camera 52 by β degrees around the x-axis with the gaze point RC as the center. Specifically, if θx≦-α, the processing unit 210 rotates the viewpoint by -β degrees, and if θx≧ +α If so, rotate the viewpoint by +β degrees. Next, proceed to step S25.

[0056] In step S25, the processing unit 210 judges whether or not the curvature angle around the y axis is |θy|≧α. If |θy|<α, the process proceeds to step S27. If |θy|≧α, in step S26, the processing unit 210 rotates the viewpoint of the virtual camera 52 by β degrees around the y axis with the gaze point RC as the center. Specifically, if θy≦-α, the processing unit 210 rotates the viewpoint by -β degrees, and if θy≧ +α If so, rotate the viewpoint by +β degrees. Next, proceed to step S27.

[0057] In step S27, the processing unit 210 generates a CG image of the endoscope 3D model as seen by the virtual camera 52. In step S28, the processing unit 210 displays the generated CG image on the display 20 together with the endoscope image.

[0058] Although the above describes an example in which the viewpoint is switched only once, the viewpoint may be switched multiple times.

[0059] In the above embodiment, the processing unit 210 fixes the rotation angle θc to the first angle until the bending angle θr exceeds the threshold value α. In the example of this embodiment, the first angle is 0 degrees, but is not limited to this. When the bending angle θr exceeds the threshold value α, the processing unit 210 switches the rotation angle θc to a second angle β that is larger than the first angle.

[0060] According to this embodiment, the viewpoint is switched when the bending angle θr exceeds a predetermined angle, so the imaging direction IDr in the endoscopic image and the imaging direction IDm in the CG image of the endoscopic 3D model can be brought closer together than in the case where the viewpoint is not switched. Since the viewpoint does not rotate frequently, it is expected that the curved shape can be more easily visually recognized from the CG image.

[0061] 4. Third embodiment Fig. 11 shows the relationship between the curvature angle θm and the rotation angle θc of the line of sight direction SDc in the third embodiment. Fig. 12 shows an example of a display of a CG image in the third embodiment.

[0062] As shown in FIG. 11, the processing unit 210 sets θc=0 degrees in the range of |θm|≦α1 degrees, changes θc from 0 degrees to 180 degrees in the range of α1 degrees<|θm|≦α2 degrees, and sets θc=β degrees in the range of α2 degrees<|θm|≦180 degrees. 0<α1<α2<180, 0<β<180. FIG. 12 shows a display example when α1=75 degrees, α2=105 degrees, and β=180 degrees. When the bending angle θm changes from 75 degrees to 105 degrees, the line of sight direction of the virtual camera rotates from θc=0 degrees to θc=180 degrees. This makes the difference between the imaging direction in the endoscopic image and the imaging direction in the CG image 75 degrees or less, enabling intuitive bending operation. In addition, an effect is obtained that shows that the viewpoint of the virtual camera has been intentionally changed and the meaning of the viewpoint switching, making it easy for the surgeon to understand the bending state and bending operation direction when looking at the CG image.

[0063] 13 is a flowchart of the process in the third embodiment. In step S41, the processing unit 210 estimates the bending angle θr of the endoscope 10 from the wire pulling amount detected by the sensor 103.

[0064] In step S42, the processing unit 210 sets the viewpoint of the virtual camera 52 at a predetermined position behind the endoscope. Specifically, the processing unit 210 sets the gaze point RC at the base end of the bending portion, and sets the angle of the line of sight direction SDc to θcx = θcy = 0 degrees.

[0065] In step S43, the processing unit 210 judges whether the curvature angle around the x-axis is |θx|≧α1. If |θx|<α1, the process proceeds to step S46. If |θx|≧α1, the process proceeds to step S44, where the processing unit 210 calculates the transition rate γ. min((|θx|-α1) / (α2-α1),1) is a function that selects the smaller of (|θx|-α1) / (α2-α1) and 1, and sgn(θx) is a function that extracts the sign of θx. 0≦γ≦1 when α1≦|θx|≦α2, and γ=1 when α2<|θx|. In step S45, the processing unit 210 rotates the viewpoint of the virtual camera 52 by θcx=β×γ degrees around the x-axis with the gaze point RC as the center. Next, the process proceeds to step S46.

[0066] In step S46, the processing unit 210 judges whether the bending angle around the y axis is |θy|≧α1. If |θy|<α1, the processing unit 210 proceeds to step S49. If |θy|≧α1, the processing unit 210 calculates the transition rate γ in step S47. 48 In the above, the processing unit 210 performs the following with respect to the gaze point RC. y The viewpoint of virtual camera 52 is rotated around the axis by θcy=β×γ degrees. Next, the process proceeds to step S49.

[0067] In step S49, the processing unit 210 generates a CG image of the endoscope 3D model as seen by the virtual camera 52. In step S50, the processing unit 210 displays the generated CG image on the display 20 together with the endoscope image.

[0068] In the above embodiment, when the bending angle θr is smaller than the first threshold value α1, the processing unit 210 fixes the rotation angle θc to the first fixed angle. In the example of this embodiment, the first fixed angle is 0 degrees, but is not limited to this. When the bending angle θr is between the first threshold value α1 and the second threshold value α2 larger than the first threshold value α1, the processing unit 210 changes the rotation angle θc from the first fixed angle to the second fixed angle β larger than the first fixed angle according to the bending angle θr. When the bending angle θr is larger than the second threshold value α2, the processing unit 210 fixes the rotation angle θc to the second fixed angle β.

[0069] According to this embodiment, the difference between the imaging direction in the endoscopic image and the imaging direction in the CG image can be made smaller than when the viewpoint is not rotated, so that intuitive bending operation is possible. In addition, since an effect is obtained that shows that the viewpoint of the virtual camera has been intentionally changed and the meaning of the viewpoint switching, it is easy for the surgeon to understand the bending state and bending operation direction when looking at the CG image.

[0070] 5. Fourth embodiment In the fourth embodiment, it is possible to switch between a fixed mode in which the viewpoint of the virtual camera 52 is fixed regardless of the bending angle, and a rotation mode in which the viewpoint of the virtual camera 52 is rotated according to the bending angle.

[0071] The operation unit 400 is configured to allow an operation input for switching between a fixed mode and a rotation mode. When the mode is switched by the operation unit 400, the processing unit 210 sets the fixed mode or the rotation mode in response to the input. In the fixed mode, the processing unit 210 sets, for example, θc=0 degrees, and in the rotation mode, performs viewpoint rotation according to any of the first to third embodiments described above and the fifth and sixth embodiments described below.

[0072] In the above embodiment, the processing unit 210 can switch between a rotation mode in which a rotation process is performed to rotate the line of sight direction SDc of the virtual camera 52, and a fixed mode in which the rotation angle θc is fixed regardless of the shape of the manipulator.

[0073] According to this embodiment, it is possible to switch between the rotation mode and the fixed mode depending on the surgeon. This makes it possible to accommodate both surgeons who can easily grasp the state and direction of the bending operation from a fixed viewpoint and surgeons who can easily grasp the state and direction of the bending operation from a rotating viewpoint.

[0074] 6. Fifth embodiment In the fifth embodiment, when the curvature angle θx around the x axis is larger than the curvature angle θy around the y axis, the viewpoint of the virtual camera 52 is rotated around the x axis, and when the curvature angle θy around the y axis is larger than the curvature angle θx around the x axis, the viewpoint of the virtual camera 52 is rotated around the y axis. Below, an example in which the fifth embodiment is combined with the second and third embodiments will be described, but the fifth embodiment may also be combined with the first embodiment.

[0075] 14 is a flowchart when the fifth embodiment is combined with the second embodiment. In step S61, the processing unit 210 estimates the bending angle θr of the endoscope 10 from the wire pulling amount detected by the sensor 103.

[0076] In step S62, the processing unit 210 sets the viewpoint of the virtual camera 52 at a predetermined position behind the endoscope. Specifically, the processing unit 210 sets the gaze point RC at the base end of the bending portion, and sets the angle of the line of sight SDc to θc = 0 degrees.

[0077] In step S63, the processing unit 210 determines whether the curvature angle around the x-axis is equal to or greater than the curvature angle around the x-axis, that is, whether |θx|≧|θy|. If |θx|≧|θy|, in step S64, the processing unit 210 determines whether the curvature angle around the x-axis is |θx|≧α. If |θx|<α, the process proceeds to step S68. If |θx|≧α, in step S65, the processing unit 210 rotates the viewpoint of the virtual camera 52 by β degrees around the x-axis with the gaze point RC as the center. Next, the process proceeds to step S68.

[0078] If |θx|<|θy| in step S63, then in step S66, the processing unit 210 determines whether the curvature angle around the y axis is |θy|≧α. If |θy|<α, then the process proceeds to step S68. If |θy|≧α, then in step S67, the processing unit 210 rotates the viewpoint of the virtual camera 52 by β degrees around the y axis with the gaze point RC as the center. Then, the process proceeds to step S68.

[0079] In step S68, the processing unit 210 generates a CG image of the endoscope 3D model as seen by the virtual camera 52. In step S69, the processing unit 210 displays the generated CG image on the display 20 together with the endoscope image.

[0080] 15 is a flowchart when the fifth embodiment is combined with the third embodiment. In step S81, the processing unit 210 estimates the bending angle θr of the endoscope 10 from the wire pulling amount detected by the sensor 103.

[0081] In step S82, the processing unit 210 sets the viewpoint of the virtual camera 52 at a predetermined position behind the endoscope. Specifically, the processing unit 210 sets the gaze point RC at the base end of the bending portion, and sets the angle of the line of sight direction SDc to θcx = θcy = 0 degrees.

[0082] In step S83, the processing unit 210 determines whether the curvature angle around the x-axis is y In step S86, the processing unit 210 rotates the viewpoint of the virtual camera 52 by θcx=β×γ degrees around the x-axis, centering on the gaze point RC. Next, the processing unit 210 proceeds to step S90. If |θx|≧|θy|, in step S84, the processing unit 210 determines whether the curvature angle around the x-axis is |θx|≧α1. If |θx|<α1, the processing unit 210 proceeds to step S90. If |θx|≧α1, in step S85, the processing unit 210 calculates the transition rate γ. In step S86, the processing unit 210 rotates the viewpoint of the virtual camera 52 by θcx=β×γ degrees around the x-axis, centering on the gaze point RC. Next, the processing unit 210 proceeds to step S90.

[0083] If |θx|<|θy| in step S83, then in step S87, the processing unit 210 determines whether the curvature angle around the y-axis is |θy|≧α1. If |θy|<α1, then the processing unit 210 proceeds to step S90. If |θy|≧α1, then in step S88, the processing unit 210 calculates the transition rate γ. In step S89, the processing unit 210 rotates the viewpoint of the virtual camera 52 by θcy=β×γ degrees around the x-axis with the gaze point RC as the center. Next, the processing unit 210 proceeds to step S90.

[0084] In step S90, the processing unit 210 generates a CG image of the endoscope 3D model viewed from the virtual camera 52. In step S91, the processing unit 210 displays the generated CG image on the display 20 together with the endoscope image.

[0085] In the above embodiment, the acquisition unit 220 acquires information on a first bending angle θx around a first axis and a second bending angle θy around a second axis intersecting the first axis. In the example of this embodiment, the x-axis corresponds to the first axis, and the y-axis corresponds to the second axis. The processing unit 210 performs a rotation process at a rotation angle θc according to the larger bending angle of the first bending angle θx and the second bending angle θy.

[0086] According to this embodiment, the viewpoint of the virtual camera 52 rotates only around the x-axis or the y-axis, so viewpoint rotation in the CG image is simplified compared to when the viewpoint rotates around both axes. This is expected to make it easier to visually recognize curved shapes from the CG image.

[0087] 7. Sixth embodiment 16 shows an example of the configuration of the bending portion in the sixth embodiment. The bending portion 40 includes a first bending portion 41 and a second bending portion 42. The first bending portion 41 is also called a distal bending portion, and the second bending portion 42 is also called a proximal bending portion.

[0088] The base end of the second bending section 42 is the base end of the bending section 40 and is connected to the tip of the flexible section. The base end of the first bending section 41 is connected to the tip of the second bending section 42, and the rigid section and the imaging section 30 are provided at the tip of the first bending section 41. In the fifth embodiment, an electric endoscope is assumed, and the first bending section 41 and the second bending section 42 are controlled according to a mode described later in FIG. 17. Depending on the mode, the first bending section 41 and the second bending section 42 may be controlled independently or in cooperation with each other.

[0089] 16, the axial direction of the tip of the flexible section is NDm, the axial direction of the tip of the second bending section 42 is ID2, and the axial direction of the tip of the first bending section 41 is ID1. In this case, the angle between ID1 and ID2 is the bending angle θ1 of the first bending section 41, and the angle between ID2 and NDm is the bending angle θ2 of the second bending section 42. The bending angle of the entire bending section 40 is θA=θ1+θ2, which is referred to as the overall bending angle.

[0090] FIG. 17 shows an example of a control mode of two-stage bending. The control device 100 has a distal end bending mode, a proximal end bending mode, a pseudo one-stage mode, and a parallel mode as control modes of two-stage bending. The operation unit 400 is configured to be able to accept a mode switching operation and a bending operation in each mode. The control device 100 also includes a first motor for bending the first bending portion 41 and a second motor for bending the second bending portion 42, and drives the first bending portion 41 and the second bending portion 42 to bend by controlling the first motor and the second motor according to an operation input of the operation unit 400. The processing unit 210 switches the viewpoint control of the virtual camera 52 according to the mode set by the operation unit 400.

[0091] In the distal end bending mode, the bending angle θ2 of the second bending portion 42 is fixed, and the bending angle θ1 of the first bending portion 41 is variably controlled in response to an operational input.

[0092] In the proximal end bending mode, the bending angle θ1 of the first bending portion 41 is fixed, and the bending angle θ2 of the second bending portion 42 is variably controlled in response to an operational input.

[0093] In the distal end bending mode and the proximal end bending mode, the processing unit 210 sets a fixation point RC at the distal end of the second bending portion 42, and sets the line of sight direction of the virtual camera 52 to an angle θc=θ2. Note that the fixation point RC may be within the second bending portion 42, for example, near the distal end of the second bending portion 42. In addition, the viewpoint of the virtual camera 52 is not limited to θc=θ2, and the methods described in the second to fifth embodiments may be combined.

[0094] Since the movable range of the second bending portion 42 is larger than that of the first bending portion 41, a rough bend is determined in the base end bending mode, and then the bend is adjusted in the tip bending mode based on the rough bend and treatment is performed. For this reason, by determining the viewpoint according to the bend of the second bending portion 42, which is the reference, that is, by setting the line of sight of the virtual camera 52 to an angle θc=θ2, the surgeon can easily intuitively grasp the bending operation.

[0095] In the pseudo single-stage mode, the first bending portion 41 and the second bending portion 42 are cooperatively controlled to behave as one bending portion. For example, when a left bending operation is performed, θ1 and θ2 are both controlled in the + direction, and when a right bending operation is performed, θ1 and θ2 are both controlled in the - direction. As a result, the bending portion as a whole is bent at a bending angle θA=θ1+θ2.

[0096] In the parallel mode, the first bending portion 41 and the second bending portion 42 are controlled so that the imaging direction moves in parallel. For example, when operated to the left, θ2 is controlled in the + direction and θ1 is controlled in the - direction. At this time, by controlling so that θ1=-θ2, the imaging unit moves in parallel to the left while the imaging direction is kept constant.

[0097] In the pseudo single-stage mode and the parallel mode, the processing unit 210 sets a focus point RC at the tip of the first bending portion 41, and sets the line of sight direction of the virtual camera 52 to an angle θc=θA. Note that the focus point RC may be within the first bending portion 41, for example, near the tip of the first bending portion 41. In addition, the viewpoint of the virtual camera 52 is not limited to θc=θA, and the methods described in the second to fifth embodiments may be combined.

[0098] In the above embodiment, the bending portion 40 includes a first bending portion 41 and a second bending portion 42 provided on the proximal side of the first bending portion 41. The processing portion 210 performs a rotation process at a rotation angle θc according to a second bending portion bending angle θ2 which is the bending angle of the second bending portion 42, or an overall bending angle θA which is the bending angle of the bending portion 40 as a whole.

[0099] According to this embodiment, in two-stage bending by the first bending portion 41 and the second bending portion 42, the viewpoint of the virtual camera 52 can be made to follow the imaging direction according to the bending state. This makes it possible to intuitively grasp the bending operation direction even in two-stage bending.

[0100] In this embodiment, the processing unit 210 performs a rotation process at a rotation angle θc according to the second bending portion bending angle θ2. Specifically, this rotation process is performed in a distal end bending mode and a proximal end bending mode in which the first bending portion 41 and the second bending portion 42 are individually controlled.

[0101] By rotating the viewpoint according to the bending angle θ2 of the second bending portion 42 that determines the rough bending that serves as a reference, the surgeon can easily intuitively grasp the bending operation.

[0102] Furthermore, in this embodiment, the processing unit 210 performs rotation processing at a rotation angle θc according to the overall bending angle θA when the bending direction of the first bending portion 41 is controlled to be the same direction as or opposite to the bending direction of the second bending portion 42. Specifically, this rotation processing is performed in the pseudo single-stage mode and the parallel mode in which the first bending portion 41 and the second bending portion 42 are controlled in cooperation with each other.

[0103] When the first bending portion 41 and the second bending portion 42 are controlled in coordination, it is assumed that it will be easier to grasp the bending state from the CG image if the viewpoint is controlled according to the direction in which the tip of the bending portion 40 faces.

[0104] In this embodiment, the processing unit 210 performs rotation processing at a rotation angle θc corresponding to the second bending angle θ2 in the first bending mode in which the second bending portion 42 is controlled to be bent. The processing unit 210 performs rotation processing at a rotation angle θc corresponding to the entire bending angle θA in the second bending mode in which the bending direction of the first bending portion 41 is controlled to be the same as the bending direction of the second bending portion 42, or in the third bending mode in which the bending direction of the first bending portion 41 is controlled to be the opposite direction to the bending direction of the second bending portion 42. In this embodiment, the base end bending mode corresponds to the first bending mode, the pseudo single-stage mode corresponds to the second bending mode, and the parallel mode corresponds to the third bending mode.

[0105] According to this embodiment, it is possible to perform viewpoint rotation appropriate for each of the various two-stage bending control modes. This makes it possible to present the surgeon with a CG display of an appropriate 3D model in each bending control mode.

[0106] Although the present embodiment and its modified examples have been described above, the present disclosure is not limited to each embodiment or its modified example as it is, and in the implementation stage, the components can be modified and embodied within the scope of the gist. In addition, a plurality of components disclosed in each of the above-mentioned embodiments and modified examples can be appropriately combined. For example, some components may be deleted from all the components described in each embodiment or modified example. Furthermore, the components described in different embodiments or modified examples may be appropriately combined. In this way, various modifications and applications are possible within the scope of the gist of the present disclosure. In addition, a term described at least once in the specification or drawings together with a different term having a broader meaning or the same meaning can be replaced with the different term in any part of the specification or drawings. [Explanation of symbols]

[0107] 10 endoscope, 20 display, 30 imaging unit, 40 bending unit, 41 first bending unit, 42 second bending unit, 50 3D model, 52 virtual camera, 100 control device, 101 motor, 102 drive circuit, 103 sensor, 104 image interface, 200 image processing device, 210 processing unit, 220 acquisition unit, 300 endoscope system, 400 operation unit, IDm, IDr imaging direction, RC gaze point, SDc line of sight direction, θc rotation angle, θm, θr bending angle

Claims

1. an acquisition unit that acquires information on a bending angle of a manipulator having a bending portion whose bending angle is variable and an imaging unit provided at a tip end thereof; a processing unit that displays, on a display, a CG image of a manipulator model viewed from a virtual camera in an object space based on information about the bending angle; Including, The processing unit includes: The image processing device further comprises a rotation process for rotating the line of sight of the virtual camera by a rotation angle corresponding to the curvature angle.

2. In claim 1, The manipulator includes a flexible portion, The bending portion is connected to a tip of the flexible portion, The imaging unit is provided at a tip of the bending portion, The processing unit displays on the display the CG image of the manipulator model showing at least a portion of the tip side of the flexible section and the bending section as seen from the virtual camera with the line of sight rotated by the rotation angle corresponding to the bending angle.

3. In claim 1, The processing unit includes: The image processing device is characterized in that the rotation angle is changed in synchronization with the curvature angle.

4. In claim 1, The processing unit includes: Fixing the rotation angle at a first angle until the bending angle exceeds a threshold; When the bending angle exceeds the threshold value, the rotation angle is switched to a second angle that is larger than the first angle.

5. In claim 1, The processing unit includes: When the bending angle is smaller than a first threshold, the rotation angle is fixed to a first fixed angle; When the bending angle is between the first threshold value and a second threshold value that is greater than the first threshold value, the rotation angle is changed from the first fixed angle to a second fixed angle that is greater than the first fixed angle in accordance with the bending angle; When the bending angle is greater than the second threshold value, the rotation angle is fixed to the second fixed angle.

6. In claim 1, The processing unit includes: The image processing device is characterized in that it is possible to switch between a rotation mode in which the rotation processing is performed and a fixed mode in which the rotation angle is fixed regardless of the curvature angle.

7. In claim 1, The acquisition unit is obtaining information on a first bending angle about a first axis and a second bending angle about a second axis intersecting the first axis; The processing unit includes: The image processing device according to claim 1, wherein the rotation process is performed at a rotation angle corresponding to a larger one of the first bending angle and the second bending angle.

8. In claim 1, The curved portion is A first curved portion and a second curved portion provided on a proximal end side of the first curved portion, The processing unit includes: The image processing device performs the rotation processing at a rotation angle corresponding to a second bending portion bending angle, which is a bending angle of the second bending portion, or an overall bending angle, which is a bending angle of the bending portion as a whole.

9. In claim 8, The processing unit includes: The image processing device according to claim 1, wherein the rotation process is performed at a rotation angle corresponding to a bending angle of the second bending portion.

10. In claim 8, The processing unit includes: The image processing device performs the rotation process at the rotation angle corresponding to the overall bending angle when the bending direction of the first bending portion is controlled to be the same direction as or opposite to the bending direction of the second bending portion.

11. In claim 8, The processing unit includes: In a first bending mode in which the second bending portion is controlled to be bent, the rotation process is performed at the rotation angle corresponding to a bending angle of the second bending portion, an image processing device that performs the rotation process at the rotation angle corresponding to the overall bending angle when in a second bending mode in which the bending direction of the first bending part is controlled to be the same direction as the bending direction of the second bending part, or in a third bending mode in which the bending direction of the first bending part is controlled to be the opposite direction to the bending direction of the second bending part.

12. In claim 1, The image processing device according to claim 1, wherein the rotation process is a process of rotating the line of sight of the virtual camera around a set point set on the manipulator model.

13. A manipulator having a bending portion and an imaging unit provided at a tip portion thereof; An image processing device; A display and Including, The image processing device includes: displaying, on the display, a CG image of a manipulator model in a line of sight direction of a virtual camera that follows the imaging direction of the imaging unit; A manipulator system comprising: a manipulator unit configured to change a line of sight of the virtual camera in accordance with a bending angle of the bending portion.

14. In claim 13, The manipulator includes a flexible portion, The bending portion is connected to a tip of the flexible portion, The imaging unit is provided at a tip of the bending portion, The image processing device displays on the display the CG image of the manipulator model showing at least a portion of the tip side of the flexible section and the bending section as seen from the virtual camera whose line of sight direction follows the imaging direction according to the bending angle.

15. In claim 13, The image processing device includes: A manipulator system characterized in that the line of sight direction is changed in synchronization with the bending angle.

16. In claim 13, The curved portion is A first curved portion and a second curved portion provided on a proximal end side of the first curved portion, The image processing device includes: A manipulator system characterized in that the line of sight direction of the virtual camera is changed according to a second bending portion bending angle, which is a bending angle of the second bending portion, or an overall bending angle, which is a bending angle of the bending portion as a whole.

17. 1. A method of operating an image processing device, comprising: a step in which the image processing device acquires information on a bending angle of a manipulator having a bending portion whose bending angle is variable and an imaging unit is provided at a tip portion of the manipulator; a step of performing a rotation process by the image processing device to rotate a line of sight direction of a virtual camera in an object space by a rotation angle according to the curvature angle based on information of the curvature angle; a step of displaying, on a display, a CG image of a manipulator model seen from the virtual camera in the object space by the image processing device; 2. A method for operating an image processing device, comprising:

18. In claim 17, The manipulator includes a flexible portion, The bending portion is connected to a tip of the flexible portion, The imaging unit is provided at a tip of the bending portion, A method for operating an image processing device, characterized in that the image processing device performs a step of displaying, on the display, the CG image of the manipulator model showing at least a portion of the tip side of the flexible section and the bending section as seen from the virtual camera with the line of sight rotated by the rotation angle corresponding to the bending angle.

19. 1. A method of operating an image processing device, comprising: a step of displaying, by the image processing device, a CG image of a manipulator model in a line of sight direction of a virtual camera that follows an imaging direction of an imaging unit provided at a tip portion of a manipulator having a bending portion, on a display; The image processing device changes the line of sight direction of the virtual camera according to a bending angle of the bending portion; 2. A method for operating an image processing device, comprising:

20. 20. In claim 19, The manipulator includes a flexible portion, The bending portion is connected to a tip of the flexible portion, The imaging unit is provided at a tip of the bending portion, A method for operating an image processing device, characterized in that the image processing device performs a step of displaying, on the display, the CG image of the manipulator model showing at least a portion of the tip side of the flexible section and the bending section as seen from the virtual camera whose line of sight direction follows the imaging direction according to the bending angle.

Images