Three-dimensional image display device, three-dimensional image display method, and program

Abstract

This three-dimensional image display device has a control unit. The control unit displays a three-dimensional shape of a subject on a display on the basis of three-dimensional image data including three-dimensional coordinates of three or more points on the subject. The control unit sets a reference plane on the basis of one or more points included in the three or more points, and divides the three-dimensional image data into a first region and a second region with the reference plane as a boundary. The control unit makes the display state of the three-dimensional shape of the subject in one of the first region and the second region different from the display state of the three-dimensional shape of the subject in the other of the first region and the second region in order to display a cross-section of the subject set on the basis of the reference plane.

Description

Three-dimensional image display device, three-dimensional image display method, and program

[0001] The present invention relates to a three-dimensional image display device, a three-dimensional image display method, and a program. This application claims priority to Japanese Patent Application No. 2024-118529, filed on July 24, 2024, the contents of which are incorporated herein by reference.

[0002] Industrial endoscope devices are used for inspecting (endoscopic inspection) for internal abnormalities, corrosion, etc. in boilers, pipes, aircraft engines, heat exchangers, etc. The endoscope device disclosed in Patent Document 1 displays the shape of a cross section formed when an object to be observed is cut along a plane passing through two points set on an image of the object to be observed.

[0003] Patent No. 4343341

[0004] In one method (FIG. 9) disclosed in Patent Document 1, an image of an object is displayed and the outline of the cross-sectional shape is displayed on the image. In another method (FIG. 12) disclosed in Patent Document 1, the cross-sectional shape as seen from a viewpoint different from the viewpoint from which the object is photographed is displayed. However, in the method disclosed in Patent Document 1, the cross-sectional shape is displayed at a position different from the original position of the cross-sectional shape in the image of the object, or the cross-sectional shape is displayed in an image other than the image of the object. As a result, it is difficult for the user to intuitively understand the cross-sectional shape of the object.

[0005] An object of the present invention is to provide a three-dimensional image display device, a three-dimensional image display method, and a program that can facilitate intuitive understanding of the three-dimensional shape of a subject.

[0006] A three-dimensional image display device according to an aspect of the present invention includes a control unit. The control unit acquires three-dimensional image data including three-dimensional coordinates of three or more points on a subject calculated based on an endoscopic image of the subject. The control unit displays a three-dimensional shape of the subject on a display based on the three-dimensional image data. The control unit sets a reference plane based on one or more points included in the three or more points. The control unit divides the three-dimensional image data into a first region and a second region using the reference plane as a boundary. The control unit differentiates the display state of the three-dimensional shape of the subject in one of the first region and the second region from the display state of the three-dimensional shape of the subject in the other of the first region and the second region in order to display a cross-section of the subject set based on the reference plane.

[0007] In the three-dimensional image display device according to this aspect of the present invention, the control unit may hide one of the first area and the second area.

[0008] In the three-dimensional image display device according to this aspect of the present invention, the control unit may make the transparency of one of the first region and the second region higher than the transparency of the other of the first region and the second region.

[0009] In a three-dimensional image display device according to an aspect of the present invention, the control unit may acquire area information indicating one of the first area and the second area, and may change the display state of the area indicated by the area information.

[0010] In the three-dimensional image display device according to this aspect of the present invention, the control unit may change the display state of the cross section so that the cross section is emphasized.

[0011] In the three-dimensional image display device according to this aspect of the present invention, the control unit may receive an instruction to rotate the three-dimensional shape, and may display the three-dimensional shape rotated in accordance with the instruction on the display.

[0012] In a three-dimensional image display device according to an aspect of the present invention, the control unit may accept an instruction to enlarge or reduce the three-dimensional shape, and may display the three-dimensional shape enlarged or reduced in accordance with the instruction on the display.

[0013] In a three-dimensional image display device according to an aspect of the present invention, the control unit may display, based on the three-dimensional image data, a three-dimensional shape of the entire or part of the subject next to the three-dimensional shape of the subject, which includes the first region and the second region having a display state different from the display state of the first region.

[0014] In a three-dimensional image display device according to an aspect of the present invention, the control unit may set a reference point based on one or more points included in the three or more points, and may set the reference plane based on the reference point.

[0015] In the three-dimensional image display device according to this aspect of the present invention, the control unit may set two or more reference points.

[0016] In the three-dimensional image display device according to this aspect of the present invention, the control unit may set a reference figure based on the one or more points, and may set the reference plane based on the reference figure.

[0017] In a three-dimensional image display device according to an aspect of the present invention, the control unit may detect a characteristic area in the endoscopic image, and may set the reference plane based on one or more points among the three or more points that are included in the characteristic area.

[0018] In a three-dimensional image display device according to this aspect of the present invention, the control unit may set a measurement reference based on one or more points included in the three or more points, and may calculate the distance between the measurement reference and a point on the cross section.

[0019] In a three-dimensional image display device according to an aspect of the present invention, the control unit may calculate the distance between the measurement reference and each of two or more points on the cross section, and may identify the point among the two or more points where the distance is maximum or minimum.

[0020] In the three-dimensional image display device according to this aspect of the present invention, the control unit may superimpose information indicating the point at which the distance is maximum or minimum on the three-dimensional shape.

[0021] In a three-dimensional image display device according to an aspect of the present invention, the control unit may set a range including a portion of the cross-section in the three-dimensional image data, and may calculate the distance between the measurement reference and each of two or more points included in the range.

[0022] In the three-dimensional image display device according to an aspect of the present invention, the endoscopic image may include a first image and a second image that are stereo images, and the control unit may generate the three-dimensional image data based on the first image and the second image.

[0023] In the three-dimensional image display device according to an aspect of the present invention, the endoscopic image may be generated based on an optical image of the subject formed by a monocular optical system, and the control unit may generate the three-dimensional image data based on two or more of the endoscopic images.

[0024] In the three-dimensional image display device according to this aspect of the present invention, the control unit may acquire the three-dimensional image data from a storage medium.

[0025] A three-dimensional image display method according to an aspect of the present invention includes the following processes: a control unit acquires three-dimensional image data including three-dimensional coordinates of three or more points on a subject calculated based on an endoscopic image of the subject; the control unit displays a three-dimensional shape of the subject on a display based on the three-dimensional image data; the control unit sets a reference plane based on one or more points included in the three or more points; the control unit divides the three-dimensional image data into a first region and a second region using the reference plane as a boundary; and the control unit differentiates the display state of the three-dimensional shape of the subject in one of the first region and the second region from the display state of the three-dimensional shape of the subject in the other of the first region and the second region in order to display a cross-section of the subject set based on the reference plane.

[0026] A program according to an aspect of the present invention causes a computer to execute the following processes: the computer acquires three-dimensional image data including three-dimensional coordinates of three or more points on a subject calculated based on an endoscopic image of the subject; the computer displays a three-dimensional shape of the subject on a display based on the three-dimensional image data; the computer sets a reference plane based on one or more points included in the three or more points; the computer divides the three-dimensional image data into a first region and a second region using the reference plane as a boundary; and the computer differentiates the display state of the three-dimensional shape of the subject in one of the first region and the second region from the display state of the three-dimensional shape of the subject in the other of the first region and the second region in order to display a cross-section of the subject set based on the reference plane.

[0027] According to the above aspects, the three-dimensional image display device, the three-dimensional image display method, and the program can facilitate intuitive understanding of the three-dimensional shape of the subject.

[0028] FIG. 1 is a block diagram showing an example of the configuration of an endoscope system according to a first embodiment of the present invention. FIG. 2 is a flowchart showing an example of the procedure of three-dimensional (3D) shape display processing in the first embodiment of the present invention. FIG. 3 is a diagram showing an example of an image displayed on a display included in the endoscope system according to the first embodiment of the present invention. FIG. 4 is a diagram showing an example of an image displayed on a display included in the endoscope system according to the first embodiment of the present invention. FIG. 5 is a diagram showing an example of an image displayed on a display included in the endoscope system according to the first embodiment of the present invention. FIG. 6 is a diagram showing an example of a 3D space defined in 3D image data in the first embodiment of the present invention. FIG. 7 is a diagram showing an example of an image displayed on a display included in the endoscope system according to the first embodiment of the present invention. FIG. 8 is a diagram showing an example of an image displayed on a display included in the endoscope system according to the first embodiment of the present invention. FIG. 10 is a diagram showing an example of surface information in a second embodiment of the present invention. FIG. 11 is a diagram showing an example of a 3D shape used in the 3D shape display processing in the second embodiment of the present invention. FIG. 12 is a diagram showing an example of a 3D shape used in the 3D shape display processing in the second embodiment of the present invention. FIG. 13 is a diagram showing an example of a 3D shape used in the 3D shape display processing in the second embodiment of the present invention. FIG. 14 is a flowchart showing an example of the procedure of measurement processing in a third embodiment of the present invention. FIG. 15 is a diagram showing an example of a 3D shape used in the measurement processing in the third embodiment of the present invention.Fig. 10 is a diagram showing an example of an image displayed on a display included in an endoscope system according to a modified example of the third embodiment of the present invention. Fig. 11 is a flowchart showing an example of a procedure for rotation processing in a fourth embodiment of the present invention. Fig. 12 is a flowchart showing an example of a procedure for 3D shape display processing in a fifth embodiment of the present invention. Fig. 13 is a flowchart showing an example of a procedure for 3D shape display processing in the fifth embodiment of the present invention.

[0029] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the accompanying drawings. An endoscope system will be described below as an example of a three-dimensional (3D) image display device.

[0030] First Embodiment Fig. 1 shows an example of the configuration of an endoscope system 1 according to a first embodiment of the present invention. The endoscope system 1 shown in Fig. 1 has an insertion section 2, a scope unit 3, a base unit 4, and a main unit 5. The insertion section 2, the scope unit 3, and the base unit 4 constitute an endoscope device 10. The main unit 5 is an operation device.

[0031] The insertion section 2 is inserted into the inside of the subject to be observed. The subject is an industrial product. The insertion section 2 is a long, thin tube that is bendable. A user performs an insertion operation to insert the insertion section 2 into the subject. An optical adapter is attached to the tip of the insertion section 2. The insertion section 2 acquires an optical image of the inside of the subject. The insertion section 2 has an imaging section 20, a bending section 21, and an illumination window 22.

[0032] The imaging unit 20 is disposed in the distal end portion 2a including the distal end of the insertion portion 2. The imaging unit 20 is an image sensor such as a CCD (Charge-Coupled Device) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor. The imaging unit 20 generates an image based on an optical image acquired by the insertion portion 2. The image generated by the imaging unit 20 is output to the scope unit 3.

[0033] The bending section 21 bends the insertion section 2 in the upward (U), downward (D), leftward (L), or rightward (R) direction. Alternatively, the bending section 21 bends the insertion section 2 in the upward-left (UL), upward-right (UR), downward-left (DL), or downward-right (DR) direction.

[0034] Illumination light is generated by a light source 35 included in the scope unit 3 and output to the distal end portion 2a through a light guide disposed in the insertion portion 2. The illumination light is irradiated from the illumination window 22 into the inside of the subject.

[0035] The scope unit 3 has an imaging drive circuit 30, an image processing unit 31, a UD drive unit 32, an RL drive unit 33, a bending control unit 34, a light source 35, and a light source control unit 36. The base unit 4 has a control unit 40, a communication unit 41, a volatile memory 42, and a non-volatile memory 43.

[0036] The imaging drive circuit 30 controls the imaging unit 20 and outputs the image output from the imaging unit 20 to the image processing unit 31. The image processing unit 31 performs image processing such as noise reduction on the image output from the imaging unit 20 and outputs the image to the control unit 40.

[0037] The UD driving unit 32 is connected to a UD bending wire for bending the bending portion 21 in the U direction or the D direction. The UD driving unit 32 has a motor, and bends the bending portion 21 in the U direction or the D direction by pulling the UD bending wire. The RL driving unit 33 is connected to an RL bending wire for bending the bending portion 21 in the R direction or the L direction. The RL driving unit 33 has a motor, and bends the bending portion 21 in the R direction or the L direction by pulling the RL bending wire. The bending control unit 34 controls the UD driving unit 32 and the RL driving unit 33.

[0038] The UD drive unit 32 and the RL drive unit 33 can operate simultaneously. For example, the UD drive unit 32 and the RL drive unit 33 can bend the bending portion 21 in the UL direction.

[0039] The light source 35 is a light-emitting diode (LED) or the like, and generates illumination light. The illumination light is output from the light source 35 to a light guide. The light source control unit 36 ​​controls the light source 35.

[0040] The control unit 40 controls each unit of the scope unit 3 and the base unit 4. At least one of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36 ​​may be configured with at least one of a processor and a logic circuit. For example, the processor is at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). At least one of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36 ​​may include one or more processors. At least one of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36 ​​may include one or more logic circuits.

[0041] The computer of the endoscope system 1 may load a program and execute the loaded program. The program includes instructions that define the operation of at least one of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36. In other words, at least one function of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36 ​​may be realized by software.

[0042] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer storing the program to the endoscope system 1 via a transmission medium or by transmission waves in the transmission medium. The "transmission medium" that transmits the program is a medium that has the function of transmitting information. Media that have the function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the functions described above. Furthermore, the above program may be a difference file (difference program). The functions described above may be realized by combining a program already recorded on the computer with the difference program.

[0043] The communication unit 41 has a communication circuit and performs wired or wireless communication with the main unit 5 for curvature control and the like. The communication unit 41 transmits images generated by the imaging unit 20 to the main unit 5. The volatile memory 42 is a RAM (Random Access Memory) or a DRAM (Dynamic RAM), for example. The volatile memory 42 stores various information processed by the control unit 40. The non-volatile memory 43 is a SRAM (Static RAM), an EPROM (Erasable Programmable Read-Only Memory), an EEPROM (Electrically Erasable Programmable ROM), or a flash memory. The non-volatile memory 43 may be detachable from the base unit 4. The non-volatile memory 43 stores images generated by the imaging unit 20 and various information processed by the control unit 40 .

[0044] The main unit 5 has a control unit 50, a display 51, a touch panel 52, operation buttons 53, a communication unit 54, a communication unit 55, a volatile memory 56, and a non-volatile memory 57. The main unit 5 may be an information terminal such as a smartphone or a tablet terminal.

[0045] The control unit 50 controls each part of the main unit 5. The control unit 50 may be composed of at least one of a processor and a logic circuit. The control unit 50 may include one or more processors. The control unit 50 may include one or more logic circuits. The computer of the endoscope system 1 may load a program and execute the loaded program. The program includes instructions that define the operation of the control unit 50. In other words, the functions of the control unit 50 may be realized by software. The program that realizes the functions of the control unit 50 may be realized in the same way as the program that realizes the functions of the control unit 40, etc.

[0046] The display 51 is a monitor such as an LCD (Liquid Crystal Display). The display 51 displays an image generated by the imaging unit 20. The touch panel 52 accepts operations for inputting information necessary for controlling the endoscope system 1. The touch panel 52 is arranged on the screen of the display 51. By operating the touch panel 52, the user can input to the endoscope system 1 instructions for changing the settings of the endoscope system 1, instructions necessary for operating the endoscope system 1, and the like.

[0047] The operation button 53 accepts various instructions from the user. By pressing the operation button 53, the user can input instructions regarding power or lighting to the endoscope system 1. The communication unit 54 performs wired or wireless communication with the base unit 4 for bending control and the like. The communication unit 54 receives images generated by the imaging unit 20 from the base unit 4. The communication unit 55 performs wired or wireless communication with the external device 11. The external device 11 is a remote control, a mouse, or the like.

[0048] The control unit 50 executes a 3D shape display process for displaying an image of the 3D shape of the subject on the display 51. An outline of the 3D shape display process will be described below.

[0049] The control unit 50 acquires 3D image data including 3D coordinates of three or more points on the subject. The 3D coordinates are defined in a 3D space corresponding to real space. The control unit 50 displays an image of the 3D shape of the subject on the display 51 based on the 3D image data. The control unit 50 accepts designation of one or more reference points on the subject. In the first embodiment, the control unit 50 accepts designation of two reference points.

[0050] The control unit 50 sets a reference plane (cutting reference) in the 3D space based on one or more reference points. Specifically, the control unit 50 sets a reference figure based on one or more reference points, and sets a reference plane based on the reference figure. The reference figure is a line, a plane, or a curved surface. In the first embodiment, an example will be described in which a boundary line is used as the reference figure.

[0051] The control unit 50 divides the entire area of ​​the subject into two areas by the reference plane. The cross section of the subject is formed at the intersection of the reference plane and the subject. The control unit 50 changes the display state of one of the two areas. In the first embodiment, the control unit 50 hides one of the two areas. As a result, the cross section of the subject becomes visible.

[0052] 2 shows an example of the procedure for the 3D shape display process. The operation of the endoscope system 1 will be described with reference to FIG.

[0053] The control unit 50 acquires 3D image data (step S100).

[0054] The control unit 50 executes the following process in step S100. First to third examples will be described below.

[0055] First, a first example will be described. The optical adapter in the first example is a stereo optical adapter with two fields of view. The optical adapter has a first optical system and a second optical system corresponding to the two fields of view. The first optical system and the second optical system form two optical images of the subject on the imaging unit 20. The imaging unit 20 generates a stereo image corresponding to the first optical image and the second optical image. The stereo image includes a pair of two images (a first image and a second image). That is, the stereo image includes an image of the subject viewed from a first viewpoint and an image of the subject viewed from a second viewpoint. The control unit 50 calculates the 3D coordinates of three or more points on the subject by using one or more stereo images generated by the imaging unit 20, and generates 3D image data including the 3D coordinates.

[0056] Next, a second example will be described. The optical adapter in the second example is a monocular optical adapter with one field of view. While the optical adapter in the first example forms two optical images of the subject, the optical adapter in the second example forms one optical image of the subject. The imaging unit 20 generates an image corresponding to the optical image formed by the optical adapter. The imaging unit 20 performs imaging from two or more different viewpoints and generates two or more images. The control unit 50 calculates the 3D coordinates of three or more points on the subject by using the two or more images generated by the imaging unit 20, and generates 3D image data including the 3D coordinates.

[0057] Next, a third example will be described. The 3D image data generated in the first or second example is stored in advance in the nonvolatile memory 57. The control unit 50 acquires the 3D image data from the nonvolatile memory 57.

[0058] After step S100, the control unit 50 generates a 3D image based on the 3D image data and outputs the 3D image to the display 51. The display 51 displays the 3D image (step S101). The 3D image is an image of the 3D shape of the subject.

[0059] FIG. 3 shows an example of a 3D image displayed on the display 51 in step S101. The control unit 50 displays the 3D image IMG10 shown in FIG. 3 on the display 51. The 3D image IMG10 includes a 3D shape SH10 of the subject. The control unit 50 displays an icon IC10 on the 3D image IMG10. The icon IC10 includes an illustration of the subject and prompts the user to set a first reference point. The icon IC10 also notifies the user of areas that will be hidden. For example, the icon IC10 notifies the user that an area in front of a reference plane that passes through two reference points will be hidden.

[0060] The control unit 50 displays a cursor CS10 on the 3D image IMG10. The user moves the cursor CS10 on the 3D image IMG10 by operating the touch panel 52. The user may also move the cursor CS10 on the 3D image IMG10 by operating the operation button 53, the external device 11, or the like.

[0061] The user operates the touch panel 52 or the like to input an instruction to set a first reference point to the endoscope system 1. The control unit 50 determines whether or not the instruction has been input (step S102). If the control unit 50 determines in step S102 that the instruction has not been input, the control unit 50 repeats step S102.

[0062] When the control unit 50 determines in step S102 that the instruction has been input, the control unit 50 accepts the instruction and sets a first reference point at the position of the cursor CS10 (step S103). The first reference point is one of three or more points included in the 3D image data. Information about the first reference point is stored in the volatile memory 56.

[0063] 4 shows an example of a 3D image displayed on the display 51 when the first reference point is set in step S103. Portions that are the same as those shown in FIG.

[0064] The control unit 50 displays an icon IC11 on the 3D image IMG10. The icon IC11 includes an illustration of the subject and prompts the user to set the second reference point. The icon IC11 also notifies the user of the area that will be hidden.

[0065] The user operates the touch panel 52 or the like to move the cursor CS10 on the 3D image IMG10. The user also operates the touch panel 52 or the like to input an instruction to set a second reference point to the endoscope system 1. The control unit 50 determines whether or not the instruction has been input (step S104).

[0066] If the control unit 50 determines in step S104 that the instruction has not been input, the control unit 50 highlights the line connecting the first reference point and the point on the cursor on the 3D image (step S109). After step S109, step S104 is executed.

[0067] 5 shows an example of a 3D image displayed on the display 51 in step S109. Portions that are the same as those shown in FIG. 4 will not be described.

[0068] The control unit 50 displays a line L10 on the 3D image IMG10. The line L10 is a straight line connecting the first reference point set in step S103 and the point indicated by the cursor CS10. The line L10 is displayed as a thick line. When the user moves the cursor CS10, the line L10 moves.

[0069] When the control unit 50 determines in step S104 that an instruction to set a second reference point has been input, the control unit 50 accepts the instruction and sets the second reference point at the position of the cursor CS10 (step S105). The second reference point is one of three or more points included in the 3D image data and is different from the first reference point. Information about the second reference point is stored in the volatile memory 56.

[0070] After step S105, the control unit 50 sets a reference plane that passes through the first reference point and the second reference point (step S106).

[0071] Step S106 will be described in detail. Fig. 6 shows the 3D space defined in the 3D image data. The 3D image data includes 3D coordinates of three or more points constituting the 3D shape SH11 of the subject. The 3D coordinates of the three or more points are composed of coordinate values ​​on the mutually orthogonal X-axis, Y-axis, and Z-axis.

[0072] The control unit 50 sets a first reference point RP10 in step S103, and sets a second reference point RP11 in step S105. The control unit 50 sets a boundary line BL10 that passes through the first reference point RP10 and the second reference point RP11 in step S106. The control unit 50 also sets a plane PL10 that includes the boundary line BL10 and is parallel to the Z axis in step S106. The plane PL10 is a reference plane.

[0073] After step S106, the control unit 50 divides the entire region of the 3D image data into a first region and a second region (step S107). Dividing the entire region into two regions means defining two regions within the entire region. In other words, dividing the entire region into two regions means assigning each point included in the entire region to one of the two regions. The boundary between the first region and the second region is the reference plane set in step S106. Each of the three or more points included in the 3D image data is included in the first region or the second region.

[0074] After step S107, the control unit 50 changes the display state of the 3D image displayed on the display 51. Specifically, the control unit 50 changes the display state of one of the first region and the second region. For example, the control unit 50 hides one of the first region and the second region (step S108). When step S108 is executed, the 3D shape display process shown in FIG. 2 ends.

[0075] Instead of hiding one of the first region and the second region, the control unit 50 may increase the transparency of one of the first region and the second region. For example, before step S108 is performed, the transparency of the first region and the second region is 0%. In step S108, the control unit 50 sets the transparency of one of the first region and the second region to a value greater than 0% and less than or equal to 100%. The control unit 50 may also set the transparency of one of the first region and the second region to 50% or more.

[0076] When the first region or the second region is hidden, the cross section hidden by the first region or the second region is displayed. When the transparency of one of the first region or the second region is increased, the cross section hidden by the first region or the second region becomes visible. This allows the user to easily understand the 3D shape of the cross section. Since the 3D shape of the cross section as well as the 3D shape of the surrounding area of ​​the cross section are displayed, the user can easily intuitively understand the 3D shape of the subject.

[0077] 7 shows an example of a 3D image displayed on the display 51 in step S108. The same parts as those shown in FIG. 5 will not be described.

[0078] For example, the first region is located on the positive side of the Y axis from the reference plane set in step S106. For example, the second region is located on the negative side of the Y axis from the reference plane. The control unit 50 hides one or more points included in the second region. As a result, the cross section of the 3D shape SH10 is visualized. The control unit 50 also displays a line L11 on the 3D image IMG10 to highlight the cross section. For example, the line L11 is displayed with a specific thickness and a specific color.

[0079] After step S108, the user may input an instruction to rotate, enlarge, or reduce the 3D image displayed on the display 51 by operating the touch panel 52 or the like. The control unit 50 may accept the instruction. When an instruction to rotate the 3D image is accepted, the control unit 50 may rotate the 3D image displayed on the display 51. When an instruction to enlarge or reduce the 3D image is accepted, the control unit 50 may enlarge or reduce the 3D image displayed on the display 51.

[0080] 8 shows an example of a 3D image displayed on the display 51 when an instruction to rotate the 3D image is received. Portions that are the same as those shown in FIG. 7 will not be described.

[0081] The control unit 50 receives an instruction to rotate the 3D image IMG10 shown in Fig. 7. At this time, the control unit 50 displays the 3D image IMG11 shown in Fig. 8 on the display 51. The 3D shape SH10 in the 3D image IMG11 is displayed at a different angle from the 3D shape SH10 in the 3D image IMG10.

[0082] 9 shows an example of a 3D image displayed on the display 51 when an instruction to enlarge the 3D image is received. Portions that are the same as those shown in FIG. 7 will not be described.

[0083] The control unit 50 receives an instruction to rotate the 3D image IMG11 shown in Fig. 8. At this time, the control unit 50 displays the 3D image IMG12 shown in Fig. 9 on the display 51. The 3D shape SH10 in the 3D image IMG12 is enlarged more than the 3D shape SH10 in the 3D image IMG11.

[0084] The user may operate the touch panel 52 with a finger, a stylus, or the like to input a reference point. After the first reference point is set, the user may perform a drag (swipe) operation to move the cursor, or may perform a drop (release) operation to input a second reference point. The user may also perform a drop (release) operation to specify the position of the second reference point, or may perform a long tap (click) operation to confirm the position.

[0085] The user may operate the operation button 53 to input the reference point. The main unit 5 may have a joystick, and the user may operate the joystick to input the reference point. The external device 11 may be a mouse, and the user may operate the mouse to input the reference point.

[0086] The control unit 50 may display an image of the 3D shape of the entire or part of the subject based on the 3D image data on the display 51. The control unit 50 may display the image of the 3D shape of the entire or part of the subject next to the image IMG10 shown in FIG.

[0087] The control unit 50 may display a two-dimensional (2D) cross-sectional shape outline showing the 3D shape in a particular cross section on the display 51. The control unit 50 may display the 2D cross-sectional shape outline next to the image IMG10 shown in FIG.

[0088] The control unit 50 may display a 3D color map that indicates the distance between a specific reference position and each point on the 3D shape using a color on the display 51. The control unit 50 may display the 3D color map next to the image IMG10 shown in FIG.

[0089] The control unit 50 may have a function to modify the reference plane as follows. For example, the control unit 50 cancels the setting of the first reference point or the second reference point in accordance with an instruction from the user. After the setting of the first reference point or the second reference point is canceled, the control unit 50 sets a new reference point in accordance with an instruction from the user. For example, after the setting of the first reference point is canceled, the control unit 50 sets a new first reference point. The control unit 50 sets a new reference plane based on two reference points including the newly set reference point.

[0090] The control unit 50 may correct the reference plane as follows. For example, the control unit 50 moves the line L11 shown in FIG. 7 in accordance with an instruction from the user. While the line L11 is moving, the control unit 50 sets a new reference plane that passes through the line L11. The instruction may be, for example, an instruction to directly move the line L11, or an instruction to move the line L11 by moving a reference point.

[0091] After the reference plane is corrected as described above, steps S107 and S108 shown in FIG. 2 are executed, and the cross section of the 3D shape of the subject is displayed again.

[0092] The control unit 50 may execute the following process instead of steps S102 to S105 and S109 shown in FIG. 2 . After step S101, the control unit 50 detects a characteristic region in the image used to generate the 3D image data. That is, the control unit 50 detects a characteristic region in a stereo image including two optical images formed by a stereo optical adapter. Alternatively, the control unit 50 detects a characteristic region in an image including an optical image formed by a monocular optical adapter. For example, the characteristic region is an abnormal region such as a scratch. The control unit 50 detects an abnormal region having a predetermined color or shape.

[0093] Machine learning may be used to detect abnormal regions. For example, deep learning may be used as machine learning. For example, a user observes an image acquired in a previous examination and determines whether a region shown in the image is abnormal or normal. The control unit 50 analyzes the features of the image using the image and correct answer data (teaching data) indicating the result of the user's judgment, and generates a trained model. The non-volatile memory 57 stores the trained model.

[0094] After the image capturing unit 20 captures an image, the control unit 50 inputs the image into the trained model and determines whether the area in the image is abnormal or normal. If the area is abnormal, the control unit 50 extracts the area as an abnormal area.

[0095] The positions on the image used to generate the 3D image data and each point included in the 3D image data are associated with each other. The control unit 50 identifies points in the 3D image data that correspond to each point included in the feature region. The control unit 50 sets a reference plane based on one or more identified points.

[0096] The control unit 50 may display 3D shapes of two or more cross sections on the display 51. For example, the control unit 50 executes the following process. In step S106 shown in FIG. 2 , the control unit 50 sets a first reference plane that includes a boundary line connecting the first reference point and the second reference point and is parallel to the Z axis. The control unit 50 also sets a line that is perpendicular to the boundary line connecting the first reference point and the second reference point set in the process shown in FIG. 2 . The set line passes through two or more points on the 3D shape of the subject. The control unit 50 then sets a second reference plane that includes the set line and is parallel to the Z axis. The control unit 50 executes steps S107 and S108 shown in FIG. 2 for each of the first reference plane and the second reference plane, and displays the 3D shapes of the two cross sections on the display 51.

[0097] The control unit 50 may receive an instruction to set three or more reference points. The control unit 50 may select two reference points from the three or more reference points and set a reference plane based on the two reference points. The two selected reference points may be switchable.

[0098] The control unit 50 may calculate the 3D coordinates of a point that does not exist on the 3D shape of the subject and use the point with the 3D coordinates as a reference point. For example, the control unit 50 may calculate the 3D coordinates of the center of gravity of three points on the 3D shape of the subject and use the center of gravity as a reference point. Alternatively, the control unit 50 may estimate a plane or curved surface based on three or more points on the 3D shape of the subject and use a point on that plane or curved surface outside the 3D shape as a reference point. Alternatively, the control unit 50 may use the origin of the 3D coordinates in the 3D image data (camera origin) as a reference point. Alternatively, the control unit 50 may calculate a line based on two points on the 3D shape of the subject and use a point on that line outside the 3D shape as a reference point.

[0099] A 3D image display device according to each aspect of the present invention includes a control unit 50. The control unit 50 acquires 3D image data including 3D coordinates of three or more points on a subject calculated based on an endoscopic image of the subject. The control unit 50 displays a 3D shape of the subject on a display 51 based on the 3D image data. The control unit 50 sets a reference plane based on one or more points included in the three or more points. The control unit 50 divides the 3D image data into a first region and a second region using the reference plane as a boundary. The control unit 50 makes the display state of the 3D shape of the subject in one of the first region and the second region different from the display state of the 3D shape of the subject in the other of the first region and the second region in order to display a cross section of the subject set based on the reference plane.

[0100] A 3D image display method according to each aspect of the present invention includes first to fifth steps. In a first step (step S100), the control unit 50 acquires 3D image data including 3D coordinates of three or more points calculated based on an endoscopic image of the subject. In a second step (step S101), the control unit 50 displays a 3D shape of the subject on the display 51 based on the 3D image data. In a third step (step S106), the control unit 50 sets a reference plane based on one or more points included in the three or more points. In a fourth step (step S107), the control unit 50 divides the 3D image data into a first region and a second region using the reference plane as a boundary. In a fifth step (step S108), the control unit 50 differentiates the display state of the 3D shape of the subject in one of the first region and the second region from the display state of the 3D shape of the subject in the other of the first region and the second region in order to display a cross section of the subject set based on the reference plane.

[0101] The program according to each aspect of the present invention causes a computer to execute the first to fifth steps described above.

[0102] Each aspect of the present invention may include the following modifications: The control unit 50 hides one of the first area and the second area.

[0103] Each aspect of the present invention may include the following modifications: The control unit 50 makes the transparency of one of the first region and the second region higher than the transparency of the other of the first region and the second region.

[0104] Each aspect of the present invention may include the following modifications: The control unit 50 changes the display state of the cross section of the subject so that the cross section is emphasized.

[0105] Each aspect of the present invention may include the following modifications: The control unit 50 receives an instruction to rotate the 3D shape of the subject, and the control unit 50 displays the 3D shape rotated in accordance with the instruction on the display 51.

[0106] Each aspect of the present invention may include the following modifications: The control unit 50 receives an instruction to enlarge or reduce the 3D shape of the subject, and the control unit 50 displays the enlarged or reduced 3D shape on the display 51 in accordance with the instruction.

[0107] Each aspect of the present invention may include the following modifications: The control unit 50 displays, based on 3D image data, a 3D shape of the entire or part of the subject adjacent to a 3D shape of the subject including a first region and a second region having a display state different from the display state of the first region.

[0108] Each aspect of the present invention may include the following modifications: The control unit 50 sets a reference point based on one or more points included in the three or more points described above. The control unit 50 sets a reference plane based on the reference point.

[0109] Each aspect of the present invention may include the following modifications: The control unit 50 sets two or more reference points.

[0110] Each aspect of the present invention may include the following modifications: The control unit 50 sets a reference graphic based on one or more points included in the three or more points. The control unit 50 sets a reference plane based on the reference graphic.

[0111] Each aspect of the present invention may include the following modifications: The control unit 50 detects a characteristic region in an endoscopic image, and sets a reference plane based on one or more points included in the characteristic region among the three or more points.

[0112] Each aspect of the present invention may include the following modifications: The endoscopic image includes a first image and a second image that are stereo images, and the control unit 50 generates 3D image data based on the first image and the second image.

[0113] Each aspect of the present invention may include the following modifications: The endoscopic image is generated based on an optical image of the subject formed by a monocular optical system (optical adapter), and the control unit 50 generates 3D image data based on two or more endoscopic images.

[0114] Each aspect of the present invention may include the following modifications: The control unit 50 acquires 3D image data from the nonvolatile memory 57 (storage medium).

[0115] As described above, the control unit 50 displays the 3D shape of the cross section of the 3D shape of the subject on the display 51. Therefore, the endoscope system 1 can facilitate intuitive understanding of the 3D shape of the subject.

[0116] The user can change the angle of the 3D shape by rotating the 3D image including the cross-sectional 3D shape. Alternatively, the user can change the size of the 3D shape by enlarging or reducing the 3D image including the cross-sectional 3D shape. Therefore, the endoscope system 1 can improve the visibility of the cross-sectional 3D shape.

[0117] (First Modification of First Embodiment) A first modification of the first embodiment of the present invention will be described. In the first modification of the first embodiment, information on the area to be hidden is stored in non-volatile memory 57, and control unit 50 hides the first area or the second area in accordance with the information.

[0118] The first region is set on the positive side of the Y axis from the reference plane. The second region is set on the negative side of the Y axis from the reference plane. The user selects the first region or the second region by operating the touch panel 52 or the like. The control unit 50 generates region information indicating the selected region and stores the region information in the non-volatile memory 57.

[0119] 10 shows an example of the procedure for 3D shape display processing. The operation of the endoscope system 1 will be described with reference to Fig. 10. Processing that is the same as the processing shown in Fig. 2 will not be described.

[0120] After step S107, the control unit 50 acquires the area information from the nonvolatile memory 57 (step S110).

[0121] After step S110, the control unit 50 changes the display state of the first region or the second region indicated by the region information. For example, if the region information indicates the first region, the control unit 50 hides the first region (step S108a). When step S108a is executed, the 3D shape display process shown in FIG. 10 ends.

[0122] Each aspect of the present invention may include the following modifications: The control unit 50 acquires area information indicating one of the first area and the second area, and changes the display state of the area indicated by the area information.

[0123] In a first variation of the first embodiment, the user can select the first area or the second area to be hidden.

[0124] (Second Modification of First Embodiment) A second modification of the first embodiment of the present invention will be described. Fig. 11 shows an example of the configuration of an endoscope system 1a according to the second modification of the first embodiment. Portions that are the same as those shown in Fig. 1 will not be described.

[0125] 11 includes an insertion portion 2 and a main unit 6. The insertion portion 2 and the main unit 6 constitute an endoscope device 10a.

[0126] The insertion section 2 shown in Fig. 11 is the same as the insertion section 2 shown in Fig. 1. The main unit 6 has an imaging drive circuit 30, an image processing section 31, a UD drive section 32, an RL drive section 33, a bending control section 34, a light source 35, a light source control section 36, a display 51, a touch panel 52, operation buttons 53, a communication section 55, a volatile memory 56, a non-volatile memory 57, and a control section 60. Blocks that are the same as those shown in Fig. 1 are assigned the same reference symbols as those shown in Fig. 1.

[0127] The control unit 60 has both the functions of the control unit 40 shown in Fig. 1 and the functions of the control unit 50 shown in Fig. 1. The control unit 60 executes the process shown in Fig. 2.

[0128] Each aspect of the present invention may include the following modifications: The endoscope system 1 a includes an endoscope device 10 a, and the endoscope device 10 a includes an insertion section 2, a touch panel 52 (device), and a control section 60.

[0129] In the second modification of the first embodiment, the endoscope system 1a can facilitate intuitive understanding of the 3D shape of the subject, similarly to the first embodiment.

[0130] (Third Modification of First Embodiment) A third modification of the first embodiment of the present invention will be described. Fig. 12 shows an example of the configuration of an endoscope system 1b according to the third modification of the first embodiment. Portions that are the same as those shown in Fig. 1 will not be described.

[0131] 12 includes an insertion portion 2, a scope unit 3b, and a base unit 7. The insertion portion 2 and the scope unit 3b constitute an endoscope device 10b. The scope unit 3b and the base unit 7 are connected by a cable 8.

[0132] The insertion section 2 shown in Fig. 12 is the same as the insertion section 2 shown in Fig. 1. The scope unit 3b shown in Fig. 12 is the same as the scope unit 3 shown in Fig. 1 except that it does not have the image processing section 31. The base unit 7 has the image processing section 31, a display 51, a touch panel 52, operation buttons 53, a communication section 55, a volatile memory 56, a non-volatile memory 57, and a control section 70. The same reference symbols as those in Fig. 1 are assigned to blocks that are the same as those in Fig. 1.

[0133] The control unit 70 has both the functions of the control unit 40 shown in Fig. 1 and the functions of the control unit 50 shown in Fig. 1. The control unit 70 executes the process shown in Fig. 2.

[0134] In the third modified example of the first embodiment, the endoscope system 1b can facilitate intuitive understanding of the 3D shape of the subject, similarly to the first embodiment.

[0135] Second Embodiment A second embodiment of the present invention will be described. In the second embodiment, an endoscope system 1 shown in Fig. 1 is used. An endoscope system 1a shown in Fig. 11 or an endoscope system 1b shown in Fig. 12 may also be used.

[0136] A method for hiding a part of the 3D shape of the object will be described in detail below. The method described below can be applied to the first embodiment.

[0137] In the second embodiment, the 3D image data includes vertex information about three or more points (vertices) on the subject and surface information about a surface formed by three vertices. In the following example, the 3D image data includes vertex information about four or more vertices and surface information about two or more surfaces.

[0138] The 3D image data includes vertex information VI1 and surface information SI1 shown in FIG. 13. The vertex information VI1 includes the vertex index and 3D coordinates of each vertex in the 3D image data. The surface information SI1 includes the surface index of each surface in the 3D image data and the vertex indexes of the three vertices included in each surface. In step S101, the control unit 50 displays a 3D shape SH1 on the display 51 based on the vertex information VI1 and the surface information SI1.

[0139] The endoscope system 1 executes the processing shown in Fig. 14 instead of steps S106 to S108 shown in Fig. 2. That is, the endoscope system 1 executes steps S100 to S105 and S109 shown in Fig. 2 and the processing shown in Fig. 14. Fig. 14 shows an example of the procedure for 3D shape display processing. The operation of the endoscope system 1 will be described using Fig. 14.

[0140] The control unit 50 calculates an equation of a boundary line that passes through the first reference point and the second reference point (step S120).

[0141] The control unit 50 executes the following process in step S120. The control unit 50 sets a first reference point P1 in step S103 shown in FIG. 2, and sets a second reference point P2 in step S105 shown in FIG. 2. The equation of the boundary line that passes through the first reference point P1 and has a direction vector d is expressed by the following formula (1). In formula (1), the parameter is represented as t. P=P1+t*d (1)

[0142] For example, when the 3D coordinates of the first reference point P1 are (2, 3, 1) and the direction vector d is (1, −1, 1), the equation of the boundary line is expressed by the following formula (2): (x, y, z) = (2, 3, 1) + t * (1, −1, 1) (2)

[0143] For example, when the parameter t is -1, (x, y, z) = (3, 2, 2) holds. The point (3, 2, 2) is on the boundary line.

[0144] The direction vector d is expressed by the following equation (3) using the first reference point P1 and the second reference point P2.

[0145]

[0146] The equation of the boundary line is expressed as the following equation (4) using the elements of the vector:

[0147]

[0148] After step S120, the control unit 50 refers to the positional relationship between each vertex on the subject and the boundary line, and identifies vertices that will not be displayed (step S121).

[0149] The control unit 50 executes the following process in step S121: Hereinafter, a vertex that is displayed in the 3D image is defined as a first vertex, and a vertex that is not displayed in the 3D image is defined as a second vertex.

[0150] The control unit 50 identifies the second vertex according to the direction vector d of the boundary line. Figures 15A and 15B show the relationship between the regions in the 3D image data and the boundary lines.

[0151] 15A and 15B passes through the first reference point P1 and the second reference point P2. The arrow of the boundary line L1 indicates the direction of the direction vector d. The entire area in the 3D image data is divided into a first area R1 and a second area R2. The boundary line L1 constitutes the boundary between the first area R1 and the second area R2.

[0152] In the example shown in Figure 15A, the second vertex is included in the second region R2. That is, the second vertex is located on the negative side of the Y axis relative to the boundary line L1. On the other hand, in the example shown in Figure 15B, the second vertex is included in the first region R1. That is, the second vertex is located on the positive side of the Y axis relative to the boundary line L1. As described above, the region that is not displayed in the 3D image is set according to the direction vector d of the boundary line L1.

[0153] After step S121, the control unit 50 identifies a face including the second vertex (step S122). After step S122, the control unit 50 deletes information about the face including the second vertex from the face information and hides the face including the second vertex (step S123).

[0154] FIG. 16 shows an example of face information. The face information SI2 shown in FIG. 16 includes a face index and a vertex index. For the 3D shape SH2 shown in FIG. 16, the control unit 50 calculates a boundary line L2. The control unit 50 identifies a second vertex that is located on the negative side of the Y axis from the boundary line L2. The control unit 50 identifies the vertex with vertex index 5 and the vertex with vertex index 7 as the second vertices.

[0155] In step S123, the control unit 50 generates face information SI2 by deleting the information on the second vertex from the face information SI1 shown in Fig. 13. The face information SI2 does not include information on the face including the vertex with vertex index 5 and information on the face including the vertex with vertex index 7.

[0156] The face information SI2 includes the changed face index. For example, in the face information SI1 shown in Fig. 13, the face with face index 3 includes a vertex with vertex index 2, a vertex with vertex index 4, and a vertex with vertex index 3. In step S123, the control unit 50 deletes the information about the face with face index 2 from the face information. Therefore, the face with face index 3 is changed to a face with face index 2 in the face information SI2.

[0157] After step S123, the control unit 50 extracts two or more vertices in the cross section of the subject (step S124). After step S124, the control unit 50 rearranges the two or more extracted vertices in a predetermined order (step S125). After step S125, the control unit 50 connects the two or more extracted vertices with a line and highlights the line on the display 51 (step S126).

[0158] Steps S121 to S126 will be described in detail with reference to Figures 17 to 19. Figures 17 to 19 show examples of 3D shapes used in the 3D shape display process.

[0159] 2, the control unit 50 displays a 3D image of the 3D shape SH12 shown in Fig. 17 on the display 51. In step S106 shown in Fig. 2, the control unit 50 sets a boundary line L12 shown in Fig. 17.

[0160] In step S121, the control unit 50 identifies a second vertex to be hidden. In step S122, the control unit 50 identifies a face that includes the second vertex. One face includes three vertices. If at least one of the three vertices is the second vertex, the control unit 50 identifies the face. In step S123, the control unit 50 hides the face.

[0161] The control unit 50 displays the 3D shape SH13 shown in Fig. 18 on the display 51 instead of the 3D shape SH12 shown in Fig. 17. The surface identified in step S123 is not displayed.

[0162] In step S124, the control unit 50 executes the following process: The control unit 50 identifies a surface including the first vertex and the second vertex, and the control unit 50 selects the first vertex of the identified surface as a vertex of the cross-sectional portion.

[0163] In step S125, the control unit 50 executes the following process: The control unit 50 sorts the information about two or more vertices of the cross-sectional portion in descending order of the X coordinate of each vertex. The control unit 50 also sorts the information about two or more vertices having the same X coordinate in descending order of the Y coordinate of each vertex.

[0164] In step S126, the control unit 50 executes the following process. The control unit 50 connects two or more vertices of the cross-sectional portion with a thick line in the sorted order. The control unit 50 displays the line connecting the two or more vertices of the cross-sectional portion on the display 51. For example, the control unit 50 displays the line L13 shown in FIG. 19 on the display 51.

[0165] In the second embodiment, the endoscope system 1 can facilitate intuitive understanding of the 3D shape of the subject, similarly to the first embodiment.

[0166] Third Embodiment A third embodiment of the present invention will be described. In the third embodiment, the endoscope system 1 shown in Fig. 1 is used. An endoscope system 1a shown in Fig. 11 or an endoscope system 1b shown in Fig. 12 may also be used.

[0167] The endoscope system 1 according to the third embodiment has a measurement function. The endoscope system 1 sets a measurement reference that is different from the cross section of the subject and measures the 3D distance between the measurement reference and each point on the cross section. The measurement reference is a point, a line, or a surface. The endoscope system 1 displays the deepest point and the highest point on the cross section.

[0168] For example, the control unit 50 sets the center of the tip of the insertion unit 2 as the measurement reference point. Alternatively, the user operates the touch panel 52 or the like to input an instruction to set the measurement reference point to the endoscope system 1. The control unit 50 sets the measurement reference point in accordance with the instruction.

[0169] Alternatively, the user operates the touch panel 52 or the like to input an instruction to set two points to the endoscope system 1. The control unit 50 sets a measurement reference line connecting those two points. Alternatively, the user operates the touch panel 52 or the like to input an instruction to set three or more points to the endoscope system 1. The control unit 50 sets a measurement reference plane that passes through those three or more points.

[0170] After executing the 3D shape display process shown in Fig. 2, the endoscope system 1 executes the measurement process shown in Fig. 20. Fig. 20 shows an example of the procedure of the measurement process. The operation of the endoscope system 1 will be described using Fig. 20.

[0171] The control unit 50 acquires the 3D coordinates of the points on the boundary line set in step S106. If the points on the boundary line are on the subject, the control unit 50 acquires the 3D coordinates of the points from the 3D image data. If the points on the boundary line are not on the subject, the control unit 50 calculates the 3D coordinates of the points (step S130).

[0172] After step S130, the control unit 50 determines whether or not a measurement reference line or a measurement reference plane has been set (step S131).

[0173] When the control unit 50 determines in step S131 that a measurement reference line or a measurement reference plane has been set, the control unit 50 calculates the 3D distance between the measurement reference line and each point on the boundary line. Alternatively, the control unit 50 calculates the 3D distance between the measurement reference plane and each point on the boundary line. At this time, the control unit 50 calculates signed 3D distances. The 3D distance of each point on the front side of the measurement reference plane as viewed from the tip of the insertion unit 2 has a positive sign. The 3D distance of each point on the back side of the measurement reference plane as viewed from the tip of the insertion unit 2 has a negative sign (step S132).

[0174] When the control unit 50 determines in step S131 that a measurement reference line or measurement reference plane has not been set, the control unit 50 calculates the 3D distance between the tip of the insertion unit 2 and each point on the boundary line. At this time, the control unit 50 calculates the 3D distance without a sign (step S133). The Z axis in the 3D space defined in the 3D image data is parallel to the optical axis of the optical adapter. The origin of the 3D space is set to the tip of the insertion unit 2. Therefore, the 3D distance is expressed by the Z coordinate of each point.

[0175] The control unit 50 calculates the 3D distance at each of two or more points on the boundary line in step S132 or step S133. If an unsigned 3D distance is used in step S133, the control unit 50 identifies the point whose 3D distance is maximum (the farthest point) and the point whose 3D distance is minimum (the closest point). The control unit 50 highlights the two identified points on the 3D image displayed on the display 51. As a result, the control unit 50 highlights the farthest point and the closest point on the cross section (step S134). Furthermore, if a signed 3D distance is used in step S132, the control unit 50 identifies the point whose 3D distance is maximum in the positive direction (the highest point) and the point whose 3D distance is maximum in the negative direction (the deepest point). The control unit 50 highlights the two identified points on the 3D image displayed on the display 51. As a result, the control unit 50 highlights the highest point and the deepest point on the cross section (step S134).

[0176] After step S134, the control unit 50 displays the 3D distance values ​​of the farthest point and the nearest point (or the deepest point and the highest point) on the 3D image (step S135). When step S135 is executed, the measurement process shown in FIG. 20 ends.

[0177] FIG. 21 shows an example of the 3D shape of the subject displayed on the display 51 in steps S134 and S135.

[0178] The control unit 50 displays the 3D shape SH13 on the display 51. The control unit 50 displays a line L13 connecting two or more vertices of the cross-sectional portion on the 3D shape SH13. The control unit 50 displays the deepest point DP10 and the highest point HP10 on the 3D shape SH13. The control unit 50 displays the 3D distance value DV10 at the deepest point DP10 on the 3D shape SH13, and displays the 3D distance value DV11 at the highest point HP10 on the 3D shape SH13.

[0179] The control unit 50 may display the measurement reference on the display 51. The user can easily understand the reference position for measuring the 3D distance.

[0180] The control unit 50 may perform point-to-point distance measurement or surface-based measurement at any measurement position on the 3D shape of the object. After such a measurement is performed, the control unit 50 may display a cross section of the object. In this example, the control unit 50 can display the cross section of the object with the measurement reference, measurement results, etc. displayed.

[0181] Each aspect of the present invention may include the following modifications: The control unit 50 sets a measurement reference based on one or more points on the object, and calculates the distance between the measurement reference and a point on the cross section of the object.

[0182] Each aspect of the present invention may include the following modifications: The control unit 50 calculates the distance between the measurement reference and each of two or more points on the cross section of the subject, and identifies the point among the two or more points with the maximum or minimum distance.

[0183] Each aspect of the present invention may include the following modifications: The control unit 50 superimposes information indicating the point at which the above-mentioned distance is maximum or minimum on the 3D shape of the subject.

[0184] In the third embodiment, the endoscope system 1 can perform measurements on a cross section of a subject.

[0185] (Modification of the Third Embodiment) A modification of the third embodiment of the present invention will be described. In the modification of the third embodiment, the endoscope system 1 performs measurement on a part of a cross section of the subject.

[0186] After executing the 3D shape display process shown in Fig. 2, the endoscope system 1 executes the following process. The control unit 50 sets a target range including a portion of a cross section of the subject. In step S130 shown in Fig. 20, the control unit 50 acquires the 3D coordinates of two or more points included in the cross section in the target range. Thereafter, the control unit 50 executes steps S131 to S135.

[0187] Fig. 22 shows an example of a 3D image displayed on the display 51 after step S108 shown in Fig. 2. Portions that are the same as those shown in Fig. 7 will not be described.

[0188] The user operates the touch panel 52 or the like to input an instruction to the endoscope system 1 to set two range reference points on a cross section of the subject. The control unit 50 accepts the instruction and sets a first range reference point P10 and a second range reference point P11 shown in FIG. 22 . For example, the first range reference point P10 and the second range reference point P11 are set at positions other than the ends of the cross section. The control unit 50 sets a target range R10 that includes the first range reference point P10 and the second range reference point P11. The target range R10 includes two or more points included in the cross section. The control unit 50 displays the first range reference point P10, the second range reference point P11, and the target range R10 on the 3D shape SH10.

[0189] Each aspect of the present invention may include the following modifications: The control unit 50 sets a range including a part of a cross section of the subject in the 3D image data, and calculates the distance between the measurement reference and each of two or more points included in the range.

[0190] In a modification of the third embodiment, the control unit 50 calculates the distance between the measurement reference and each of two or more points included in a target range set on a cross section of the subject. 3D image data at the edge of the cross section may contain noise. By setting the target range, the endoscope system 1 can eliminate the influence of noise on the measurement results.

[0191] (Fourth embodiment) A fourth embodiment of the present invention will be described. In the fourth embodiment, the endoscope system 1 shown in Fig. 1 is used. An endoscope system 1a shown in Fig. 11 or an endoscope system 1b shown in Fig. 12 may also be used.

[0192] In the fourth embodiment, the control unit 50 sets the reference plane based on three reference points. First to fourth methods for setting the reference plane will be described below.

[0193] First, the first method will be described. The control unit 50 sets a plane passing through the first reference point, the second reference point, and the origin of the 3D image data as a reference plane. The first reference point and the second reference point are set by the method of the first embodiment. The origin of the 3D image data corresponds to the camera position when the imaging unit 20 generates the image. The origin of the 3D image data may coincide with the center of the tip of the insertion unit 2. In the first method, the reference plane corresponds to a straight line on a distortion-corrected image generated by correcting the optical distortion of the image generated by the imaging unit 20. Therefore, the user can easily understand the position through which the reference plane passes on the distortion-corrected image.

[0194] Next, the second method will be described. With the first method, it is difficult to visualize a cross section of a region that is not viewed from the front in the image (the image generated by the imaging unit 20) used to generate the 3D image data that is the basis for the 3D image displayed on the display 51. With the second method, the control unit 50 rotates the 3D image so that the region is viewed from the front.

[0195] When the area to be visualized as a cross section is not viewed from the front, the user rotates the 3D image displayed on the display 51 by operating the touch panel 52 or the like. This causes the area to be viewed from the front on the 3D image. The control unit 50 then sets a plane passing through the first reference point, the second reference point, and the origin of the 3D view camera as a reference plane. The first reference point and the second reference point are set by the method of the first embodiment. The origin of the 3D view camera corresponds to the viewpoint of the perspective projection transformation when a 3D image is generated from 3D image data and displayed on the display 51.

[0196] The control unit 50 may rotate the 3D image so that the area near the point set in accordance with an instruction from the user is viewed from the front. The rotation process for rotating the 3D image will be described in detail below.

[0197] The endoscope system 1 executes the rotation process shown in Fig. 23 before executing the 3D shape display process shown in Fig. 2. Alternatively, the endoscope system 1 executes the rotation process during the 3D shape display process shown in Fig. 23. Fig. 23 shows an example of the procedure for the rotation process. The operation of the endoscope system 1 will be described using Fig. 23.

[0198] The user operates the touch panel 52 or the like to input an instruction to set a rotation reference point to the endoscope system 1. The control unit 50 accepts the instruction and sets the rotation reference point (step S140). The rotation reference point may be the first reference point set in step S103 shown in FIG. 2 or the second reference point set in step S105 shown in FIG. 2. The rotation reference point may be a point different from the first reference point or the second reference point. The rotation reference point may be the midpoint of a line segment connecting the first reference point and the second reference point.

[0199] After step S140, the control unit 50 sets a sphere having a predetermined radius and centered on the rotation reference point as an extraction range (step S141). The extraction range includes three or more points included in the 3D image data.

[0200] After step S141, the control unit 50 extracts three or more points from the extraction range (step S142).

[0201] After step S142, the control unit 50 estimates a plane based on three or more points extracted from the extraction range. When the normal vector of the plane is defined as (a, b, c), the distance between the plane and the origin is defined as d, and the equation of the plane is defined as ax+by+cz+d=0, the control unit 50 uses singular value decomposition (SVD) to find coefficients a, b, c, and d that minimize the sum of squares of the distances from the plane to three or more points. This allows the control unit 50 to estimate the plane (step S143). The plane approximates the surface of the object in the extraction range.

[0202] After step S143, the control unit 50 changes the position and orientation of the 3D view camera so that the normal to the plane faces the 3D view camera (step S144). When step S144 is executed, the rotation process shown in FIG. 23 ends.

[0203] Next, a third method will be described. The user operates the touch panel 52 or the like to input an instruction to set three reference points to the endoscope system 1. The control unit 50 accepts the instruction and sets the three reference points. The control unit 50 sets a plane passing through the three reference points as a reference plane.

[0204] Next, a fourth method will be described. After the first and second reference points have been set, the user operates the touch panel 52 or the like to input an instruction to set a third reference point to the endoscope system 1. The control unit 50 accepts the instruction and sets the third reference point. The control unit 50 estimates a plane that approximates the area near the third reference point. This plane is a reference figure. The control unit 50 sets a reference plane that passes through the first and second reference points and is parallel to the normal vector of the estimated plane.

[0205] In the fourth embodiment, the endoscope system 1 can facilitate intuitive understanding of the 3D shape of the subject, similarly to the first embodiment.

[0206] Fifth Embodiment A fifth embodiment of the present invention will be described. In the fifth embodiment, the endoscope system 1 shown in Fig. 1 is used. An endoscope system 1a shown in Fig. 11 or an endoscope system 1b shown in Fig. 12 may also be used.

[0207] In the fifth embodiment, the control unit 50 sets a reference plane based on one or more reference points and a predetermined direction. First to eighth methods for setting the reference plane will be described below.

[0208] First, the first method will be described. The control unit 50 sets a plane that passes through the first reference point and is perpendicular to the line of sight from the origin of the 3D view camera to the first reference point as a reference plane.

[0209] Next, the second method will be described. The control unit 50 sets a reference plane that passes through the first and second reference points and is parallel to the optical axis of the 3D view camera.

[0210] Next, a third method will be described. After the first and second reference points are set, the control unit 50 estimates a plane that approximates an area near at least one of the two reference points. This plane is a reference figure. The control unit 50 sets a plane that passes through the first and second reference points and is parallel to the normal vector of the estimated plane as a reference plane.

[0211] Next, a fourth method will be described. After the first and second reference points are set, the control unit 50 calculates a straight line that passes through the first and second reference points. The control unit 50 then sets, as a reference plane, a plane that passes through the first reference point and has a normal vector parallel to the straight line.

[0212] Next, a fifth method will be described. After a first reference point is set, the control unit 50 estimates a plane that approximates the area near the first reference point. This plane is a reference figure. The control unit 50 sets a plane that passes through the first reference point and has the same normal vector as the normal vector of the estimated plane as the reference plane.

[0213] 24 shows an example of the procedure for 3D shape display processing in the fifth method. The operation of the endoscope system 1 will be described using Fig. 24. Processing that is the same as the processing shown in Fig. 2 will not be described.

[0214] After step S103, the control unit 50 sets a sphere having a predetermined radius and centered on the first reference point as an extraction range (step S150). The extraction range includes three or more points included in the 3D image data.

[0215] After step S150, the control unit 50 extracts three or more points from the extraction range (step S151).

[0216] After step S151, the control unit 50 estimates a plane based on the three or more points extracted from the extraction range (step S152). The plane approximates the surface of the object in the extraction range.

[0217] After step S152, the control unit 50 sets a plane that passes through the first reference point and has the same normal vector as the normal vector of the estimated plane as a reference plane (step S153). After step S153, step S107 is executed. In the above method, the user needs to perform fewer operations than when two reference points are set.

[0218] The control unit 50 may set a second reference point in addition to the first reference point. The control unit 50 may execute the same processes as steps S150 to S153 based on the second reference point. The control unit 50 may calculate the average of the normal vectors of the two estimated planes. The control unit 50 may set, as the reference plane, a plane that passes through the first reference point or the second reference point and has the same normal vector as the calculated average normal vector.

[0219] Next, a sixth method will be described. The control unit 50 estimates a curved surface (cylindrical surface) by using an arbitrary 3D point cloud. When the direction vector of the central axis of the cylindrical surface is defined as (a, b, c) and the coordinates of the point through which the central axis of the cylindrical surface passes are defined as (x0, y0, z0), the equation of the cylindrical axis is expressed as (x-x0) / a = (y-y0) / b = (z-z0) / c. The control unit 50 calculates the parameters of the cylinder a, b, c, x0, y0, z0, r using RANSAC (Random Sample Consensus) so that the sum of squares of the difference between the distance d from the cylindrical axis to each point in the 3D point cloud and the radius r of the cylindrical surface is minimized. The cylindrical surface is a reference figure.

[0220] The control unit 50 sets a reference point according to an instruction from the user. Alternatively, the control unit 50 sets a point on the subject that is farthest or closest to the central axis of the cylinder as the reference point. The control unit 50 sets a plane that passes through the reference point and has the central axis of the cylinder as its normal line as the reference plane. In the sixth method, the endoscope system 1 can display the 3D shape of a cross section formed when the 3D shape of the subject is cut along a plane perpendicular to the central axis of the cylinder.

[0221] The user may input an instruction to set a reference point to the endoscope system 1 by operating the touch panel 52 or the like. The control unit 50 may display the central axis of the cylinder on the display 51, and may set a reference point on the central axis in accordance with an instruction from the user.

[0222] The control unit 50 may set a point that is included in the 3D image data and closest to the central axis of the cylinder as the reference point. The control unit 50 may set a point that is included in the 3D image data and farthest from the central axis of the cylinder as the reference point. The control unit 50 may set a target range, and may set a point that is included in the target range and closest to the central axis of the cylinder as the reference point. Alternatively, the control unit 50 may set a point that is included in the target range and farthest from the central axis of the cylinder as the reference point.

[0223] Fig. 25 shows an example of the procedure for 3D shape display processing in the sixth method. The operation of the endoscope system 1 will be described using Fig. 25. Processing that is the same as the processing shown in Fig. 2 will not be described.

[0224] After step S101, the control unit 50 estimates the cylindrical surface and calculates the central axis of the cylinder (step S160).

[0225] After step S160, the control unit 50 extracts the point closest to the central axis of the cylinder or the point farthest from the central axis of the cylinder as a reference point (step S161).

[0226] After step S161, the control unit 50 sets a plane that passes through the reference point and has the central axis of the cylinder as its normal line as a reference plane (step S162). After step S162, step S107 is executed.

[0227] Next, a seventh method will be described. The control unit 50 sets a plane that passes through the reference point and includes the central axis of the cylinder as the reference plane. In the seventh method, the endoscope system 1 can display the 3D shape of a cross section formed when the 3D shape of the subject is cut along the plane that includes the central axis of the cylinder.

[0228] Next, an eighth method will be described. The control unit 50 sets a plane that passes through the reference point and the reference line and is parallel to the optical axis of the 3D view camera as a reference plane. The reference line is a horizontal or vertical line in the 3D image displayed on the display 51.

[0229] In the fifth embodiment, the endoscope system 1 can facilitate intuitive understanding of the 3D shape of the subject, similarly to the first embodiment.

[0230] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Addition, omission, substitution, and other modifications of the configuration are possible within the scope of the spirit of the present invention. Furthermore, the present invention is not limited by the above description, but is limited only by the scope of the appended claims.

[0231] According to each embodiment of the present invention, the three-dimensional image display device, the three-dimensional image display method, and the program can facilitate intuitive understanding of the three-dimensional shape of a subject.

[0232] REFERENCE SIGNS LIST 1, 1a, 1b Endoscope system 2 Insertion section 2a Tip section 3, 3b Scope unit 4, 7 Base unit 5, 6 Main unit 10, 10a, 10b Endoscope device 11 External device 20 Imaging section 21 Bending section 22 Illumination window 30 Imaging drive circuit 31 Image processing section 32 UD drive section 33 RL drive section 34 Bending control section 35 Light source 36 Light source control section 40, 50, 60, 70 Control section 41, 54, 55 Communication section 42, 56 Volatile memory 43, 57 Non-volatile memory 51 Display 52 Touch panel 53 Operation button

Claims

1. A three-dimensional image display device having a control unit, which acquires three-dimensional image data including three-dimensional coordinates of three or more points on a subject calculated based on an endoscopic image of the subject, displays the three-dimensional shape of the subject on a display based on the three-dimensional image data, sets a reference plane based on one or more points included in the three or more points, divides the three-dimensional image data into a first region and a second region using the reference plane as a boundary, and makes the display state of the three-dimensional shape of the subject in one of the first region and the second region different from the display state of the three-dimensional shape of the subject in the other of the first region and the second region in order to display a cross-section of the subject set based on the reference plane.

2. The three-dimensional image display device according to claim 1, wherein the control unit makes one of the first area and the second area non-displayable.

3. The three-dimensional image display device according to claim 1, wherein the control unit makes the transparency of one of the first region and the second region higher than the transparency of the other of the first region and the second region.

4. The three-dimensional image display device according to claim 1, wherein the control unit acquires area information indicating one of the first area and the second area, and changes the display state of the area indicated by the area information.

5. The three-dimensional image display device according to claim 1, wherein the control unit changes the display state of the cross section so that the cross section is emphasized.

6. The three-dimensional image display device according to claim 1, wherein the control unit receives an instruction to rotate the three-dimensional shape, and displays the three-dimensional shape rotated in accordance with the instruction on the display.

7. The three-dimensional image display device according to claim 1, wherein the control unit receives an instruction to enlarge or reduce the three-dimensional shape, and displays the three-dimensional shape enlarged or reduced in accordance with the instruction on the display.

8. A three-dimensional image display device as described in claim 1, wherein the control unit displays the three-dimensional shape of the entire or part of the subject next to the three-dimensional shape of the subject, including the first region and the second region having a display state different from the display state of the first region, based on the three-dimensional image data.

9. The three-dimensional image display device according to claim 1, wherein the control unit sets a reference point based on one or more points included in the three or more points, and sets the reference plane based on the reference point.

10. The three-dimensional image display device according to claim 9, wherein the control unit sets two or more reference points.

11. The three-dimensional image display device according to claim 1, wherein the control unit sets a reference figure based on the one or more points, and sets the reference plane based on the reference figure.

12. A three-dimensional image display device as described in claim 1, wherein the control unit detects a characteristic area in the endoscopic image and sets the reference plane based on one or more points among the three or more points that are included in the characteristic area.

13. The three-dimensional image display device according to claim 1, wherein the control unit sets a measurement reference based on one or more points included in the three or more points, and calculates the distance between the measurement reference and a point on the cross section.

14. The three-dimensional image display device according to claim 13, wherein the control unit calculates the distance between the measurement reference and each of two or more points on the cross section, and identifies the point among the two or more points at which the distance is maximum or minimum.

15. The three-dimensional image display device according to claim 14, wherein the control unit superimposes information indicating the point at which the distance is maximum or minimum on the three-dimensional shape.

16. A three-dimensional image display device according to claim 14, wherein the control unit sets a range including a portion of the cross section in the three-dimensional image data, and calculates the distance between the measurement reference and each of two or more points included in the range.

17. A three-dimensional image display device as described in claim 1, wherein the endoscopic image includes a first image and a second image which are stereo images, and the control unit generates the three-dimensional image data based on the first image and the second image.

18. A three-dimensional image display device according to claim 1, wherein the endoscopic image is generated based on an optical image of the subject formed by a monocular optical system, and the control unit generates the three-dimensional image data based on two or more of the endoscopic images.

19. The three-dimensional image display device according to claim 1, wherein the control unit acquires the three-dimensional image data from a storage medium.

20. A three-dimensional image display method, in which a control unit acquires three-dimensional image data including three-dimensional coordinates of three or more points on a subject calculated based on an endoscopic image of the subject, displays the three-dimensional shape of the subject on a display based on the three-dimensional image data, sets a reference plane based on one or more points included in the three or more points, divides the three-dimensional image data into a first region and a second region using the reference plane as a boundary, and makes the display state of the three-dimensional shape of the subject in one of the first region and the second region different from the display state of the three-dimensional shape of the subject in the other of the first region and the second region in order to display a cross-section of the subject set based on the reference plane.

21. A program for causing a computer to execute the following steps: acquiring three-dimensional image data including three-dimensional coordinates of three or more points on a subject calculated based on an endoscopic image of the subject; displaying the three-dimensional shape of the subject on a display based on the three-dimensional image data; setting a reference plane based on one or more points included in the three or more points; dividing the three-dimensional image data into a first region and a second region using the reference plane as a boundary; and differentiating the display state of the three-dimensional shape of the subject in one of the first region and the second region from the display state of the three-dimensional shape of the subject in the other of the first region and the second region in order to display a cross-section of the subject set based on the reference plane.

Images