Three dimensional image display device, three dimensional image display method, and program

Abstract

To provide a three dimensional image display device, a three dimensional image display method, and a program capable of facilitating intuitive understanding of a three dimensional shape of a subject.SOLUTION: The three dimensional image display device includes a control unit. The processing circuitry displays a three dimensional shape of a subject on a display based on three dimensional image data including three dimensional coordinates of three or more points on the subject. The processing circuitry sets a reference plane based on one or more points included in the three or more points, and divides the three dimensional image data into a first region and a second region with the reference plane as a boundary. The control unit makes a display state of a three dimensional shape of the subject in one of the first region and the second region different from a display state of a three dimensional shape of the subject in the other of the first region and the second region in order to display a cross section of the subject set based on the reference plane.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a three-dimensional image display device, a three-dimensional image display method, and a program. [Background technology]

[0002] Industrial endoscope devices are used for inspecting (endoscopic inspection) for abnormalities and corrosion inside boilers, pipes, aircraft engines, heat exchangers, etc. The endoscope device disclosed in Patent Document 1 displays the shape of a cross section formed when an object to be observed is cut along a plane passing through two points set on an image of the object to be observed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4343341 Summary of the Invention [Problem to be solved by the invention]

[0004] In one method (FIG. 9) disclosed in Patent Document 1, an image of the subject is displayed and the outline of the cross-sectional shape is displayed on the image. In another method (FIG. 12) disclosed in Patent Document 1, the cross-sectional shape as seen from a viewpoint different from the viewpoint from which the observation object is photographed is displayed. However, in the method disclosed in Patent Document 1, the cross-sectional shape is displayed at a position different from the original position of the cross-sectional shape in the image of the subject, or the cross-sectional shape is displayed in an image other than the image of the subject. As a result, it is difficult for the user to intuitively understand the cross-sectional shape of the subject.

[0005] An object of the present invention is to provide a three-dimensional image display device, a three-dimensional image display method, and a program that can facilitate intuitive understanding of the three-dimensional shape of a subject. [Means for solving the problem]

[0006] The present invention is a three-dimensional image display device that has a control unit, which acquires three-dimensional image data including three-dimensional coordinates of three or more points on a subject calculated based on an endoscopic image of the subject, displays the three-dimensional shape of the subject on a display based on the three-dimensional image data, sets a reference plane based on one or more points included in the three or more points, divides the three-dimensional image data into a first region and a second region using the reference plane as a boundary, and makes the display state of the three-dimensional shape of the subject in one of the first region and the second region different from the display state of the three-dimensional shape of the subject in the other of the first region and the second region in order to display a cross-section of the subject set based on the reference plane.

[0007] In the three-dimensional image display device of the present invention, the control unit hides one of the first area and the second area.

[0008] In the three-dimensional image display device of the present invention, the control unit makes the transparency of one of the first region and the second region higher than the transparency of the other of the first region and the second region.

[0009] In the three-dimensional image display device of the present invention, the control unit acquires region information indicating one of the first region and the second region, and changes the display state of the region indicated by the region information.

[0010] In the three-dimensional image display device of the present invention, the control unit changes the display state of the cross section so that the cross section is emphasized.

[0011] In the three-dimensional image display device of the present invention, the control unit receives an instruction to rotate the three-dimensional shape, and displays the three-dimensional shape rotated in accordance with the instruction on the display.

[0012] In the three-dimensional image display device of the present invention, the control unit receives an instruction to enlarge or reduce the three-dimensional shape, and displays the three-dimensional shape enlarged or reduced in accordance with the instruction on the display.

[0013] In the three-dimensional image display device of the present invention, the control unit displays the three-dimensional shape of the entire or part of the subject next to the three-dimensional shape of the subject, which includes the first region and the second region having a display state different from the display state of the first region, based on the three-dimensional image data.

[0014] In the three-dimensional image display device of the present invention, the control unit sets a reference point based on one or more points included in the three or more points, and sets the reference plane based on the reference point.

[0015] In the three-dimensional image display device of the present invention, the control unit sets two or more of the reference points.

[0016] In the three-dimensional image display device of the present invention, the control unit sets a reference figure based on the one or more points, and sets the reference plane based on the reference figure.

[0017] In the three-dimensional image display device of the present invention, the control unit detects a characteristic area in the endoscopic image, and sets the reference plane based on one or more points included in the characteristic area among the three or more points.

[0018] In the three-dimensional image display device of the present invention, the control unit sets a measurement reference based on one or more points included in the three or more points, and calculates the distance between the measurement reference and a point on the cross section.

[0019] In the three-dimensional image display device of the present invention, the control unit calculates the distance between the measurement reference and each of two or more points on the cross section, and identifies the point among the two or more points where the distance is maximum or minimum.

[0020] In the three-dimensional image display device of the present invention, the control unit superimposes information indicating the point at which the distance is maximum or minimum on the three-dimensional shape.

[0021] In the three-dimensional image display device of the present invention, the control unit sets a range including a portion of the cross section in the three-dimensional image data, and calculates the distance between the measurement reference and each of two or more points included in the range.

[0022] In the three-dimensional image display device of the present invention, the endoscopic image includes a first image and a second image which are stereo images, and the control unit generates the three-dimensional image data based on the first image and the second image.

[0023] In the three-dimensional image display device of the present invention, the endoscopic image is generated based on an optical image of the subject formed by a monocular optical system, and the control unit generates the three-dimensional image data based on two or more of the endoscopic images.

[0024] In the three-dimensional image display device of the present invention, the control unit acquires the three-dimensional image data from a storage medium.

[0025] The present invention is a three-dimensional image display method in which a control unit acquires three-dimensional image data including three-dimensional coordinates of three or more points on a subject calculated based on an endoscopic image of the subject, displays the three-dimensional shape of the subject on a display based on the three-dimensional image data, sets a reference plane based on one or more points included in the three or more points, divides the three-dimensional image data into a first region and a second region using the reference plane as a boundary, and makes the display state of the three-dimensional shape of the subject in one of the first region and the second region different from the display state of the three-dimensional shape of the subject in the other of the first region and the second region in order to display a cross-section of the subject set based on the reference plane.

[0026] The present invention is a program for causing a computer to execute the following steps: acquiring three-dimensional image data including three-dimensional coordinates of three or more points on a subject calculated based on an endoscopic image of the subject; displaying the three-dimensional shape of the subject on a display based on the three-dimensional image data; setting a reference plane based on one or more points included in the three or more points; dividing the three-dimensional image data into a first region and a second region using the reference plane as a boundary; and differentiating the display state of the three-dimensional shape of the subject in one of the first region and the second region from the display state of the three-dimensional shape of the subject in the other of the first region and the second region in order to display a cross-section of the subject set based on the reference plane. [Effects of the Invention]

[0027] According to the present invention, the three-dimensional image display device, the three-dimensional image display method, and the program can facilitate intuitive understanding of the three-dimensional shape of a subject. [Brief explanation of the drawings]

[0028] [Figure 1] 1 is a block diagram showing an example of the configuration of an endoscope system according to a first embodiment of the present invention. [Figure 2] 5 is a flowchart showing an example of a procedure for a three-dimensional (3D) shape display process according to the first embodiment of the present invention. [Figure 3] 3A to 3C are diagrams showing examples of images displayed on a display included in the endoscope system according to the first embodiment of the present invention. [Figure 4] 3A to 3C are diagrams showing examples of images displayed on a display included in the endoscope system according to the first embodiment of the present invention. [Figure 5] 3A to 3C are diagrams showing examples of images displayed on a display included in the endoscope system according to the first embodiment of the present invention. [Figure 6] FIG. 1 is a diagram showing a 3D space defined in 3D image data in a first embodiment of the present invention. [Figure 7]3A to 3C are diagrams showing examples of images displayed on a display included in the endoscope system according to the first embodiment of the present invention. [Figure 8] 3A to 3C are diagrams showing examples of images displayed on a display included in the endoscope system according to the first embodiment of the present invention. [Figure 9] 3A to 3C are diagrams showing examples of images displayed on a display included in the endoscope system according to the first embodiment of the present invention. [Figure 10] 10 is a flowchart showing an example of a procedure for 3D shape display processing in a first modified example of the first embodiment of the present invention. [Figure 11] FIG. 10 is a block diagram showing an example of the configuration of an endoscope system according to a second modified example of the first embodiment of the present invention. [Figure 12] FIG. 10 is a block diagram showing an example of the configuration of an endoscope system according to a third modified example of the first embodiment of the present invention. [Figure 13] 10A and 10B are diagrams illustrating examples of vertex information and face information according to the second embodiment of the present invention. [Figure 14] 10 is a flowchart showing an example of a partial procedure of a 3D shape display process according to the second embodiment of the present invention. [Figure 15] FIG. 10 is a diagram showing the relationship between regions and boundary lines in 3D image data according to the second embodiment of the present invention. [Figure 16] FIG. 10 is a diagram showing an example of face information according to the second embodiment of the present invention. [Figure 17] 10A to 10C are diagrams illustrating examples of 3D shapes used in the 3D shape display processing according to the second embodiment of the present invention. [Figure 18] 10A to 10C are diagrams illustrating examples of 3D shapes used in the 3D shape display processing according to the second embodiment of the present invention. [Figure 19] 10A to 10C are diagrams illustrating examples of 3D shapes used in the 3D shape display processing according to the second embodiment of the present invention. [Figure 20] 10 is a flowchart showing an example of a procedure of a measurement process according to a third embodiment of the present invention. [Figure 21]10A to 10C are diagrams illustrating examples of 3D shapes used in measurement processing in the third embodiment of the present invention. [Figure 22] FIG. 10 is a diagram showing an example of an image displayed on a display included in an endoscope system according to a modified example of the third embodiment of the present invention. [Figure 23] 10 is a flowchart showing an example of a procedure for a rotation process according to a fourth embodiment of the present invention. [Figure 24] 13 is a flowchart showing an example of a procedure for 3D shape display processing according to the fifth embodiment of the present invention. [Figure 25] 13 is a flowchart showing an example of a procedure for 3D shape display processing according to the fifth embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0029] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will now be described with reference to the accompanying drawings. An endoscope system will be described below as an example of a three-dimensional (3D) image display device.

[0030] (First embodiment) Fig. 1 shows an example of the configuration of an endoscope system 1 according to a first embodiment of the present invention. The endoscope system 1 shown in Fig. 1 includes an insertion section 2, a scope unit 3, a base unit 4, and a main unit 5. The insertion section 2, the scope unit 3, and the base unit 4 constitute an endoscope device 10. The main unit 5 is an operation device.

[0031] The insertion section 2 is inserted into the inside of the subject to be observed. The subject is an industrial product. The insertion section 2 is a long, thin tube that is bendable. The user performs an insertion operation to insert the insertion section 2 into the subject. An optical adapter is attached to the tip of the insertion section 2. The insertion section 2 acquires an optical image of the inside of the subject. The insertion section 2 has an imaging section 20, a bending section 21, and an illumination window 22.

[0032] The imaging unit 20 is disposed in the distal end portion 2a including the distal end of the insertion portion 2. The imaging unit 20 is an image sensor such as a CCD (Charge-Coupled Device) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor. The imaging unit 20 generates an image based on an optical image acquired by the insertion portion 2. The image generated by the imaging unit 20 is output to the scope unit 3.

[0033] The bending section 21 bends the insertion section 2 in the upward (U), downward (D), leftward (L), or rightward (R) direction. Alternatively, the bending section 21 bends the insertion section 2 in the upward-left (UL), upward-right (UR), downward-left (DL), or downward-right (DR) direction.

[0034] Illumination light is generated by a light source 35 included in the scope unit 3 and output to the distal end portion 2a through a light guide disposed in the insertion portion 2. The illumination light is irradiated from the illumination window 22 into the inside of the subject.

[0035] The scope unit 3 has an imaging drive circuit 30, an image processing unit 31, a UD drive unit 32, a RL drive unit 33, a bending control unit 34, a light source 35, and a light source control unit 36. The base unit 4 has a control unit 40, a communication unit 41, a volatile memory 42, and a non-volatile memory 43.

[0036] The imaging drive circuit 30 controls the imaging unit 20 and outputs the image output from the imaging unit 20 to the image processing unit 31. The image processing unit 31 performs image processing such as noise reduction on the image output from the imaging unit 20 and outputs the image to the control unit 40.

[0037] The UD driving unit 32 is connected to a UD bending wire for bending the bending portion 21 in the U direction or the D direction. The UD driving unit 32 has a motor, and bends the bending portion 21 in the U direction or the D direction by pulling the UD bending wire. The RL driving unit 33 is connected to an RL bending wire for bending the bending portion 21 in the R direction or the L direction. The RL driving unit 33 has a motor, and bends the bending portion 21 in the R direction or the L direction by pulling the RL bending wire. The bending control unit 34 controls the UD driving unit 32 and the RL driving unit 33.

[0038] The UD drive unit 32 and the RL drive unit 33 can operate simultaneously. For example, the UD drive unit 32 and the RL drive unit 33 can bend the bending section 21 in the UL direction.

[0039] The light source 35 is an LED (Light-Emitting Diode) or the like, and generates illumination light. The illumination light is output from the light source 35 to a light guide. The light source control unit 36 ​​controls the light source 35.

[0040] The control unit 40 controls each unit of the scope unit 3 and the base unit 4. At least one of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36 ​​may be configured with at least one of a processor and a logic circuit. For example, the processor is at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). At least one of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36 ​​may include one or more processors. At least one of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36 ​​may include one or more logic circuits.

[0041] The computer of the endoscope system 1 may load a program and execute the loaded program. The program includes instructions that define the operation of at least one of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36. In other words, the function of at least one of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36 ​​may be realized by software.

[0042] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer storing the program to the endoscope system 1 via a transmission medium or by transmission waves in the transmission medium. A "transmission medium" that transmits the program is a medium that has the function of transmitting information. Media that have the function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the above functions. Furthermore, the above program may be a difference file (difference program). The above functions may be realized by combining a program already recorded on a computer with a difference program.

[0043] The communication unit 41 has a communication circuit and performs wired or wireless communication with the main unit 5 for curvature control and the like. The volatile memory 42 is RAM (Random Access Memory) or DRAM (Dynamic RAM), etc. The volatile memory 42 stores various information processed by the control unit 40. The non-volatile memory 43 is SRAM (Static RAM), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable ROM), or flash memory. The non-volatile memory 43 may be detachable from the base unit 4. The non-volatile memory 43 stores images generated by the imaging unit 20 and various information processed by the control unit 40.

[0044] The main unit 5 has a control unit 50, a display 51, a touch panel 52, operation buttons 53, a communication unit 54, a communication unit 55, a volatile memory 56, and a non-volatile memory 57. The main unit 5 may be an information terminal such as a smartphone or a tablet terminal.

[0045] The control unit 50 controls each part of the main unit 5. The control unit 50 may be composed of at least one of a processor and a logic circuit. The control unit 50 may include one or more processors. The control unit 50 may include one or more logic circuits. The computer of the endoscope system 1 may load a program and execute the loaded program. The program includes instructions that define the operation of the control unit 50. In other words, the functions of the control unit 50 may be realized by software. The program that realizes the functions of the control unit 50 may be realized in the same way as the program that realizes the functions of the control unit 40, etc.

[0046] The display 51 is a monitor such as an LCD (Liquid Crystal Display). The display 51 displays an image generated by the imaging unit 20. The touch panel 52 accepts operations for inputting information necessary for controlling the endoscope system 1. The touch panel 52 is arranged on the screen of the display 51. By operating the touch panel 52, the user can input instructions to change the settings of the endoscope system 1, instructions necessary for operating the endoscope system 1, and the like to the endoscope system 1.

[0047] The operation button 53 accepts various instructions from the user. By pressing the operation button 53, the user can input instructions regarding the power supply or lighting to the endoscope system 1. The communication unit 54 performs wired or wireless communication with the base unit 4 for bending control and the like. The communication unit 55 performs wired or wireless communication with the external device 11. The external device 11 is a remote control, a mouse, or the like.

[0048] The control unit 50 executes a 3D shape display process for displaying an image of the 3D shape of the subject on the display 51. An outline of the 3D shape display process will be described below.

[0049] The control unit 50 acquires 3D image data including 3D coordinates of three or more points on the subject. The 3D coordinates are defined in a 3D space corresponding to real space. The control unit 50 displays an image of the 3D shape of the subject on the display 51 based on the 3D image data. The control unit 50 accepts designation of one or more reference points on the subject. In the first embodiment, the control unit 50 accepts designation of two reference points.

[0050] The control unit 50 sets a reference plane (cutting reference) in the 3D space based on one or more reference points. Specifically, the control unit 50 sets a reference figure based on one or more reference points, and sets a reference plane based on the reference figure. The reference figure is a line, a plane, or a curved surface. In the first embodiment, an example will be described in which a boundary line is used as the reference figure.

[0051] The control unit 50 divides the entire area of ​​the subject into two areas by the reference plane. The cross section of the subject is formed at the intersection of the reference plane and the subject. The control unit 50 changes the display state of one of the two areas. In the first embodiment, the control unit 50 hides one of the two areas. As a result, the cross section of the subject becomes visible.

[0052] 2 shows an example of the procedure for the 3D shape display process. The operation of the endoscope system 1 will be described with reference to FIG.

[0053] The control unit 50 acquires 3D image data (step S100).

[0054] The control unit 50 executes the following process in step S100. First to third examples will be described below.

[0055] First, a first example will be described. The optical adapter in the first example is a stereo optical adapter with two fields of view. The optical adapter has a first optical system and a second optical system corresponding to the two fields of view. The first optical system and the second optical system form two optical images of the subject on the imaging unit 20. The imaging unit 20 generates a stereo image corresponding to the first optical image and the second optical image. The stereo image includes a pair of two images (a first image and a second image). In other words, the stereo image includes an image of the subject viewed from a first viewpoint and an image of the subject viewed from a second viewpoint. The control unit 50 calculates the 3D coordinates of three or more points on the subject by using one or more stereo images generated by the imaging unit 20, and generates 3D image data including the 3D coordinates.

[0056] Next, a second example will be described. The optical adapter in the second example is a monocular optical adapter with one field of view. While the optical adapter in the first example forms two optical images of the subject, the optical adapter in the second example forms one optical image of the subject. The imaging unit 20 generates an image corresponding to the optical image formed by the optical adapter. The imaging unit 20 performs imaging from two or more different viewpoints and generates two or more images. The control unit 50 calculates the 3D coordinates of three or more points on the subject by using the two or more images generated by the imaging unit 20, and generates 3D image data including the 3D coordinates.

[0057] Next, a third example will be described. The 3D image data generated in the first or second example is stored in advance in the nonvolatile memory 57. The control unit 50 acquires the 3D image data from the nonvolatile memory 57.

[0058] After step S100, the control unit 50 generates a 3D image based on the 3D image data and outputs the 3D image to the display 51. The display 51 displays the 3D image (step S101). The 3D image is an image of the 3D shape of the subject.

[0059] FIG. 3 shows an example of a 3D image displayed on the display 51 in step S101. The control unit 50 displays the 3D image IMG10 shown in FIG. 3 on the display 51. The 3D image IMG10 includes a 3D shape SH10 of the subject. The control unit 50 displays an icon IC10 on the 3D image IMG10. The icon IC10 includes an illustration of the subject and prompts the user to set a first reference point. The icon IC10 also notifies the user of areas that will be hidden. For example, the icon IC10 notifies the user that the area in front of a reference plane that passes through two reference points will be hidden.

[0060] The control unit 50 displays a cursor CS10 on the 3D image IMG10. The user moves the cursor CS10 on the 3D image IMG10 by operating the touch panel 52. The user may also move the cursor CS10 on the 3D image IMG10 by operating the operation button 53, the external device 11, or the like.

[0061] The user operates the touch panel 52 or the like to input an instruction to set a first reference point to the endoscope system 1. The control unit 50 determines whether or not the instruction has been input (step S102). When the control unit 50 determines in step S102 that the instruction has not been input, the control unit 50 repeats step S102.

[0062] When the control unit 50 determines in step S102 that the instruction has been input, the control unit 50 accepts the instruction and sets a first reference point at the position of the cursor CS10 (step S103). The first reference point is one of three or more points included in the 3D image data. Information about the first reference point is stored in the volatile memory 56.

[0063] 4 shows an example of a 3D image displayed on the display 51 when the first reference point is set in step S103. Portions that are the same as those shown in FIG.

[0064] The control unit 50 displays an icon IC11 on the 3D image IMG 10. The icon IC11 includes an illustration of the subject and prompts the user to set a second reference point. The icon IC11 also notifies the user of the area that will be hidden.

[0065] The user operates the touch panel 52 or the like to move the cursor CS10 on the 3D image IMG10. The user operates the touch panel 52 or the like to input an instruction to set a second reference point to the endoscope system 1. The control unit 50 determines whether or not the instruction has been input (step S104).

[0066] If the control unit 50 determines in step S104 that the instruction has not been input, the control unit 50 highlights the line connecting the first reference point and the point on the cursor on the 3D image (step S109). After step S109, step S104 is executed.

[0067] 5 shows an example of a 3D image displayed on the display 51 in step S109. Portions that are the same as those shown in FIG. 4 will not be described.

[0068] The control unit 50 displays a line L10 on the 3D image IMG10. The line L10 is a straight line connecting the first reference point set in step S103 and the point indicated by the cursor CS10. The line L10 is displayed as a thick line. When the user moves the cursor CS10, the line L10 moves.

[0069] When the control unit 50 determines in step S104 that an instruction to set a second reference point has been input, the control unit 50 accepts the instruction and sets the second reference point at the position of the cursor CS10 (step S105). The second reference point is one of three or more points included in the 3D image data and is different from the first reference point. Information about the second reference point is stored in the volatile memory 56.

[0070] After step S105, the control unit 50 sets a reference plane that passes through the first reference point and the second reference point (step S106).

[0071] Step S106 will now be described in detail. Fig. 6 shows the 3D space defined in the 3D image data. The 3D image data includes 3D coordinates of three or more points that form the 3D shape SH11 of the subject. The 3D coordinates of the three or more points are composed of coordinate values ​​along the mutually orthogonal X-axis, Y-axis, and Z-axis.

[0072] The control unit 50 sets a first reference point RP10 in step S103, and sets a second reference point RP11 in step S105. The control unit 50 sets a boundary line BL10 that passes through the first reference point RP10 and the second reference point RP11 in step S106. The control unit 50 also sets a plane PL10 that includes the boundary line BL10 and is parallel to the Z axis in step S106. The plane PL10 is a reference plane.

[0073] After step S106, the control unit 50 divides the entire region of the 3D image data into a first region and a second region (step S107). Dividing the entire region into two regions means defining the two regions in the entire region. In other words, dividing the entire region into two regions means assigning each point included in the entire region to one of the two regions. The boundary between the first region and the second region is the reference plane set in step S106. Each of the three or more points included in the 3D image data is included in the first region or the second region.

[0074] After step S107, the control unit 50 changes the display state of the 3D image displayed on the display 51. Specifically, the control unit 50 changes the display state of one of the first region and the second region. For example, the control unit 50 hides one of the first region and the second region (step S108). When step S108 is executed, the 3D shape display process shown in FIG. 2 ends.

[0075] Instead of hiding one of the first region and the second region, the control unit 50 may increase the transparency of one of the first region and the second region. For example, before step S108 is performed, the transparency of the first region and the second region is 0%. In step S108, the control unit 50 sets the transparency of one of the first region and the second region to a value greater than 0% and less than or equal to 100%. The control unit 50 may also set the transparency of one of the first region and the second region to 50% or more.

[0076] When the first or second region is hidden, the cross-section hidden by the first or second region is displayed. When the transparency of either the first or second region is increased, the cross-section hidden by the first or second region becomes visible. This allows the user to easily understand the 3D shape of the cross-section. Since the 3D shape of the cross-section itself as well as the 3D shape of the area surrounding the cross-section are displayed, the user can easily intuitively understand the 3D shape of the subject.

[0077] 7 shows an example of a 3D image displayed on the display 51 in step S108. Portions that are the same as those shown in FIG. 5 will not be described.

[0078] For example, the first region is located on the positive side of the Y axis from the reference plane set in step S106. For example, the second region is located on the negative side of the Y axis from the reference plane. The control unit 50 hides one or more points included in the second region. As a result, the cross section of the 3D shape SH10 is visualized. The control unit 50 also displays a line L11 on the 3D image IMG10 that highlights the cross section. For example, the line L11 is displayed with a specific thickness and a specific color.

[0079] After step S108, the user may input an instruction to rotate, enlarge, or reduce the 3D image displayed on the display 51 by operating the touch panel 52 or the like. The control unit 50 may accept the instruction. When an instruction to rotate the 3D image is accepted, the control unit 50 may rotate the 3D image displayed on the display 51. When an instruction to enlarge or reduce the 3D image is accepted, the control unit 50 may enlarge or reduce the 3D image displayed on the display 51.

[0080] 8 shows an example of a 3D image displayed on the display 51 when an instruction to rotate the 3D image is received. Portions that are the same as those shown in FIG. 7 will not be described.

[0081] The control unit 50 receives an instruction to rotate the 3D image IMG10 shown in Fig. 7. At this time, the control unit 50 displays the 3D image IMG11 shown in Fig. 8 on the display 51. The 3D shape SH10 in the 3D image IMG11 is displayed at a different angle from the 3D shape SH10 in the 3D image IMG10.

[0082] 9 shows an example of a 3D image displayed on the display 51 when an instruction to enlarge the 3D image is received. Portions that are the same as those shown in FIG. 7 will not be described.

[0083] The control unit 50 receives an instruction to rotate the 3D image IMG11 shown in Fig. 8. At this time, the control unit 50 displays the 3D image IMG12 shown in Fig. 9 on the display 51. The 3D shape SH10 in the 3D image IMG12 is enlarged more than the 3D shape SH10 in the 3D image IMG11.

[0084] The user may operate the touch panel 52 with a finger or a stylus to input a reference point. After the first reference point is set, the user may move the cursor by performing a drag (swipe) operation, or may input a second reference point by performing a drop (release) operation. The user may also specify the position of the second reference point by performing a drop (release) operation, or may confirm the position by performing a long tap (click) operation.

[0085] The user may operate the operation button 53 to input the reference point. The main unit 5 may have a joystick, and the user may operate the joystick to input the reference point. The external device 11 may be a mouse, and the user may operate the mouse to input the reference point.

[0086] The control unit 50 may display an image of the 3D shape of the entire or part of the subject based on the 3D image data on the display 51. The control unit 50 may display the image of the 3D shape of the entire or part of the subject next to the image IMG10 shown in FIG.

[0087] The control unit 50 may display a two-dimensional (2D) cross-sectional shape outline showing the 3D shape in a specific cross section on the display 51. The control unit 50 may display the 2D cross-sectional shape outline next to the image IMG10 shown in FIG.

[0088] The control unit 50 may display a 3D color map that indicates the distance between a specific reference position and each point on the 3D shape using a color on the display 51. The control unit 50 may display the 3D color map next to the image IMG10 shown in FIG.

[0089] The control unit 50 may have a function to modify the reference plane as follows. For example, the control unit 50 cancels the setting of the first reference point or the second reference point in accordance with an instruction from the user. After the setting of the first reference point or the second reference point is canceled, the control unit 50 sets a new reference point in accordance with an instruction from the user. For example, after the setting of the first reference point is canceled, the control unit 50 sets a new first reference point. The control unit 50 sets a new reference plane based on two reference points including the newly set reference point.

[0090] The control unit 50 may correct the reference plane as follows. For example, the control unit 50 moves the line L11 shown in FIG. 7 in accordance with an instruction from the user. While the line L11 is moving, the control unit 50 sets a new reference plane that passes through the line L11. The instruction may be, for example, an instruction to directly move the line L11, or an instruction to move the line L11 by moving a reference point.

[0091] After the reference plane is corrected as described above, steps S107 and S108 shown in FIG. 2 are executed, and the cross section of the 3D shape of the subject is displayed again.

[0092] The control unit 50 may execute the following processes instead of steps S102 to S105 and S109 shown in FIG. 2. After step S101, the control unit 50 detects a characteristic region in an image used to generate the 3D image data. That is, the control unit 50 detects a characteristic region in a stereo image including two optical images formed by a stereo optical adapter. Alternatively, the control unit 50 detects a characteristic region in an image including an optical image formed by a monocular optical adapter. For example, the characteristic region is an abnormal region such as a scratch. The control unit 50 detects an abnormal region having a predetermined color or shape.

[0093] Machine learning may be used to detect abnormal regions. For example, deep learning may be used as machine learning. For example, a user observes an image acquired in a previous examination and determines whether a region shown in the image is abnormal or normal. The control unit 50 analyzes the features of the image using the image and correct answer data (teaching data) indicating the result of the user's judgment, and generates a trained model. The non-volatile memory 57 stores the trained model.

[0094] After the image capturing unit 20 captures an image, the control unit 50 inputs the image into the trained model and determines whether the area in the image is abnormal or normal. If the area is abnormal, the control unit 50 extracts the area as an abnormal area.

[0095] The positions on the image used to generate the 3D image data and each point included in the 3D image data are associated with each other. The control unit 50 identifies points in the 3D image data that correspond to each point included in the feature region. The control unit 50 sets a reference plane based on one or more identified points.

[0096] The control unit 50 may display the 3D shapes of two or more cross sections on the display 51. For example, the control unit 50 executes the following process. In step S106 shown in FIG. 2, the control unit 50 sets a first reference plane that includes a boundary line connecting the first reference point and the second reference point and is parallel to the Z axis. The control unit 50 also sets a line that is perpendicular to the boundary line connecting the first reference point and the second reference point set in the process shown in FIG. 2. The set line passes through two or more points on the 3D shape of the subject. The control unit 50 then sets a second reference plane that includes the set line and is parallel to the Z axis. The control unit 50 executes steps S107 and S108 shown in FIG. 2 for each of the first reference plane and the second reference plane, and displays the 3D shapes of the two cross sections on the display 51.

[0097] The control unit 50 may receive an instruction to set three or more reference points. The control unit 50 may select two reference points from the three or more reference points and set a reference plane based on the two reference points. The two selected reference points may be switchable.

[0098] The control unit 50 may calculate the 3D coordinates of a point that does not exist on the 3D shape of the subject and may use the point with the 3D coordinates as a reference point. For example, the control unit 50 may calculate the 3D coordinates of the center of gravity of three points on the 3D shape of the subject and use the center of gravity as a reference point. Alternatively, the control unit 50 may estimate a plane or a curved surface based on three or more points on the 3D shape of the subject and use a point on that plane or curved surface outside the 3D shape as a reference point. Alternatively, the control unit 50 may use the origin of the 3D coordinates in the 3D image data (camera origin) as a reference point. Alternatively, the control unit 50 may calculate a line based on two points on the 3D shape of the subject and use a point on that line outside the 3D shape as a reference point.

[0099] A 3D image display device according to each aspect of the present invention includes a control unit 50. The control unit 50 acquires 3D image data including 3D coordinates of three or more points on a subject calculated based on an endoscopic image of the subject. The control unit 50 displays a 3D shape of the subject on a display 51 based on the 3D image data. The control unit 50 sets a reference plane based on one or more points included in the three or more points. The control unit 50 divides the 3D image data into a first region and a second region using the reference plane as a boundary. The control unit 50 makes the display state of the 3D shape of the subject in one of the first region and the second region different from the display state of the 3D shape of the subject in the other of the first region and the second region in order to display a cross section of the subject set based on the reference plane.

[0100] A 3D image display method according to each aspect of the present invention includes first to fifth steps. In a first step (step S100), the control unit 50 acquires 3D image data including 3D coordinates of three or more points calculated based on an endoscopic image of the subject. In a second step (step S101), the control unit 50 displays a 3D shape of the subject on the display 51 based on the 3D image data. In a third step (step S106), the control unit 50 sets a reference plane based on one or more points included in the three or more points. In a fourth step (step S107), the control unit 50 divides the 3D image data into a first region and a second region using the reference plane as a boundary. In a fifth step (step S108), the control unit 50 differentiates the display state of the 3D shape of the subject in one of the first region and the second region from the display state of the 3D shape of the subject in the other of the first region and the second region in order to display a cross section of the subject set based on the reference plane.

[0101] The program according to each aspect of the present invention causes a computer to execute the first to fifth steps described above.

[0102] Each aspect of the present invention may include the following modifications: The control unit 50 hides one of the first area and the second area.

[0103] Each aspect of the present invention may include the following modifications: The control unit 50 makes the transparency of one of the first region and the second region higher than the transparency of the other of the first region and the second region.

[0104] Each aspect of the present invention may include the following modifications: The control unit 50 changes the display state of the cross section of the subject so that the cross section is emphasized.

[0105] Each aspect of the present invention may include the following modifications: The control unit 50 receives an instruction to rotate the 3D shape of the subject, and the control unit 50 displays the 3D shape rotated in accordance with the instruction on the display 51.

[0106] Each aspect of the present invention may include the following modifications: The control unit 50 receives an instruction to enlarge or reduce the 3D shape of the subject, and the control unit 50 displays the enlarged or reduced 3D shape on the display 51 in accordance with the instruction.

[0107] Each aspect of the present invention may include the following modifications: The control unit 50 displays, based on 3D image data, a 3D shape of the entire or part of the subject adjacent to the 3D shape of the subject, which includes a first region and a second region having a display state different from the display state of the first region.

[0108] Each aspect of the present invention may include the following modifications: The control unit 50 sets a reference point based on one or more points included in the above three or more points. The control unit 50 sets a reference plane based on the reference point.

[0109] Each aspect of the present invention may include the following modifications: The control unit 50 sets two or more reference points.

[0110] Each aspect of the present invention may include the following modifications: The control unit 50 sets a reference graphic based on one or more points included in the above three or more points. The control unit 50 sets a reference plane based on the reference graphic.

[0111] Each aspect of the present invention may include the following modifications: The control unit 50 detects a characteristic region in an endoscopic image, and sets a reference plane based on one or more points included in the characteristic region out of the three or more points.

[0112] Each aspect of the present invention may include the following modifications: The endoscopic image includes a first image and a second image that are stereo images. The control unit 50 generates 3D image data based on the first image and the second image.

[0113] Each aspect of the present invention may include the following modifications: An endoscopic image is generated based on an optical image of a subject formed by a monocular optical system (optical adapter). The control unit 50 generates 3D image data based on two or more endoscopic images.

[0114] Each aspect of the present invention may include the following modifications: The control unit 50 acquires 3D image data from the nonvolatile memory 57 (storage medium).

[0115] As described above, the control unit 50 displays the 3D shape of the cross section of the 3D shape of the subject on the display 51. Therefore, the endoscope system 1 can facilitate intuitive understanding of the 3D shape of the subject.

[0116] The user can change the angle of the 3D shape by rotating the 3D image including the cross-sectional 3D shape. Alternatively, the user can change the size of the 3D shape by zooming in or out on the 3D image including the cross-sectional 3D shape. Therefore, the endoscope system 1 can improve the visibility of the cross-sectional 3D shape.

[0117] (First Modification of the First Embodiment) A first modification of the first embodiment of the present invention will be described. In the first modification of the first embodiment, information on the area to be hidden is stored in nonvolatile memory 57, and control unit 50 hides the first area or the second area in accordance with the information.

[0118] The first region is set on the positive side of the Y axis from the reference plane. The second region is set on the negative side of the Y axis from the reference plane. The user selects the first region or the second region by operating the touch panel 52 or the like. The control unit 50 generates region information indicating the selected region and stores the region information in the non-volatile memory 57.

[0119] Fig. 10 shows an example of the procedure for 3D shape display processing. The operation of the endoscope system 1 will be described using Fig. 10. Processing that is the same as the processing shown in Fig. 2 will not be described.

[0120] After step S107, the control unit 50 acquires the area information from the nonvolatile memory 57 (step S110).

[0121] After step S110, the control unit 50 changes the display state of the first region or the second region indicated by the region information. For example, if the region information indicates the first region, the control unit 50 hides the first region (step S108a). When step S108a is executed, the 3D shape display process shown in FIG. 10 ends.

[0122] Each aspect of the present invention may include the following modifications: The control unit 50 acquires region information indicating one of the first region and the second region, and changes the display state of the region indicated by the region information.

[0123] In a first variation of the first embodiment, the user can select the first area or the second area to be hidden.

[0124] (Second Modification of the First Embodiment) A second modified example of the first embodiment of the present invention will be described. Fig. 11 shows an example of the configuration of an endoscope system 1a according to the second modified example of the first embodiment. The same parts as those shown in Fig. 1 will not be described.

[0125] An endoscope system 1a shown in Fig. 11 includes an insertion portion 2 and a main unit 6. The insertion portion 2 and the main unit 6 constitute an endoscope device 10a.

[0126] The insertion section 2 shown in Fig. 11 is the same as the insertion section 2 shown in Fig. 1. The main unit 6 has an imaging drive circuit 30, an image processing section 31, a UD drive section 32, a RL drive section 33, a bending control section 34, a light source 35, a light source control section 36, a display 51, a touch panel 52, operation buttons 53, a communication section 55, a volatile memory 56, a non-volatile memory 57, and a control section 60. The same reference symbols as those in Fig. 1 are assigned to blocks that are the same as those in Fig. 1.

[0127] The control unit 60 has both the functions of the control unit 40 shown in Fig. 1 and the functions of the control unit 50 shown in Fig. 1. The control unit 60 executes the processing shown in Fig. 2.

[0128] Each aspect of the present invention may include the following modifications: The endoscope system 1a includes an endoscope device 10a, and the endoscope device 10a includes an insertion section 2, a touch panel 52 (device), and a control section 60.

[0129] In the second modified example of the first embodiment, the endoscope system 1a can facilitate intuitive understanding of the 3D shape of the subject, similarly to the first embodiment.

[0130] (Third Modification of the First Embodiment) A third modified example of the first embodiment of the present invention will be described. Fig. 12 shows an example of the configuration of an endoscope system 1b according to the third modified example of the first embodiment. The same parts as those shown in Fig. 1 will not be described.

[0131] 12 includes an insertion section 2, a scope unit 3b, and a base unit 7. The insertion section 2 and the scope unit 3b constitute an endoscope device 10b. The scope unit 3b and the base unit 7 are connected by a cable 8.

[0132] The insertion section 2 shown in Fig. 12 is the same as the insertion section 2 shown in Fig. 1. The scope unit 3b shown in Fig. 12 is the same as the scope unit 3 shown in Fig. 1, except that it does not have the image processing section 31. The base unit 7 has the image processing section 31, a display 51, a touch panel 52, operation buttons 53, a communication section 55, a volatile memory 56, a non-volatile memory 57, and a control section 70. The same reference symbols as those in Fig. 1 are assigned to blocks that are the same as those in Fig. 1.

[0133] The control unit 70 has both the functions of the control unit 40 shown in Fig. 1 and the functions of the control unit 50 shown in Fig. 1. The control unit 70 executes the processing shown in Fig. 2.

[0134] In the third modified example of the first embodiment, the endoscope system 1b can facilitate intuitive understanding of the 3D shape of the subject, similarly to the first embodiment.

[0135] (Second embodiment) A second embodiment of the present invention will be described. In the second embodiment, an endoscope system 1 shown in Fig. 1 is used. An endoscope system 1a shown in Fig. 11 or an endoscope system 1b shown in Fig. 12 may also be used.

[0136] A method for hiding a part of the 3D shape of the object will be described in detail below. The method described below can be applied to the first embodiment.

[0137] The 3D image data in the second embodiment includes vertex information about three or more points (vertices) on the object and surface information about a surface formed by three vertices. In the following example, the 3D image data includes vertex information about four or more vertices and surface information about two or more surfaces.

[0138] The 3D image data includes vertex information VI1 and surface information SI1 shown in Fig. 13. The vertex information VI1 includes a vertex index and 3D coordinates of each vertex in the 3D image data. The surface information SI1 includes a surface index of each surface in the 3D image data and vertex indexes of three vertices included in each surface. In step S101, the control unit 50 displays a 3D shape SH1 on the display 51 based on the vertex information VI1 and the surface information SI1.

[0139] The endoscope system 1 executes the processing shown in Fig. 14 instead of steps S106 to S108 shown in Fig. 2. That is, the endoscope system 1 executes steps S100 to S105 and S109 shown in Fig. 2 and the processing shown in Fig. 14. Fig. 14 shows an example of the procedure for 3D shape display processing. The operation of the endoscope system 1 will be described using Fig. 14.

[0140] The control unit 50 calculates an equation of a boundary line that passes through the first reference point and the second reference point (step S120).

[0141] The control unit 50 executes the following process in step S120. The control unit 50 sets a first reference point P1 in step S103 shown in Fig. 2, and sets a second reference point P2 in step S105 shown in Fig. 2. The equation of the boundary line that passes through the first reference point P1 and has a direction vector d is expressed by the following formula (1). In formula (1), a parameter is expressed as t. P=P1+t*d (1)

[0142] For example, when the 3D coordinates of the first reference point P1 are (2, 3, 1) and the direction vector d is (1, -1, 1), the equation of the boundary line is expressed by the following formula (2). (x,y,z)=(2,3,1)+t*(1,-1, 1) (2)

[0143] For example, when the parameter t is -1, (x,y,z)=(3,2,2) holds. The point (3,2,2) is on the boundary line.

[0144] The direction vector d is expressed by the following equation (3) using the first reference point P1 and the second reference point P2.

[0145]

number

[0146] The equation of the boundary line is expressed as the following equation (4) using vector elements:

[0147]

number

[0148] After step S120, the control unit 50 refers to the positional relationship between each vertex on the object and the boundary line, and identifies vertices that will not be displayed (step S121).

[0149] The control unit 50 executes the following process in step S121: In the following, a vertex that is displayed in the 3D image is defined as a first vertex, and a vertex that is not displayed in the 3D image is defined as a second vertex.

[0150] The control unit 50 identifies the second vertex according to the direction vector d of the boundary line. Figures 15(a) and 15(b) show the relationship between the regions in the 3D image data and the boundary lines.

[0151] The boundary line L1 shown in Figures 15(a) and 15(b) passes through the first reference point P1 and the second reference point P2. The arrow of the boundary line L1 indicates the direction of the direction vector d. The entire area in the 3D image data is divided into a first area R1 and a second area R2. The boundary line L1 constitutes the boundary between the first area R1 and the second area R2.

[0152] In the example shown in Figure 15(a), the second vertex is included in the second region R2. That is, the second vertex is located on the negative side of the Y axis relative to the boundary line L1. On the other hand, in the example shown in Figure 15(b), the second vertex is included in the first region R1. That is, the second vertex is located on the positive side of the Y axis relative to the boundary line L1. As described above, the region that is not displayed in the 3D image is set according to the direction vector d of the boundary line L1.

[0153] After step S121, the control unit 50 identifies a face including the second vertex (step S122). After step S122, the control unit 50 deletes information about the face including the second vertex from the face information and hides the face including the second vertex (step S123).

[0154] FIG. 16 shows an example of face information. The face information SI2 shown in FIG. 16 includes a face index and a vertex index. For the 3D shape SH2 shown in FIG. 16, the control unit 50 calculates a boundary line L2. The control unit 50 identifies a second vertex that is located on the negative side of the Y axis from the boundary line L2. The control unit 50 identifies a vertex with a vertex index 5 and a vertex with a vertex index 7 as the second vertices.

[0155] In step S123, the control unit 50 generates face information SI2 by deleting information about the second vertex from the face information SI1 shown in Fig. 13. The face information SI2 does not include information about the face including the vertex with vertex index 5 and information about the face including the vertex with vertex index 7.

[0156] The surface information SI2 includes the changed surface index. For example, in the surface information SI1 shown in FIG. 13, the surface having surface index 3 includes a vertex having vertex index 2, a vertex having vertex index 4, and a vertex having vertex index 3. In step S123, the control unit 50 deletes the information about the surface having surface index 2 from the surface information. Therefore, the surface having surface index 3 is changed to a surface having surface index 2 in the surface information SI2.

[0157] After step S123, the control unit 50 extracts two or more vertices in the cross section of the subject (step S124). After step S124, the control unit 50 rearranges the two or more extracted vertices in a predetermined order (step S125). After step S125, the control unit 50 connects the two or more extracted vertices with a line and highlights the line on the display 51 (step S126).

[0158] Steps S121 to S126 will be described in detail with reference to Figures 17 to 19. Figures 17 to 19 show examples of 3D shapes used in the 3D shape display process.

[0159] 2, the control unit 50 displays a 3D image of the 3D shape SH12 shown in Fig. 17 on the display 51. In step S106 shown in Fig. 2, the control unit 50 sets a boundary line L12 shown in Fig. 17.

[0160] In step S121, the control unit 50 identifies a second vertex to be hidden. In step S122, the control unit 50 identifies a face that includes the second vertex. One face includes three vertices. If at least one of the three vertices is the second vertex, the control unit 50 identifies the face. In step S123, the control unit 50 hides the face.

[0161] The control unit 50 displays the 3D shape SH13 shown in Fig. 18 on the display 51 instead of the 3D shape SH12 shown in Fig. 17. The surface identified in step S123 is not displayed.

[0162] In step S124, the control unit 50 executes the following process: The control unit 50 identifies a plane including the first vertex and the second vertex, and selects the first vertex of the identified plane as the vertex of the cross-sectional portion.

[0163] In step S125, the control unit 50 executes the following process. The control unit 50 sorts the information about two or more vertices of the cross-sectional portion in descending order of the X coordinate of each vertex. The control unit 50 also sorts the information about two or more vertices having the same X coordinate in descending order of the Y coordinate of each vertex.

[0164] In step S126, the control unit 50 executes the following process. The control unit 50 connects two or more vertices of the cross-sectional portion with a thick line in the sorted order. The control unit 50 displays the line connecting the two or more vertices of the cross-sectional portion on the display 51. For example, the control unit 50 displays the line L13 shown in FIG. 19 on the display 51.

[0165] In the second embodiment, the endoscope system 1 can facilitate intuitive understanding of the 3D shape of the subject, similarly to the first embodiment.

[0166] (Third embodiment) A third embodiment of the present invention will be described. In the third embodiment, an endoscope system 1 shown in Fig. 1 is used. An endoscope system 1a shown in Fig. 11 or an endoscope system 1b shown in Fig. 12 may also be used.

[0167] The endoscope system 1 according to the third embodiment has a measurement function. The endoscope system 1 sets a measurement reference that is different from the cross section of the subject and measures the 3D distance between the measurement reference and each point on the cross section. The measurement reference is a point, a line, or a surface. The endoscope system 1 displays the deepest point and the highest point on the cross section.

[0168] For example, the control unit 50 sets the center of the tip of the insertion unit 2 as the measurement reference point. Alternatively, the user operates the touch panel 52 or the like to input an instruction to set the measurement reference point to the endoscope system 1. The control unit 50 sets the measurement reference point according to the instruction.

[0169] Alternatively, the user operates the touch panel 52 or the like to input an instruction to set two points to the endoscope system 1. The control unit 50 sets a measurement reference line connecting those two points. Alternatively, the user operates the touch panel 52 or the like to input an instruction to set three or more points to the endoscope system 1. The control unit 50 sets a measurement reference plane that passes through those three or more points.

[0170] After executing the 3D shape display process shown in Fig. 2, the endoscope system 1 executes the measurement process shown in Fig. 20. Fig. 20 shows an example of the procedure of the measurement process. The operation of the endoscope system 1 will be described using Fig. 20.

[0171] The control unit 50 acquires the 3D coordinates of the points on the boundary line set in step S106. If the points on the boundary line are on the subject, the control unit 50 acquires the 3D coordinates of the points from the 3D image data. If the points on the boundary line are not on the subject, the control unit 50 calculates the 3D coordinates of the points (step S130).

[0172] After step S130, the control unit 50 determines whether or not a measurement reference line or a measurement reference plane has been set (step S131).

[0173] When the control unit 50 determines in step S131 that a measurement reference line or a measurement reference plane has been set, the control unit 50 calculates the 3D distance between the measurement reference line and each point on the boundary line. Alternatively, the control unit 50 calculates the 3D distance between the measurement reference plane and each point on the boundary line. At this time, the control unit 50 calculates signed 3D distances. The 3D distance of each point on the front side of the measurement reference plane as viewed from the tip of the insertion unit 2 has a positive sign. The 3D distance of each point on the back side of the measurement reference plane as viewed from the tip of the insertion unit 2 has a negative sign (step S132).

[0174] When the control unit 50 determines in step S131 that a measurement reference line or measurement reference plane has not been set, the control unit 50 calculates the 3D distance between the tip of the insertion unit 2 and each point on the boundary line. At this time, the control unit 50 calculates the 3D distance without a sign (step S133). The Z axis in the 3D space defined in the 3D image data is parallel to the optical axis of the optical adapter. Furthermore, the origin of the 3D space is set to the tip of the insertion unit 2. Therefore, the 3D distance is expressed by the Z coordinate of each point.

[0175] The control unit 50 calculates the 3D distance at each of two or more points on the boundary line in step S132 or step S133. If an unsigned 3D distance is used in step S133, the control unit 50 identifies the point whose 3D distance is maximum (farthest point) and the point whose 3D distance is minimum (closest point). The control unit 50 highlights the two identified points on the 3D image displayed on the display 51. As a result, the control unit 50 highlights the farthest point and the closest point on the cross section (step S134). Furthermore, if a signed 3D distance is used in step S132, the control unit 50 identifies the point whose 3D distance is maximum in the positive direction (highest point) and the point whose 3D distance is maximum in the negative direction (deepest point). The control unit 50 highlights the two identified points on the 3D image displayed on the display 51. As a result, the control unit 50 highlights the highest point and the deepest point on the cross section (step S134).

[0176] After step S134, the control unit 50 displays the 3D distance values ​​of each of the farthest point and the nearest point (or the deepest point and the highest point) on the 3D image (step S135). When step S135 is executed, the measurement process shown in Fig. 20 ends.

[0177] FIG. 21 shows an example of the 3D shape of the subject displayed on the display 51 in steps S134 and S135.

[0178] The control unit 50 displays the 3D shape SH13 on the display 51. The control unit 50 displays a line L13 connecting two or more vertices of the cross-sectional portion on the 3D shape SH13. The control unit 50 displays the deepest point DP10 and the highest point HP10 on the 3D shape SH13. The control unit 50 displays the 3D distance value DV10 at the deepest point DP10 on the 3D shape SH13, and displays the 3D distance value DV11 at the highest point HP10 on the 3D shape SH13.

[0179] The control unit 50 may display the measurement reference on the display 51. The user can easily understand the reference position for measuring the 3D distance.

[0180] The control unit 50 may perform point-to-point distance measurement or surface-based measurement at any measurement position on the 3D shape of the object. After such a measurement is performed, the control unit 50 may display a cross section of the object. In this example, the control unit 50 can display the cross section of the object with the measurement reference, measurement results, etc. displayed.

[0181] Each aspect of the present invention may include the following modifications: The control unit 50 sets a measurement reference based on one or more points on the object, and calculates the distance between the measurement reference and a point on the cross section of the object.

[0182] Each aspect of the present invention may include the following modifications: The control unit 50 calculates the distance between the measurement reference and each of two or more points on the cross section of the subject, and identifies the point among the two or more points with the maximum or minimum distance.

[0183] Each aspect of the present invention may include the following modifications: The control unit 50 superimposes information indicating the point where the above-mentioned distance is maximum or minimum on the 3D shape of the subject.

[0184] In the third embodiment, the endoscope system 1 can perform measurements on a cross section of the subject.

[0185] (Modification of the third embodiment) A modification of the third embodiment of the present invention will now be described. In the modification of the third embodiment, the endoscope system 1 performs measurements on a part of a cross section of the subject.

[0186] After executing the 3D shape display process shown in Fig. 2, the endoscope system 1 executes the following process. The control unit 50 sets a target range including a part of a cross section of the subject. In step S130 shown in Fig. 20, the control unit 50 acquires 3D coordinates of two or more points included in the cross section in the target range. Thereafter, the control unit 50 executes steps S131 to S135.

[0187] Fig. 22 shows an example of a 3D image displayed on the display 51 after step S108 shown in Fig. 2. Portions that are the same as those shown in Fig. 7 will not be described.

[0188] The user operates the touch panel 52 or the like to input an instruction to the endoscope system 1 to set two range reference points on a cross section of the subject. The control unit 50 accepts the instruction and sets a first range reference point P10 and a second range reference point P11 shown in FIG. 22. For example, the first range reference point P10 and the second range reference point P11 are set at positions other than the ends of the cross section. The control unit 50 sets a target range R10 that includes the first range reference point P10 and the second range reference point P11. The target range R10 includes two or more points included in the cross section. The control unit 50 displays the first range reference point P10, the second range reference point P11, and the target range R10 on the 3D shape SH10.

[0189] Each aspect of the present invention may include the following modifications: The control unit 50 sets a range including a part of the cross section of the subject in the 3D image data, and calculates the distance between the measurement reference and each of two or more points included in the range.

[0190] In a modification of the third embodiment, the control unit 50 calculates the distance between the measurement reference and each of two or more points included in a target range set on a cross section of the subject. 3D image data at the end of the cross section may contain noise. By setting the target range, the endoscope system 1 can eliminate the influence of noise on the measurement results.

[0191] (Fourth embodiment) A fourth embodiment of the present invention will be described. In the fourth embodiment, an endoscope system 1 shown in Fig. 1 is used. An endoscope system 1a shown in Fig. 11 or an endoscope system 1b shown in Fig. 12 may also be used.

[0192] In the fourth embodiment, the control unit 50 sets a reference plane based on three reference points. First to fourth methods for setting the reference plane will be described below.

[0193] First, the first method will be described. The control unit 50 sets a plane passing through the first reference point, the second reference point, and the origin of the 3D image data as a reference plane. The first reference point and the second reference point are set by the method of the first embodiment. The origin of the 3D image data corresponds to the camera position when the imaging unit 20 generates the image. The origin of the 3D image data may coincide with the center of the tip of the insertion unit 2. In the first method, the reference plane corresponds to a straight line on a distortion-corrected image generated by correcting the optical distortion of the image generated by the imaging unit 20. Therefore, the user can easily understand the position through which the reference plane passes on the distortion-corrected image.

[0194] Next, the second method will be described. In the first method, it is difficult to visualize a cross section of a region that is not viewed from the front in the image (the image generated by the imaging unit 20) used to generate the 3D image data that is the basis of the 3D image displayed on the display 51. In the second method, the control unit 50 rotates the 3D image so that the region is viewed from the front.

[0195] When the area to be visualized as a cross section is not viewed from the front, the user rotates the 3D image displayed on the display 51 by operating the touch panel 52 or the like. This causes the area to be viewed from the front on the 3D image. The control unit 50 then sets a plane passing through the first reference point, the second reference point, and the origin of the 3D view camera as a reference plane. The first reference point and the second reference point are set using the method of the first embodiment. The origin of the 3D view camera corresponds to the viewpoint of the perspective projection transformation when a 3D image is generated from 3D image data and displayed on the display 51.

[0196] The control unit 50 may rotate the 3D image so that the area near the point set in accordance with an instruction from the user is viewed from the front. The rotation process for rotating the 3D image will be described in detail below.

[0197] The endoscope system 1 executes the rotation process shown in Fig. 23 before executing the 3D shape display process shown in Fig. 2. Alternatively, the endoscope system 1 executes the rotation process during the 3D shape display process shown in Fig. 23. Fig. 23 shows an example of the procedure for the rotation process. The operation of the endoscope system 1 will be described using Fig. 23.

[0198] The user operates the touch panel 52 or the like to input an instruction to set a rotation reference point to the endoscope system 1. The control unit 50 accepts the instruction and sets the rotation reference point (step S140). The rotation reference point may be the first reference point set in step S103 shown in FIG. 2 or the second reference point set in step S105 shown in FIG. 2. The rotation reference point may be a point different from the first reference point or the second reference point. The rotation reference point may be the midpoint of a line segment connecting the first reference point and the second reference point.

[0199] After step S140, the control unit 50 sets a sphere having a predetermined radius and centered on the rotation reference point as an extraction range (step S141). The extraction range includes three or more points included in the 3D image data.

[0200] After step S141, the control unit 50 extracts three or more points from the extraction range (step S142).

[0201] After step S142, the control unit 50 estimates a plane based on three or more points extracted from the extraction range. When the normal vector of the plane is defined as (a, b, c), the distance between the plane and the origin is defined as d, and the equation of the plane is defined as ax+by+cz+d=0, the control unit 50 uses SVD (singular value decomposition) to find the coefficients a, b, c, and d that minimize the sum of squares of the distances from the plane to three or more points. In this way, the control unit 50 estimates the plane (step S143). The plane approximates the surface of the object in the extraction range.

[0202] After step S143, the control unit 50 changes the position and orientation of the 3D view camera so that the normal to the plane faces the 3D view camera (step S144). When step S144 has been executed, the rotation processing shown in FIG. 23 ends.

[0203] Next, a third method will be described. The user operates the touch panel 52 or the like to input an instruction to set three reference points to the endoscope system 1. The control unit 50 accepts the instruction and sets the three reference points. The control unit 50 sets a plane passing through the three reference points as a reference plane.

[0204] Next, a fourth method will be described. After the first and second reference points are set, the user operates the touch panel 52 or the like to input an instruction to set a third reference point to the endoscope system 1. The control unit 50 accepts the instruction and sets the third reference point. The control unit 50 estimates a plane that approximates the area near the third reference point. This plane is a reference figure. The control unit 50 sets a reference plane that passes through the first and second reference points and is parallel to the normal vector of the estimated plane.

[0205] In the fourth embodiment, the endoscope system 1 can facilitate intuitive understanding of the 3D shape of the subject, similarly to the first embodiment.

[0206] (Fifth embodiment) A fifth embodiment of the present invention will be described. In the fifth embodiment, an endoscope system 1 shown in Fig. 1 is used. An endoscope system 1a shown in Fig. 11 or an endoscope system 1b shown in Fig. 12 may also be used.

[0207] In the fifth embodiment, the control unit 50 sets a reference plane based on one or more reference points and a predetermined direction. First to eighth methods for setting the reference plane will be described below.

[0208] First, the first method will be described. The control unit 50 sets a plane that passes through the first reference point and is perpendicular to the line of sight from the origin of the 3D view camera to the first reference point as the reference plane.

[0209] Next, the second method will be described. The control unit 50 sets a reference plane that passes through the first and second reference points and is parallel to the optical axis of the 3D view camera.

[0210] Next, a third method will be described. After the first and second reference points are set, the control unit 50 estimates a plane that approximates an area near at least one of the two reference points. This plane is a reference figure. The control unit 50 sets a plane that passes through the first and second reference points and is parallel to the normal vector of the estimated plane as a reference plane.

[0211] Next, a fourth method will be described. After the first and second reference points are set, the control unit 50 calculates a line that passes through the first and second reference points. The control unit 50 then sets a plane that passes through the first reference point and has a normal vector parallel to the line as a reference plane.

[0212] Next, a fifth method will be described. After the first reference point is set, the control unit 50 estimates a plane that approximates the area near the first reference point. This plane is the reference figure. The control unit 50 sets a plane that passes through the first reference point and has the same normal vector as the normal vector of the estimated plane as the reference plane.

[0213] Fig. 24 shows an example of the procedure for 3D shape display processing in the fifth method. The operation of the endoscope system 1 will be described using Fig. 24. Processing that is the same as the processing shown in Fig. 2 will not be described.

[0214] After step S103, the control unit 50 sets a sphere having a predetermined radius and centered on the first reference point as an extraction range (step S150). The extraction range includes three or more points included in the 3D image data.

[0215] After step S150, the control unit 50 extracts three or more points from the extraction range (step S151).

[0216] After step S151, the control unit 50 estimates a plane based on the three or more points extracted from the extraction range (step S152). The plane approximates the surface of the object in the extraction range.

[0217] After step S152, the control unit 50 sets a plane that passes through the first reference point and has the same normal vector as the normal vector of the estimated plane as a reference plane (step S153). After step S153, step S107 is executed. In the above method, the user needs to perform fewer operations compared to when two reference points are set.

[0218] The control unit 50 may set a second reference point in addition to the first reference point. The control unit 50 may execute the same processes as steps S150 to S153 based on the second reference point. The control unit 50 may calculate the average of the normal vectors of the two estimated planes. The control unit 50 may set, as the reference plane, a plane that passes through the first reference point or the second reference point and has the same normal vector as the calculated average normal vector.

[0219] Next, a sixth method will be described. The control unit 50 estimates a curved surface (cylindrical surface) by using an arbitrary 3D point cloud. When the direction vector of the central axis of the cylindrical surface is defined as (a, b, c) and the coordinates of the point through which the central axis of the cylindrical surface passes are defined as (x0, y0, z0), the equation of the cylindrical axis is expressed as (x-x0) / a = (y-y0) / b = (z-z0) / c. The control unit 50 calculates the parameters of the cylinder a, b, c, x0, y0, z0, r using RANSAC (Random Sample Consensus) so that the sum of squares of the difference between the distance d from the cylindrical axis to each point in the 3D point cloud and the radius r of the cylindrical surface is minimized. The cylindrical surface is the reference figure.

[0220] The control unit 50 sets a reference point according to an instruction from the user. Alternatively, the control unit 50 sets a point on the subject that is farthest or closest to the central axis of the cylinder as the reference point. The control unit 50 sets a plane that passes through the reference point and has the central axis of the cylinder as its normal line as the reference plane. In a sixth method, the endoscope system 1 can display a 3D shape of a cross section formed when the 3D shape of the subject is cut along a plane perpendicular to the central axis of the cylinder.

[0221] The user may input an instruction to set a reference point to the endoscope system 1 by operating the touch panel 52 or the like. The control unit 50 may display the central axis of the cylinder on the display 51, and may set a reference point on the central axis according to an instruction from the user.

[0222] The control unit 50 may set a point that is included in the 3D image data and closest to the central axis of the cylinder as the reference point. The control unit 50 may set a point that is included in the 3D image data and farthest from the central axis of the cylinder as the reference point. The control unit 50 may set a target range and may set a point that is included in the target range and closest to the central axis of the cylinder as the reference point. Alternatively, the control unit 50 may set a point that is included in the target range and farthest from the central axis of the cylinder as the reference point.

[0223] Fig. 25 shows an example of the procedure for 3D shape display processing in the sixth method. The operation of the endoscope system 1 will be described using Fig. 25. Processing that is the same as the processing shown in Fig. 2 will not be described.

[0224] After step S101, the control unit 50 estimates the cylindrical surface and calculates the central axis of the cylinder (step S160).

[0225] After step S160, the control unit 50 extracts the point closest to the central axis of the cylinder or the point farthest from the central axis of the cylinder as a reference point (step S161).

[0226] After step S161, the control unit 50 sets a plane that passes through the reference point and has the central axis of the cylinder as its normal line as a reference plane (step S162). After step S162, step S107 is executed.

[0227] Next, a seventh method will be described. The control unit 50 sets a plane that passes through the reference point and includes the central axis of the cylinder as the reference plane. In the seventh method, the endoscope system 1 can display the 3D shape of a cross section formed when the 3D shape of the subject is cut along the plane that includes the central axis of the cylinder.

[0228] Next, an eighth method will be described. The control unit 50 sets a plane that passes through the reference point and the reference line and is parallel to the optical axis of the 3D view camera as a reference plane. The reference line is a horizontal or vertical line in the 3D image displayed on the display 51.

[0229] In the fifth embodiment, the endoscope system 1 can facilitate intuitive understanding of the 3D shape of the subject, similarly to the first embodiment.

[0230] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Addition, omission, substitution, and other modifications of the configuration are possible within the scope of the spirit of the present invention. Furthermore, the present invention is not limited by the above description, but is limited only by the scope of the appended claims. [Explanation of symbols]

[0231] 1,1a,1b Endoscope System 2 Insertion section 2a Tip 3,3b Scope unit 4,7 Base Unit 5,6 Main unit 10, 10a, 10b Endoscopic device 11 External device 20 Imaging unit 21 Curved section 22 Lighting window 30 Imaging drive circuit 31 Image processing section 32 UD drive unit 33 RL drive unit 34 Bending control section 35 Light source 36 Light source control unit 40,50,60,70 Control section 41, 54, 55 Communications Department 42,56 Volatile memory 43,57 Non-volatile memory 51 Display 52 Touch Panel 53 Operation buttons

Claims

1. The control unit includes: acquiring three-dimensional image data including three-dimensional coordinates of three or more points on the subject calculated based on an endoscopic image of the subject; displaying a three-dimensional shape of the subject on a display based on the three-dimensional image data; setting a reference plane based on one or more points included in the three or more points; Dividing the three-dimensional image data into a first region and a second region using the reference plane as a boundary; In order to display a cross section of the subject set based on the reference plane, a display state of the three-dimensional shape of the subject in one of the first region and the second region is made different from a display state of the three-dimensional shape of the subject in the other of the first region and the second region. Three-dimensional image display device.

2. The control unit hides one of the first area and the second area. The three-dimensional image display device according to claim 1 .

3. The control unit sets the transparency of one of the first region and the second region to be higher than the transparency of the other of the first region and the second region. The three-dimensional image display device according to claim 1 .

4. The control unit acquiring area information indicating one of the first area and the second area; Change the display state of the area indicated by the area information The three-dimensional image display device according to claim 1 .

5. The control unit changes the display state of the cross section so that the cross section is emphasized. The three-dimensional image display device according to claim 1 .

6. The control unit Accepting an instruction to rotate the three-dimensional shape; The three-dimensional shape rotated in accordance with the instruction is displayed on the display. The three-dimensional image display device according to claim 1 .

7. The control unit Accept an instruction to enlarge or reduce the three-dimensional shape; The three-dimensional shape is displayed on the display as enlarged or reduced in accordance with the instruction. The three-dimensional image display device according to claim 1 .

8. The control unit displays, based on the three-dimensional image data, a three-dimensional shape of the entire or part of the subject adjacent to the three-dimensional shape of the subject including the first region and the second region having a display state different from a display state of the first region. The three-dimensional image display device according to claim 1 .

9. The control unit setting a reference point based on one or more points included in the three or more points; The reference plane is set based on the reference point. The three-dimensional image display device according to claim 1 .

10. The control unit sets two or more of the reference points. The three-dimensional image display device according to claim 9 .

11. The control unit establishing a reference figure based on the one or more points; The reference surface is set based on the reference figure. The three-dimensional image display device according to claim 1 .

12. The control unit Detecting a feature region in the endoscopic image; The reference plane is set based on one or more points included in the characteristic region among the three or more points. The three-dimensional image display device according to claim 1 .

13. The control unit setting a measurement standard based on one or more points included in the three or more points; Calculating the distance between the measurement reference and a point on the cross section The three-dimensional image display device according to claim 1 .

14. The control unit calculating a distance between the measurement reference and each of two or more points on the cross section; Identifying the point among the two or more points where the distance is maximum or minimum The three-dimensional image display device according to claim 13.

15. The control unit superimposes information indicating the point at which the distance is maximum or minimum on the three-dimensional shape. The three-dimensional image display device according to claim 14.

16. The control unit a range including a part of the cross section is set in the three-dimensional image data; Calculating the distance between the measurement standard and each of two or more points included in the range The three-dimensional image display device according to claim 14.

17. the endoscopic image includes a first image and a second image that are stereo images; The control unit generates the three-dimensional image data based on the first image and the second image. The three-dimensional image display device according to claim 1 .

18. the endoscopic image is generated based on an optical image of the subject formed by a monocular optical system; The control unit generates the three-dimensional image data based on two or more of the endoscopic images. The three-dimensional image display device according to claim 1 .

19. The control unit acquires the three-dimensional image data from a storage medium. The three-dimensional image display device according to claim 1 .

20. The control unit acquiring three-dimensional image data including three-dimensional coordinates of three or more points on the subject calculated based on an endoscopic image of the subject; displaying a three-dimensional shape of the subject on a display based on the three-dimensional image data; setting a reference plane based on one or more points included in the three or more points; Dividing the three-dimensional image data into a first region and a second region using the reference plane as a boundary; In order to display a cross section of the subject set based on the reference plane, a display state of the three-dimensional shape of the subject in one of the first region and the second region is made different from a display state of the three-dimensional shape of the subject in the other of the first region and the second region. 3D image display method.

21. acquiring three-dimensional image data including three-dimensional coordinates of three or more points on the subject calculated based on an endoscopic image of the subject; displaying a three-dimensional shape of the subject on a display based on the three-dimensional image data; setting a reference plane based on one or more points included in the three or more points; Dividing the three-dimensional image data into a first region and a second region using the reference plane as a boundary; a step of differentiating a display state of a three-dimensional shape of the subject in one of the first region and the second region from a display state of the three-dimensional shape of the subject in the other of the first region and the second region in order to display a cross section of the subject set based on the reference plane; A program that causes a computer to execute the following.

Images