3D shape information display device, 3D shape information display method, and program

Abstract

This invention provides a three-dimensional shape information display device, a three-dimensional shape information display method, and a program that can present detailed three-dimensional shape information of a subject. [Solution] The 3D shape information display device has a control unit. The control unit acquires 3D data including first 3D coordinates of three or more points on a subject, and displays a 3D image on the display based on the 3D data. The control unit receives position information output from a pointing device, and calculates second 3D coordinates of one or more points based on line-of-sight information at the position indicated by the position information and polygon information at the position. The control unit generates 3D shape information of the subject based on the second 3D coordinates of the one or more points, and displays the 3D shape information on the display.

Description

Technical Field

[0002] , , , , , , , ,

[0004] , , , , , , , ,

[0005] , ,

[0001] , ,

[0003] , , , ,

[0001] The present invention relates to a three-dimensional shape information display device, a three-dimensional shape information display method, and a program.

Background Art

[0002] Industrial endoscope devices are used for inspections (endoscopic inspections) of abnormalities and corrosion inside boilers, pipes, aircraft engines, heat exchangers, etc. The device disclosed in Patent Document 1 has an elongated probe including an insertion tube that can be inserted into an observation target, and generates an image based on an optical image acquired via the probe. The device determines the three-dimensional (3D) coordinates of points in the observation target by using an image of the observation target, and determines a reference plane by using the 3D coordinates of three or more points. The device calculates the distance between the reference plane and each point, and displays a color map of each point colored according to the distance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The device disclosed in Patent Document 1 calculates the 3D coordinates of a plurality of points, and determines a reference plane by using the 3D coordinates of three or more points selected as measurement points. The 3D coordinates used to determine the reference plane are limited to the pre-calculated 3D coordinates. Since the device cannot select an arbitrary position of the observation target as a measurement point, it cannot provide detailed information on the three-dimensional shape of the observation target.

[0005] An object of the present invention is to provide a three-dimensional shape information display device, a three-dimensional shape information display method, and a program that can present detailed three-dimensional shape information of a subject. [Means for solving the problem]

[0006] The present invention relates to a three-dimensional shape information display device having a control unit, the control unit acquires three-dimensional data including first three-dimensional coordinates of three or more points on the subject calculated based on an image of the subject acquired by an endoscope, displays a three-dimensional image on a display based on the three-dimensional data, the three-dimensional image is an image of the three-dimensional shape of the subject including the points having the first three-dimensional coordinates, receives position information output from a pointing device, the position information indicates a position on the three-dimensional image, calculates second three-dimensional coordinates of one or more points based on line-of-sight information at the position and polygon information at the position, the second three-dimensional coordinates are different from the first three-dimensional coordinates of each of the three or more points, the polygon information indicates a figure generated based on the three or more points, generates three-dimensional shape information of the subject based on the second three-dimensional coordinates of one or more points, and displays the three-dimensional shape information on the display.

[0007] In the three-dimensional shape information display device of the present invention, the control unit sets a reference plane based on the second three-dimensional coordinates of one or more points, divides the three-dimensional data into a first region and a second region with the reference plane as the boundary, and generates three-dimensional shape information showing the cross-section of the subject set based on the reference plane.

[0008] In the three-dimensional shape information display device of the present invention, the control unit generates three-dimensional shape information indicating the size of the subject based on the second three-dimensional coordinates of one or more points.

[0009] In the three-dimensional shape information display device of the present invention, the control unit calculates second three-dimensional coordinates of four or more points including one or more points, sets a reference plane based on the second three-dimensional coordinates of three or more points among the four or more points, and generates three-dimensional shape information indicating the distance between one of the four or more points and the reference plane.

[0010] In the three-dimensional shape information display device of the present invention, the control unit calculates the second three-dimensional coordinates of two points as one or more points, and generates three-dimensional shape information indicating the distance between the two points.

[0011] In the three-dimensional shape information display device of the present invention, the three-dimensional data includes the polygon information.

[0012] In the three-dimensional shape information display device of the present invention, after the position information is received, the control unit generates the polygon information.

[0013] In the three-dimensional shape information display device of the present invention, the line of sight information indicates a three-dimensional line including two or more points corresponding to the position, and the control unit calculates the second three-dimensional coordinates of the intersection point of the three-dimensional line and the figure indicated by the polygon information.

[0014] In the three-dimensional shape information display device of the present invention, the polygon information represents two or more figures, and the intersection point is the intersection point of the three-dimensional line and the figure closest to the camera position among the two or more figures.

[0015] In the three-dimensional shape information display device of the present invention, the control unit displays information indicating the three-dimensional line on the display.

[0016] In the three-dimensional shape information display device of the present invention, the line of sight information indicates the two-dimensional coordinates of the position on a plane corresponding to the screen of the display, and the control unit calculates the second three-dimensional coordinates based on the two-dimensional coordinates and the two-dimensional figure obtained by projecting the figure indicated by the polygon information onto the plane.

[0017] In the three-dimensional shape information display device of the present invention, the control unit generates the three-dimensional shape information based on the second three-dimensional coordinates when a first mode is set, and generates the three-dimensional shape information based on a third three-dimensional coordinate calculated based on the second three-dimensional coordinates when a second mode different from the first mode is set, wherein the third three-dimensional coordinates are different from the second three-dimensional coordinates.

[0018] In the three-dimensional shape information display device of the present invention, the third three-dimensional coordinate is the first three-dimensional coordinate of one of the three or more points, or the three-dimensional coordinate of a point included in the edge of the figure indicated by the polygon information.

[0019] In the three-dimensional shape information display device of the present invention, the control unit calculates the third three-dimensional coordinates based on the texture information of the figure at the position, and the texture information is associated with the polygon information.

[0020] In the three-dimensional shape information display device of the present invention, the control unit generates three-dimensional shape information based on the second three-dimensional coordinates when a first mode is set, and when a second mode different from the first mode is set, it calculates a second position on the three-dimensional image based on the texture information of the figure at the position, the texture information is associated with the polygon information, the second position is different from the first position, and a third three-dimensional coordinate is calculated based on the line of sight information at the second position and the polygon information at the second position, the third three-dimensional coordinate is different from the first three-dimensional coordinate of each of the three or more points, and the three-dimensional shape information is generated based on the third three-dimensional coordinate.

[0021] In the three-dimensional shape information display device of the present invention, the control unit detects an object in the subject, receives first position information and second position information as position information, the first position information indicates a first position, the second position information indicates a second position different from the first position, and when the object including the first position and the object including the second position are the same, the control unit generates the three-dimensional shape information based on the second three-dimensional coordinates corresponding to the first position and the second three-dimensional coordinates corresponding to the second position.

[0022] In the three-dimensional shape information display device of the present invention, the control unit detects the object based on the texture information associated with the polygon information.

[0023] The present invention relates to a method for displaying 3D shape information, in which a control unit acquires 3D data including first 3D coordinates of three or more points on a subject calculated based on an image of the subject acquired by an endoscope, displays a 3D image on a display based on the 3D data, the 3D image is an image of the 3D shape of the subject including the points having the first 3D coordinates, receives position information output from a pointing device, the position information indicates a position on the 3D image, calculates second 3D coordinates of one or more points based on line-of-sight information at the position and polygon information at the position, the second 3D coordinates are different from the first 3D coordinates of each of the three or more points, the polygon information indicates a figure generated based on the three or more points, generates 3D shape information of the subject based on the second 3D coordinates of one or more points, and displays the 3D shape information on the display.

[0024] A step of obtaining three-dimensional data including first three-dimensional coordinates of three or more points on a subject calculated based on an image of the subject acquired by an endoscope; a step of displaying a three-dimensional image on a display based on the three-dimensional data, wherein the three-dimensional image is an image of the three-dimensional shape of the subject including points having the first three-dimensional coordinates; a step of receiving position information output from a pointing device, wherein the position information indicates a position on the three-dimensional image; a step of calculating second three-dimensional coordinates of one or more points based on line-of-sight information and polygon information at the position, wherein the second three-dimensional coordinates are different from the first three-dimensional coordinates of each of the three or more points, and the polygon information indicates a figure generated based on the three or more points; a step of generating three-dimensional shape information of the subject based on the second three-dimensional coordinates of the one or more points; and a step of displaying the three-dimensional shape information on the display. It is a program for causing a computer to execute these steps.

Advantages of the Invention

[0025] According to the present invention, a three-dimensional shape information display device, a three-dimensional shape information display method, and a program can present detailed three-dimensional shape information of a subject.

Brief Description of the Drawings

[0026] [Figure 1] It is a block diagram showing an example of the configuration of an endoscope system according to the first embodiment of the present invention. [Figure 2] It is a flowchart showing an example of the procedure of information display processing in the first embodiment of the present invention. [Figure 3] It is a flowchart showing an example of the procedure of information display processing in the first modification of the first embodiment of the present invention. [Figure 4] It is a diagram showing an example of an image displayed on a display included in an endoscope system according to the first modification of the first embodiment of the present invention. [Figure 5]This figure shows an example of an image displayed on the display of an endoscope system according to a first modification of the first embodiment of the present invention. [Figure 6] This figure shows an example of an image displayed on the display of an endoscope system according to a first modification of the first embodiment of the present invention. [Figure 7] This figure shows the 3D space defined in the 3D data in a first modified example of the first embodiment of the present invention. [Figure 8] This figure shows an example of an image displayed on the display of an endoscope system according to a first modification of the first embodiment of the present invention. [Figure 9] This flowchart shows an example of the procedure for information display processing in a second modified example of the first embodiment of the present invention. [Figure 10] This flowchart shows an example of the procedure for setting a reference plane in a second modified example of the first embodiment of the present invention. [Figure 11] This figure shows an example of an image displayed on the display of an endoscope system according to a second modification of the first embodiment of the invention. [Figure 12] This flowchart shows an example of a procedure for measuring the size of a subject in a second modified example of the first embodiment of the present invention. [Figure 13] This figure shows an example of an image displayed on the display of an endoscope system according to a second modification of the first embodiment of the invention. [Figure 14] This figure shows an example of an abnormal part occurring in a subject in a second modified example of the first embodiment of the present invention. [Figure 15] This flowchart shows an example of the procedure for information display processing in a third modified example of the first embodiment of the present invention. [Figure 16] This figure shows an example of the relationship between a polygon and pixels in a 3D image in a third modified example of the first embodiment of the present invention. [Figure 17] This flowchart shows an example of the information display processing procedure in a fourth modified example of the first embodiment of the present invention. [Figure 18] This block diagram shows an example of the configuration of an endoscope system according to a fifth modification of the first embodiment of the present invention. [Figure 19] This block diagram shows an example of the configuration of an endoscope system according to a sixth modification of the first embodiment of the present invention. [Figure 20] This figure shows examples of images displayed on the display of an endoscope system according to various modifications of the first embodiment of the present invention. [Figure 21] This figure shows a method for point adsorption according to a second embodiment of the present invention. [Figure 22] This flowchart shows an example of the procedure for information display processing in a second embodiment of the present invention. [Figure 23] This flowchart shows an example of the procedure for information display processing in a second embodiment of the present invention. [Figure 24] This figure shows an example of texture information in a second embodiment of the present invention. [Figure 25] This figure shows an example of the process for detecting an object in a third embodiment of the present invention. [Figure 26] This flowchart shows an example of the procedure for information display processing in the third embodiment of the present invention. [Figure 27] This flowchart shows an example of the procedure for information display processing in the third embodiment of the present invention. [Modes for carrying out the invention]

[0027] Embodiments of the present invention will be described below with reference to the drawings. In the following, an endoscope system will be described as an example of a three-dimensional (3D) shape information display device.

[0028] (First Embodiment) Figure 1 shows an example of the configuration of an endoscope system 1 according to a first embodiment of the present invention. The endoscope system 1 shown in Figure 1 has an insertion unit 2, a scope unit 3, a base unit 4, and a main unit 5. The insertion unit 2, the scope unit 3, and the base unit 4 constitute an endoscope device 10. The main unit 5 is an operating device.

[0029] The insertion unit 2 is inserted into the body of the subject being observed. The subject is an industrial product. The insertion unit 2 is a long, slender tube and is bendable. The user performs the insertion procedure and inserts the insertion unit 2 into the subject. An optical adapter is attached to the tip of the insertion unit 2. The insertion unit 2 acquires an optical image of the inside of the subject. The insertion unit 2 has an imaging unit 20, a bending unit 21, and an illumination window 22.

[0030] The imaging unit 20 is located at the tip portion 2a, which includes the tip of the insertion portion 2. The imaging unit 20 is an image sensor such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor. The imaging unit 20 generates an image based on the optical image acquired by the insertion portion 2. The image generated by the imaging unit 20 is output to the scope unit 3.

[0031] The curved portion 21 curves the insertion portion 2 upward (U), downward (D), left (L), or right (R). Alternatively, the curved portion 21 curves the insertion portion 2 upward left (UL), upward right (UR), downward left (DL), or downward right (DR).

[0032] Illumination light is generated by the light source 35 of the scope unit 3 and output to the tip 2a through a light guide (not shown) located inside the insertion section 2. The illumination light is shone into the inside of the subject through the illumination window 22.

[0033] The scope unit 3 includes an imaging drive circuit 30, an image processing unit 31, an UD drive unit 32, an RL drive unit 33, a curvature control unit 34, a light source 35, and a light source control unit 36. The base unit 4 includes a control unit 40, a communication unit 41, a volatile memory 42, and a non-volatile memory 43.

[0034] The imaging drive circuit 30 controls the imaging unit 20 and outputs the image output from the imaging unit 20 to the image processing unit 31. The image processing unit 31 performs image processing such as noise reduction on the image output from the imaging unit 20 and outputs the image to the control unit 40.

[0035] The UD drive unit 32 is connected to a UD bending wire for bending the curved section 21 in the U or D direction. The UD drive unit 32 has a motor and bends the curved section 21 in the U or D direction by pulling the UD bending wire. The RL drive unit 33 is connected to an RL bending wire for bending the curved section 21 in the R or L direction. The RL drive unit 33 has a motor and bends the curved section 21 in the R or L direction by pulling the RL bending wire. The bending control unit 34 controls the UD drive unit 32 and the RL drive unit 33.

[0036] The UD drive unit 32 and the RL drive unit 33 can operate simultaneously. For example, the UD drive unit 32 and the RL drive unit 33 can bend the curved section 21 in the UL direction.

[0037] The light source 35 is an LED (Light-Emitting Diode) or the like, and generates illumination light. The illumination light is output from the light source 35 to a light guide (not shown). The light source control unit 36 ​​controls the light source 35.

[0038] The control unit 40 controls the parts of the scope unit 3 and the base unit 4. At least one of the control unit 40, the image processing unit 31, the curvature control unit 34, and the light source control unit 36 ​​may consist of at least one processor and logic circuit. For example, the processor is at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). At least one of the control unit 40, the image processing unit 31, the curvature control unit 34, and the light source control unit 36 ​​may include one or more processors. At least one of the control unit 40, the image processing unit 31, the curvature control unit 34, and the light source control unit 36 ​​may include one or more logic circuits.

[0039] The computer of the endoscope system 1 may load a program and execute the loaded program. The program includes instructions that define the operation of at least one of the control unit 40, image processing unit 31, bending control unit 34, and light source control unit 36. In other words, at least one function of the control unit 40, image processing unit 31, bending control unit 34, and light source control unit 36 ​​may be implemented by software.

[0040] The above program may be provided on a "computer-readable recording medium," such as flash memory. The program may be transmitted from the computer holding the program to the endoscope system 1 via a transmission medium or by transmission waves within the transmission medium. The "transmission medium" for transmitting the program is a medium that has the function of transmitting information. A medium that has the function of transmitting information includes networks such as the Internet and communication lines such as telephone lines. The above program may implement some of the functions described above. Furthermore, the above program may be a differential file (differential program). The functions described above may be implemented by a combination of a program already recorded on the computer and a differential program.

[0041] The communication unit 41 has a communication circuit and performs wired or wireless communication with the main unit 5 for purposes such as curve control. The volatile memory 42 is a RAM (Random Access Memory) or DRAM (Dynamic RAM), etc. The volatile memory 42 stores various information processed by the control unit 40. The non-volatile memory 43 is an SRAM (Static RAM), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable ROM), or flash memory. The non-volatile memory 43 may be detachable from the base unit 4. The non-volatile memory 43 stores images generated by the imaging unit 20 and various information processed by the control unit 40.

[0042] The main unit 5 includes a control unit 50, a display 51, a touch panel 52, operation buttons 53, a communication unit 54, a communication unit 55, a volatile memory 56, and a non-volatile memory 57. The main unit 5 may also be an information terminal such as a smartphone or tablet.

[0043] The control unit 50 controls each part of the main unit 5. The control unit 50 may consist of at least one processor and logic circuit. The control unit 50 may include one or more processors. The control unit 50 may include one or more logic circuits. The computer of the endoscope system 1 may read a program and execute the read program. The program includes instructions that define the operation of the control unit 50. In other words, the functions of the control unit 50 may be implemented by software. The program that implements the functions of the control unit 50 may be implemented in the same way as the program that implements the functions of the control unit 40 and other units.

[0044] The display 51 is a monitor such as an LCD (Liquid Crystal Display). The display 51 displays images generated by the imaging unit 20. The touch panel 52 accepts input for information necessary to control the endoscope system 1. The touch panel 52 is located on the screen of the display 51. By operating the touch panel 52, the user can input instructions to change the settings of the endoscope system 1 and instructions necessary to operate the endoscope system 1 into the endoscope system 1.

[0045] The operation button 53 receives various instructions from the user. By pressing the operation button 53, the user can input instructions regarding power or illumination to the endoscope system 1. The communication unit 54 performs wired or wireless communication with the base unit 4 for purposes such as bending control. The communication unit 55 performs wired or wireless communication with the external device 11. The external device 11 may be a remote control, keyboard, or mouse.

[0046] The control unit 50 performs information display processing to display the 3D shape information of the subject. The 3D shape information relates to the characteristics of the subject's 3D shape. For example, the 3D shape information indicates the size or characteristic shape of the subject. An overview of the information display processing is described below.

[0047] The control unit 50 acquires 3D data including first 3D coordinates of three or more points on the subject calculated based on the image of the subject acquired by the endoscope device 10. The three or more points constitute a point cloud. The first 3D coordinates are defined in a 3D space corresponding to real space. Based on the 3D data, the control unit 50 displays an image of the subject's 3D shape (3D image) on the display 51. The user operates the touch panel 52 or the like to specify a position on the 3D image. In the following example, the touch panel 52 is used as a pointing device. Operation buttons 53 or an external device 11 may also be used as a pointing device. The main unit 5 may have a joystick or the like as a pointing device.

[0048] When the user specifies a position on the 3D image, the touch panel 52 outputs position information indicating that position. The control unit 50 receives this position information and calculates a second 3D coordinate for the point corresponding to the position indicated by the position information. The second 3D coordinate is different from the first 3D coordinate for each of the three or more points included in the 3D data. Based on the second 3D coordinate, the control unit 50 generates 3D shape information of the subject and displays the 3D shape information on the display 51.

[0049] Figure 2 shows an example of the information display processing procedure. The operation of the endoscope system 1 will be explained using Figure 2. In the first embodiment, the endoscope system 1 calculates the 3D coordinates of an arbitrary position on the subject by using the raycasting method.

[0050] The control unit 50 acquires 3D data (step S100).

[0051] The control unit 50 performs the following processing in step S100. The first to third examples are described below.

[0052] First, let's describe the first example. In the first example, the optical adapter is a stereo optical adapter with two fields of view. The optical adapter has a first optical system and a second optical system corresponding to the two fields of view. The first and second optical systems form two optical images of the subject on the imaging unit 20. The imaging unit 20 generates a stereo image corresponding to the first and second optical images. The stereo image includes a pair of two images (the first image and the second image). That is, the stereo image includes an image of the subject as seen from a first viewpoint and an image of the subject as seen from a second viewpoint. The control unit 50 calculates the 3D coordinates of three or more points on the subject using one or more stereo images generated by the imaging unit 20 and generates 3D data including those 3D coordinates.

[0053] Next, a second example will be described. In the second example, the optical adapter is a monocular optical adapter with a single field of view. In the first example, the optical adapter forms two optical images of the subject, but in the second example, the optical adapter forms one optical image of the subject. The imaging unit 20 generates an image corresponding to the optical image formed by the optical adapter. The imaging unit 20 performs imaging from two or more different viewpoints and generates two or more images. The control unit 50 uses the two or more images generated by the imaging unit 20 to calculate the 3D coordinates of three or more points on the subject and generates 3D data including those 3D coordinates.

[0054] Next, a third example will be described. The 3D data generated in the first or second example is pre-stored in the non-volatile memory 57. The control unit 50 retrieves the 3D data from the non-volatile memory 57. If two or more sets of 3D data are generated in the first or second example, the control unit 50 may perform alignment between the two or more sets of 3D data, or it may convert the coordinate systems of the two or more sets of 3D data to a common coordinate system. This allows the control unit 50 to use the two or more sets of 3D data as one large 3D data.

[0055] As described above, 3D data includes the 3D coordinates of three or more points on the subject. The 3D data also includes mesh information. A mesh is generated by combining two or more polygons. A polygon represents a basic shape composed of points, lines, or faces in 3D space. If a polygon is a triangle and the mesh information includes information on two or more polygons, the 3D data includes the 3D coordinates of four or more points.

[0056] After step S100, the control unit 50 acquires texture information for each polygon (step S101).

[0057] Texture information is information from a texture image that shows the portion of the subject in each polygon. The texture image is generated by the imaging unit 20 and is part of the image used to generate the 3D data.

[0058] After step S101, the control unit 50 generates a 3D image by combining the texture image indicated by the texture information with the 3D data (step S102). After step S102, the control unit 50 outputs the 3D image to the display 51 and displays the 3D image on the display 51 (step S103).

[0059] After the 3D image is displayed, the user inputs information indicating the position on the 3D image to the endoscope system 1 by operating the touch panel 52. The touch panel 52 outputs the position information to the control unit 50. The control unit 50 receives the position information output from the touch panel 52 (step S104).

[0060] After step S104, the control unit 50 performs a projection transformation of the point cloud coordinates. That is, the control unit 50 transforms all 3D coordinates of three or more points included in the 3D data into 2D coordinates (viewport coordinates) on a plane (viewport) corresponding to the screen of the display 51 (step S105). Step S105 may be performed before step S104.

[0061] Step S105 is explained in detail. A matrix X is defined as a matrix that stores the 3D coordinates of N points. Matrix X has 3 rows and N columns. The view matrix in the current 3D display is defined as Tview, the projection matrix as Tproj, and the viewport transformation matrix as Tviewport. The 2D coordinates (homogeneous coordinates) projected onto the viewport are represented by the x and y components in the upper two rows of equation (1) below.

[0062]

number

[0063] In equation (1), homogeneous(X) represents the homogeneous coordinate matrix of X, and hnormalized(X) in equation (1) represents the normal coordinate matrix that corresponds to the homogeneous coordinate matrix X.

[0064] The screen size of display 51 is defined as (W,H), and the depth range is defined as Dmin to Dmax. Here, the viewport transformation matrix Tviewport is expressed by the following equation (2).

[0065]

number

[0066] After step S105, the control unit 50 calculates the 3D coordinates of two points on a 3D line corresponding to the position indicated by the position information. This 3D line corresponds to the camera position and the line of sight passing through that position (step S106). The camera position indicates the position of the camera that acquired the image used to generate the 3D data. For example, this camera corresponds to the endoscope device 10.

[0067] Step S106 will now be explained in detail. When the position S indicated by the position information is defined as S=(Sx,Sy), the control unit 50 calculates the 3D coordinates of two points La and Lb on the 3D line corresponding to position S. The 3D coordinates of these two points La and Lb are expressed by the following equation (3).

[0068]

number

[0069] The matrix T in equation (3) is expressed by the following equation (4).

[0070]

number

[0071] After step S106, the control unit 50 calculates the 3D coordinates corresponding to the position S. These 3D coordinates are different from the 3D coordinates of each of the three or more points included in the 3D data (step S107).

[0072] Step S107 will now be explained in detail. The control unit 50 identifies one or more polygons from all the polygons that intersect with the 3D line calculated in step S106. The control unit 50 identifies the polygon closest to the camera position from among those one or more polygons and calculates the 3D coordinates of the intersection point between that polygon and the 3D line.

[0073] The following describes an example of step S107 when the polygon is a triangle. The control unit 50 determines whether position S is inside each triangle. The control unit 50 performs this determination for all triangles. Specifically, the control unit 50 performs the process described below.

[0074] The 3D coordinates of the three vertices of the triangle are defined as P0, P1, and P2, respectively. The 2D coordinates of the three vertices projected onto the plane corresponding to the screen of the display 51 are defined as Q0, Q1, and Q2. The control unit 50 converts the 2D coordinates of the three vertices into 3D vectors with a Z component of 0 according to equation (5) below, and calculates the cross product of each 3D vector.

[0075]

number

[0076] The control unit 50 determines whether position S lies on a triangle with vertices Q0, Q1, and Q2. Specifically, the control unit 50 determines that position S lies on a triangle when the signs of the Z components of the three cross products calculated in equation (5) are equal.

[0077] In the special condition where the three vertices of a triangle lie on the same straight line, the control unit 50 may determine that position S is on the triangle even if position S is not on the triangle. Therefore, the control unit 50 performs the determination described below.

[0078] When the control unit 50 determines that position S lies on the triangle, it determines whether a 3D line containing two points La and Lb coincides with the triangle. Specifically, the control unit 50 determines that the 3D line coincides with the triangle when the following equation (6) is defined and the conditions shown in the following equation (7) are met. The control unit 50 calculates the 3D coordinates of the intersection point of the 3D line and the triangle as the 3D coordinates corresponding to position S. These 3D coordinates are expressed as La + tLab.

[0079]

number

[0080]

number

[0081] After step S107, the control unit 50 generates 3D shape information of the subject using the 3D coordinates calculated in step S107 (step S108). Specific examples of how the control unit 50 generates 3D shape information will be described in the first and second modifications of the first embodiment, which will be described later.

[0082] After step S108, the control unit 50 displays the 3D shape information on the display 51 (step S109). When step S109 is executed, the information display process shown in Figure 2 is completed.

[0083] Each embodiment of the present invention includes a control unit 50. The control unit 50 acquires 3D data including first 3D coordinates of three or more points on a subject, calculated based on an image of the subject acquired by an endoscope device 10 (endoscope). The control unit 50 displays a 3D image on a display 51 based on the 3D data. The 3D image is an image of the 3D shape of the subject, including points having first 3D coordinates. The control unit 50 receives position information output from a pointing device (touch panel 52, etc.). The position information indicates a position on the 3D image. The control unit 50 calculates second 3D coordinates of one or more points based on line-of-sight information (3D line) at the position indicated by the position information and polygon information at that position. The second 3D coordinates are different from the first 3D coordinates of each of the three or more points. The polygon information indicates a figure generated based on those three or more points. The control unit 50 generates 3D shape information of the subject based on the second 3D coordinates of one or more points and displays the 3D shape information on the display 51.

[0084] Each aspect of the present invention provides a 3D shape information display method comprising the first to sixth steps. In the first step (step S100), the control unit 50 acquires 3D data including first 3D coordinates of three or more points on a subject calculated based on an image of the subject acquired by the endoscope device 10 (endoscope). In the second step (step S103), the control unit 50 displays a 3D image on the display 51 based on the 3D data. In the third step (step S104), the control unit 50 receives position information output from a pointing device (touch panel 52, etc.). In the fourth step (step S107), the control unit 50 calculates second 3D coordinates of one or more points based on line-of-sight information (3D line) at the position indicated by the position information and polygon information at that position. In the fifth step (step S108), the control unit 50 generates 3D shape information of the subject based on the second 3D coordinates of one or more points, and in the sixth step (step S109), displays the 3D shape information on the display 51.

[0085] A program according to each aspect of the present invention causes a computer to execute the first to sixth steps described above.

[0086] Each aspect of the present invention may include the following modifications. The 3D data includes polygon information.

[0087] Each aspect of the present invention may include the following modifications. The line of sight information indicates a 3D line that includes two or more points corresponding to the position indicated by the position information. The control unit 50 calculates the second 3D coordinates of the intersection point of the 3D line and the figure indicated by the polygon information.

[0088] Each aspect of the present invention may include the following modifications. The polygon information represents two or more figures. The intersection point is the intersection point between a 3D line and the figure closest to the camera position among the two or more figures.

[0089] In the first embodiment, the control unit 50 calculates a second 3D coordinate based on line-of-sight information at the position indicated by the position information and polygon information at that position. The control unit 50 also generates 3D shape information of the subject based on the second 3D coordinate and displays the 3D shape information on the display 51. The control unit 50 can generate 3D shape information without being limited by the first 3D coordinate included in the 3D data. Therefore, the endoscope system 1 can present detailed 3D shape information of the subject. Furthermore, in the first embodiment, even when the 3D data is made lighter and the computational load reduced by reducing the number of vertices and polygons in order to handle a wide range of 3D data, detailed 3D shape information of the subject can still be presented.

[0090] (First modification of the first embodiment) A first modification of the first embodiment of the present invention will now be described. In the first modification of the first embodiment, the control unit 50 generates 3D shape information showing the characteristic shape of the subject. Specifically, the control unit 50 generates 3D shape information showing the cross-section of the subject.

[0091] After the 3D image is displayed on the display 51, the control unit 50 accepts the designation of one or more reference points on the subject. In the following example, the control unit 50 accepts the designation of two reference points.

[0092] The control unit 50 sets a reference plane (cutting reference) in 3D space based on one or more reference points. Specifically, the control unit 50 sets a reference shape based on one or more reference points and sets a reference plane based on the reference shape. The reference shape can be a line, a plane, or a curved surface. The following describes an example in which a boundary line is used as the reference shape.

[0093] The control unit 50 divides the entire area of ​​the subject into two regions using a reference plane. The cross-section of the subject is formed by the point where the reference plane and the subject intersect. The control unit 50 changes the display state of one of the two regions. In the first embodiment, the control unit 50 hides one of the two regions. As a result, the cross-section of the subject becomes visible.

[0094] Figure 3 shows an example of the information display processing procedure. Figure 3 will be used to explain the operation of the endoscope system 1. Processes identical to those shown in Figure 2 will not be explained.

[0095] In step S103, the control unit 50 displays a 3D image on the display 51. Figure 4 shows an example of a 3D image displayed on the display 51 in step S103. The control unit 50 displays the 3D image IMG10 shown in Figure 4 on the display 51. The 3D image IMG10 includes the 3D shape SH10 of the subject. The control unit 50 displays an icon IC10 on the 3D image IMG10. The icon IC10 includes an illustration of the subject and prompts the user to set a first reference point. The icon IC10 also notifies the user of areas that will be hidden. For example, the icon IC10 notifies the user that the area in front of the reference plane passing through two reference points will be hidden.

[0096] The control unit 50 displays the cursor CS10 on the 3D image IMG10. The user moves the cursor CS10 on the 3D image IMG10 by operating the touch panel 52. The user may also move the cursor CS10 on the 3D image IMG10 by operating the operation buttons 53 or an external device 11, etc.

[0097] The user operates the touch panel 52 or the like to input an instruction to set a first reference point to the endoscope system 1. The control unit 50 determines whether or not that instruction has been input (step S110). If the control unit 50 determines in step S110 that the instruction has not been input, the control unit 50 repeats step S110.

[0098] When the control unit 50 determines in step S110 that the instruction has been input, the control unit 50 accepts the instruction and sets a first reference point at the position of the cursor CS10. The control unit 50 also executes steps S105 to S107 shown in Figure 2 to calculate the 3D coordinates of the first reference point (step S111). The information of the first reference point is stored in the volatile memory 56.

[0099] Figure 5 shows an example of a 3D image displayed on the display 51 when the first reference point is set in step S111. Parts that are the same as those shown in Figure 4 will not be explained.

[0100] The control unit 50 displays icon IC11 on the 3D image IMG10. Icon IC11 includes an illustration of the subject and prompts the user to set a second reference point. Icon IC11 also notifies the user of areas that will be hidden.

[0101] The user operates the touch panel 52, etc., to move the cursor CS10 on the 3D image IMG10. The user operates the touch panel 52, etc., to input an instruction to set a second reference point to the endoscope system 1. The control unit 50 determines whether or not the instruction has been input (step S112).

[0102] If the control unit 50 determines in step S112 that no such instruction has been entered, the control unit 50 highlights the line connecting the first reference point and the point on the cursor on the 3D image (step S117). After step S117, step S112 is executed.

[0103] Figure 6 shows an example of a 3D image displayed on the display 51 in step S117. Parts that are the same as those shown in Figure 5 will not be explained.

[0104] The control unit 50 displays line L10 on the 3D image IMG10. Line L10 is a straight line connecting the first reference point set in step S111 and the point indicated by the cursor CS10. Line L10 is displayed as a thick line. When the user moves the cursor CS10, line L10 moves.

[0105] When the control unit 50 determines in step S112 that an instruction to set a second reference point has been input, the control unit 50 accepts the instruction and sets the second reference point at the position of the cursor CS10. The control unit 50 also executes steps S106 and S107 shown in Figure 2 to calculate the 3D coordinates of the second reference point (step S113). The second reference point is different from the first reference point. Information about the second reference point is stored in the volatile memory 56.

[0106] After step S113, the control unit 50 sets a reference plane that passes through the first reference point and the second reference point (step S114).

[0107] Step S114 will now be explained in detail. Figure 7 shows the 3D space defined in the 3D data. The 3D data includes the 3D coordinates of three or more points that constitute the 3D shape SH11 of the subject. The 3D coordinates of these three or more points consist of coordinate values ​​for the mutually orthogonal X, Y, and Z axes.

[0108] In step S111, the control unit 50 sets a first reference point RP10, and in step S113, it sets a second reference point RP11. In step S114, the control unit 50 sets a boundary line BL10 that passes through the first reference point RP10 and the second reference point RP11. Also in step S114, the control unit 50 sets a plane PL10 that includes the boundary line BL10 and is parallel to the Z-axis. Plane PL10 is the reference plane.

[0109] After step S114, the control unit 50 divides the entire area of ​​the 3D data into a first area and a second area (step S115). Dividing the entire area into two areas means defining two areas within the entire area. In other words, dividing the entire area into two areas means assigning each point contained in the entire area to one of the two areas. The boundary between the first area and the second area is the reference plane set in step S114. Each of the three or more points contained in the 3D data is contained in either the first area or the second area.

[0110] After step S115, the control unit 50 changes the display state of the 3D image displayed on the display 51. Specifically, the control unit 50 changes the display state of either the first region or the second region. For example, the control unit 50 hides either the first region or the second region. As a result, the control unit 50 displays 3D shape information showing the cross-section of the subject (step S116). When step S116 is executed, the information display process shown in Figure 3 is completed.

[0111] The control unit 50 may increase the transparency of one of the first and second regions instead of hiding one of the first and second regions. For example, before step S116 is performed, the transparency of the first and second regions is 0%. In step S116, the control unit 50 sets the transparency of one of the first and second regions to a value greater than 0% and less than or equal to 100%. The control unit 50 may also set the transparency of one of the first and second regions to 50% or more.

[0112] When the first or second region is hidden, the cross-section previously hidden by the first or second region is displayed. When the transparency of either the first or second region is increased, the cross-section previously hidden by the first or second region becomes visible. Therefore, users can easily understand the 3D shape in the cross-section. In addition to the 3D shape of the cross-section, the 3D shape of the surrounding area is also displayed, making it easier for users to intuitively understand the 3D shape of the subject.

[0113] Figure 8 shows an example of a 3D image displayed on the display 51 in step S116. Parts that are the same as those shown in Figure 6 will not be explained.

[0114] For example, the first region is located on the positive Y-axis side of the reference plane set in step S114. For example, the second region is located on the negative Y-axis side of the reference plane. The control unit 50 hides one or more points included in the second region. As a result, the cross-section of the 3D shape SH10 is made visible. The control unit 50 also displays a line L11 on the 3D image IMG10 to highlight the cross-section. For example, the line L11 is displayed with a specific thickness and a specific color.

[0115] The control unit 50 may set a reference plane based on three reference points. For example, the control unit 50 sets a plane passing through a first reference point, a second reference point, and the origin of the 3D data as the reference plane. The origin of the 3D data corresponds to the camera position when the imaging unit 20 generates the image. The origin of the 3D data may coincide with the center of the tip of the insertion unit 2. In this method, the reference plane corresponds to a straight line on the distortion-corrected image generated by correcting the optical distortion of the image generated by the imaging unit 20. Therefore, the user can easily understand the position through which the reference plane passes on the distortion-corrected image.

[0116] The control unit 50 may set a reference plane based on one reference point. For example, the control unit 50 sets a plane as the reference plane that passes through a first reference point and is perpendicular to the line of sight from the origin of the 3D view camera to the first reference point. The origin of the 3D view camera corresponds to the viewpoint of the perspective projection transformation when a 3D image is generated from 3D data and that 3D image is displayed on the display 51.

[0117] Each aspect of the present invention may include the following modifications. The control unit 50 sets a reference plane based on the second 3D coordinates of one or more points and divides the 3D data into a first region and a second region with the reference plane as the boundary. The control unit 50 generates 3D shape information showing the cross-section of the subject set based on the reference plane.

[0118] In the first modification of the first embodiment, the control unit 50 generates 3D shape information showing a cross-section of the subject. The endoscope system 1 can display the detailed 3D shape of the cross-section of the subject.

[0119] (Second modification of the first embodiment) A second modification of the first embodiment of the present invention will now be described. In the second modification of the first embodiment, the control unit 50 generates 3D shape information indicating the size of the subject.

[0120] A monocular optical adapter, used for normal observation, is attached to the tip of the insertion unit 2. The imaging unit 20 generates two or more images based on the optical image formed through the monocular optical adapter. The monocular optical adapter and the imaging unit 20 constitute a monocular camera with a single field of view.

[0121] At least a portion of the surface of the object is curved. For example, the object is a pipe with a cylindrical surface. The control unit 50 measures the size of the object by using 3D data in the information display processing.

[0122] Figure 9 shows an example of the information display processing procedure. Figure 9 will be used to explain the operation of the endoscope system 1. Processes identical to those shown in Figure 2 will not be explained.

[0123] In step S100, the control unit 50 performs a 3D reconstruction process using two or more images to generate 3D data. For example, the control unit 50 generates 3D data using the method disclosed in Japanese Patent Application Publication No. 2020-12635.

[0124] The 3D data includes the 3D coordinates of three or more points (3D point cloud) of the subject, camera coordinates, and pose information. The 3D data may also include a mesh, which is a face with the 3D point cloud as its vertices, and mesh polygon data, which is a set of texture information associated with the mesh.

[0125] 3D coordinates are defined in a 3D space corresponding to real space. Camera coordinates indicate the 3D coordinates of the camera that acquired each of two or more images, and are associated with each of those two or more images. Camera coordinates are the 3D coordinates of the viewpoint at the time each image was acquired and indicate the camera's position. For example, camera coordinates indicate the 3D coordinates of the observation optical system that the camera possesses. Pose information indicates the pose of the camera that acquired each of two or more images, and is associated with each of those two or more images. For example, pose information indicates the pose of the observation optical system that the camera possesses.

[0126] 3D data represents relative shapes that do not have a length dimension. While 3D data includes the 3D coordinates of each point, the relative distances calculated using these 3D coordinates differ from the absolute size of the subject.

[0127] The 3D data generated in step S100 is stored in the non-volatile memory 57. The control unit 50 may retrieve the 3D data generated in previously performed inspections from the non-volatile memory 57.

[0128] After step S103, the control unit 50 sets the reference plane in 3D space based on three or more reference points (step S120).

[0129] Figure 10 shows an example of the procedure for the process performed in step S120. Figure 10 will be used to explain the operation of the endoscope system 1.

[0130] The control unit 50 sets the variable n, which manages the number of reference points, to 1 (step S200).

[0131] The user operates the touch panel 52 to set a reference point in the 3D image displayed on the display 51 and inputs the reference point to the endoscope system 1. For example, the user touches the position of the reference point. The control unit 50 receives the reference point input by the user and sets the reference point on the 3D image. The control unit 50 also executes steps S105 to S107 shown in Figure 2 to calculate the 3D coordinates of the reference point (step S201). If step S105 has already been executed, the execution of step S105 may be omitted. The reference point information is stored in the volatile memory 56. The reference point set in step S201 is treated as the nth reference point.

[0132] After step S201, the control unit 50 selects a region near the nth reference point in the 3D image displayed on the display 51 (step S202). For example, in step S202, the control unit 50 selects a spherical region centered on the reference point. After step S202, the control unit 50 displays the region selected in step S202 on the 3D image (step S203).

[0133] After step S203, the control unit 50 estimates a curved surface that approximates the surface of the 3D shape of the subject by using the 3D coordinates of the points included in the region selected in step S202 (step S204). After step S204, the control unit 50 displays the curved surface on the display 51 (step S205).

[0134] After step S205, the control unit 50 determines whether the curved surface has been accurately estimated (step S206).

[0135] Step S206 will be explained in detail. For example, the user checks the curved surface displayed on the display 51. The user determines the degree of agreement between the surface of the subject and the curved surface and inputs information indicating the result of the determination into the endoscope system 1. Based on the information input by the user, the control unit 50 determines whether the curved surface was accurately estimated.

[0136] When the control unit 50 determines in step S206 that the curved surface has been accurately estimated, the control unit 50 sets the curved surface as the reference surface (step S207). The reference surface information is stored in the volatile memory 56. When step S207 is executed, the process shown in Figure 10, i.e., step S120 shown in Figure 9, is completed.

[0137] If the control unit 50 determines in step S206 that the curved surface has not been accurately estimated, the control unit 50 increments the variable n by 1 (step S208). After step S208, step S201 is executed. After step S208, the control unit 50 may change the position of the already set reference point.

[0138] If steps S201 to S204 are performed two or more times, the control unit 50 sets two or more reference points and selects two or more regions. The control unit 50 estimates a surface based on the 3D coordinates of three or more points, including one or more points included in each of the two or more regions. The control unit 50 may also estimate a surface based on the 3D coordinates of four or more points, including two or more points included in each of the two or more regions.

[0139] If step S201 is executed three times and three reference points are set, the control unit 50 displays the 3D image IMG20 shown in Figure 11 on the display 51.

[0140] The control unit 50 displays three reference points and corresponding regions R1 to R3. Region R1 includes the first reference point, region R2 includes the second reference point, and region R3 includes the third reference point. In the example shown in Figure 11, the control unit 50 estimates the surface based on the 3D coordinates of six or more points, including two or more points in each region.

[0141] Figure 9 will be used again to explain the operation of the endoscope system 1. After the reference plane is set, the user operates the touch panel 52 or the like to input a known reference length of the 3D shape of the subject in the 3D data into the endoscope system 1. The control unit 50 receives the reference length input by the user and sets the reference length (step S121). The reference length information is stored in the volatile memory 56. For example, if the subject is a pipe, the reference length is the radius or diameter of the cylindrical surface that makes up the pipe.

[0142] After step S121, the control unit 50 displays the reference plane set in step S207 and the reference length set in step S121 on the display 51 (step S122).

[0143] After step S122, the control unit 50 measures the reference length and the corresponding length on the 3D data. For example, if the object is a pipe, the control unit 50 calculates the radius or diameter of the reference plane set in step S120 (step S123).

[0144] After step S123, the control unit 50 calculates the ratio between the reference length set in step S121 and the length calculated in step S123. The control unit 50 uses this ratio as a scale factor (scaling factor) to convert the scale of the 3D data (step S124).

[0145] For example, if the reference length is X and the length in the 3D data corresponding to the reference length is Y, then the scale factor is X / Y. In step S124, the control unit 50 converts the 3D data into 3D data with an absolute length dimension. After step S124 is performed, the 3D coordinates in the 3D data are converted into absolute 3D coordinates. The converted 3D data with the scale is stored in the non-volatile memory 57.

[0146] After step S124, the control unit 50 measures the size of the subject based on one or more points in the 3D data (step S125). When step S125 is performed, the information display process shown in Figure 9 is completed.

[0147] Figure 12 shows an example of the procedure for the process performed in step S125. Figure 12 will be used to explain the operation of the endoscope system 1.

[0148] The control unit 50 refers to the measurement mode stored in the volatile memory 56. The measurement mode indicates curved surface reference measurement, plane reference measurement, or two-point distance measurement. Curved surface reference measurement is a mode in which the 3D distance from the measurement point to the point where a straight line perpendicular to the reference curved surface, including the measurement point, intersects the reference curved surface. Plane reference measurement is a mode in which the 3D distance from the measurement point to the point where a straight line perpendicular to the reference plane, including the measurement point, intersects the reference plane. Two-point distance measurement is a mode in which the 3D distance between two measurement points is measured. The control unit 50 determines whether the measurement mode is curved surface reference measurement or not (step S300).

[0149] When the control unit 50 determines in step S300 that the measurement mode is curved surface reference measurement, the control unit 50 sets the measurement point. At this time, the control unit 50 executes steps S104 to S107 shown in Figure 2 to calculate the 3D coordinates of the measurement point (step S301). If step S105 has already been executed, the execution of step S105 may be omitted. The measurement point information is stored in the volatile memory 56.

[0150] For example, the user operates the touch panel 52 to set a measurement point and inputs the measurement point into the endoscope system 1. For example, the user touches the position of the measurement point in the 3D image displayed on the display 51. The control unit 50 receives the measurement point input by the user and sets the measurement point on the 3D image.

[0151] After step S301, the control unit 50 calculates the 3D distance from the reference surface to the measurement point (step S302). The reference surface is the reference surface set in step S207. Since the reference surface used to convert the scale of the 3D data is used as the reference surface in the surface reference measurement, the endoscope system 1 does not need to calculate a new reference surface.

[0152] After step S302, the control unit 50 displays the measurement result on the display 51 (step S303).

[0153] After step S303, the control unit 50 determines whether to continue the measurement (step S304). The control unit 50 may decide to continue the measurement until the user inputs information to the endoscope system 1 indicating that the measurement is to be terminated. When the user inputs information to the endoscope system 1 indicating that the measurement is to be terminated, the control unit 50 may decide not to continue the measurement.

[0154] If the control unit 50 determines in step S304 that measurement should be continued, step S300 is executed. If the control unit 50 determines in step S304 that measurement should not be continued, the process shown in Figure 12 is terminated.

[0155] If the control unit 50 determines in step S300 that the measurement mode is not curved surface reference measurement, the control unit 50 determines whether or not the measurement mode is planar reference measurement (step S305).

[0156] When the control unit 50 determines in step S305 that the measurement mode is planar reference measurement, the control unit 50 sets three reference points (step S306). The three reference points are included in three or more points in the 3D data. Information on the three reference points is stored in the volatile memory 56.

[0157] After step S306, the control unit 50 calculates a reference plane that passes through the three reference points set in step S306 (step S307). After step S307, the control unit 50 displays the reference plane on the display 51 (step S308).

[0158] After step S308, the control unit 50 sets the measurement point (step S309). The measurement point information is stored in the volatile memory 56. The method for setting the measurement point in step S309 is the same as the method for setting the measurement point in step S301.

[0159] After step S309, the control unit 50 calculates the 3D distance from the reference plane to the measurement point (step S310). After step S310, step S303 is executed.

[0160] When the control unit 50 determines in step S305 that the measurement mode is not a planar reference measurement, the control unit 50 sets two measurement points to perform distance measurement between two points (steps S311 and S312). Information on the two measurement points is stored in the volatile memory 56. The method for setting the measurement points in steps S311 and S312 is the same as the method for setting the measurement points in step S301.

[0161] After step S312, the control unit 50 calculates the 3D distance between the two measurement points (step S313). After step S313, step S303 is performed.

[0162] Figure 13 shows an example of an image displayed on the display 51 during the process shown in Figure 12. The control unit 50 displays the 3D image IMG20 on the display 51. When surface reference measurement is performed, the control unit 50 sets the measurement point MP1 shown in Figure 13 on the 3D image IMG20 and displays the measurement point MP1 on the 3D image IMG20. The control unit 50 calculates the 3D distance from the reference surface to the point in 3D space corresponding to the measurement point MP1, and displays the measurement result MR1, which indicates that 3D distance, on the display 51.

[0163] In curved surface-referenced measurement or planar surface-referenced measurement, the measurement points are set, for example, on abnormal parts of the subject. These abnormal parts are recesses or protrusions.

[0164] Figures 14(a) and 14(b) show examples of abnormal parts occurring in a test subject. The test subject SB1 shown in Figures 14(a) and 14(b) is a pipe. The test subject SB1 has an abnormal part AP1. The abnormal part AP1 is a protrusion that occurs on the inner surface of the test subject SB1. The control unit 50 sets a reference plane that approximates the inner surface of the test subject SB1 and sets a measurement point MP3. For example, the measurement point MP3 is the highest point of the abnormal part AP1. The control unit 50 calculates the distance D1 between the reference plane and the measurement point MP3.

[0165] As described above, the control unit 50 estimates the curved surface of the subject and sets a reference length on that curved surface. The control unit 50 calculates the length on the 3D data corresponding to the reference length and converts the scale of the 3D data based on that length and the reference length. The control unit 50 performs the measurement by using the 3D data with the converted scale. The size of the curved surface in the 3D data (e.g., the radius or diameter of a cylindrical surface) and the corresponding length are reflected in the scale of the 3D data. In other words, the scale of the 3D data is converted according to the shape of the subject. Therefore, the accuracy of the scale conversion is improved, and the accuracy of the measurement results is improved.

[0166] The surface possessed by the subject may be a cylindrical surface, a sphere, a torus, an ellipsoid, a parabolic surface, or a freeform surface. The subject does not need to possess the entirety of such a surface. For example, the subject may possess a portion of a sphere.

[0167] The control unit 50 may use 3D data that includes 3D coordinates calculated using stereo images. In this case, the 3D data has a length dimension. Therefore, the control unit 50 does not need to convert the scale of the 3D data.

[0168] Each aspect of the present invention may include the following modifications. The control unit 50 generates 3D shape information indicating the size of the subject based on the second 3D coordinates of one or more points.

[0169] Each aspect of the present invention may include the following modifications. The control unit 50 calculates the second 3D coordinates of four or more points. The control unit 50 sets a reference plane based on the second 3D coordinates of three or more of the four or more points. The control unit 50 generates 3D shape information indicating the distance between one of the four or more points and the reference plane.

[0170] Each aspect of the present invention may include the following modifications. The control unit 50 calculates the second 3D coordinates of two points and generates 3D shape information indicating the distance between the two points.

[0171] In a second modification of the first embodiment, the control unit 50 generates 3D shape information indicating the size of the subject. The endoscope system 1 can acquire detailed size information of the subject.

[0172] (Third modification of the first embodiment) A third modification of the first embodiment of the present invention will now be described. In the third modification of the first embodiment, the endoscope system 1 calculates the 3D coordinates of an arbitrary position on the subject by using the depth buffer method.

[0173] Figure 15 shows an example of the information display processing procedure. Figure 15 will be used to explain the operation of the endoscope system 1. Processes identical to those shown in Figure 2 will not be explained.

[0174] After step S103, step S105 is performed. After step S105, the control unit 50 determines whether each polygon projected onto the plane corresponding to the screen of the display 51 covers each pixel of the 3D image displayed on the display (step S130).

[0175] Figure 16 shows an example of the relationship between polygons and pixels in a 3D image. Polygons POL1 through POL6 are shown. Polygon POL1 covers pixel PIX1. On the other hand, polygons POL2 through POL6 do not cover pixel PIX1.

[0176] After step S130, the control unit 50 calculates the depth value of each pixel using the information of the polygon vertices covering each pixel. The control unit 50 stores the calculated depth value in a buffer (volatile memory 56) (step S131).

[0177] Step S131 will be explained in detail. The control unit 50 performs a projection transformation and calculates the depth value using the following equation (8). depth=(w / (x*px+y*py+z*pz+w)) ···(8)

[0178] In equation (8), w is the W component of the homogeneous coordinates of each vertex of the triangle that makes up each polygon, and is basically 1. In equation (8), x, y, and z are the 3D coordinates of each vertex in the world coordinate system. In equation (8), px, py, and pz represent the components of the viewpoint vector.

[0179] The control unit 50 calculates the depth value of each vertex according to equation (8). The control unit 50 interpolates the depth value of the pixels contained in a triangle by using the depth values ​​of the three vertices of the triangle. If two or more polygons cover a pixel, the control unit 50 calculates the depth value of that pixel for each polygon according to equation (8). The control unit 50 stores the depth value corresponding to the shortest distance among the two or more depth values ​​in a buffer.

[0180] After step S131, the control unit 50 receives the position information output from the touch panel 52 in step S104. After step S104, the control unit 50 obtains the depth value in 2D coordinates of the position indicated by the position information from the buffer (step S132).

[0181] After step S132, the control unit 50 converts the 2D coordinates in the camera coordinate system to 3D coordinates in the world coordinate system using the depth value acquired in step S132 (step S133). After step S133, step S108 is executed.

[0182] Step S133 will be explained in detail. The control unit 50 calculates the 3D coordinates P corresponding to the position indicated by the position information by using the following equation (9).

[0183]

number

[0184] In equation (9), (Sx,Sy) represents the 2D coordinates of the position indicated by the positional information and corresponds to the line of sight. In equation (9), D represents the depth value. The matrix T in equation (9) is expressed by equation (4) mentioned above.

[0185] Each aspect of the present invention may include the following modifications. The line-of-sight information indicates the 2D coordinates of a position on a plane corresponding to the screen of the display 51. The control unit 50 calculates a second 3D coordinate based on the 2D coordinates and the 2D figure obtained by projecting the figure indicated by the polygon information onto the plane.

[0186] In a third modification of the first embodiment, the control unit 50 calculates a second 3D coordinate by using the depth value of the position indicated by the position information. The control unit 50 also generates 3D shape information of the subject based on the second 3D coordinate and displays the 3D shape information on the display 51. The control unit 50 can generate 3D shape information without being limited by the first 3D coordinate included in the 3D data. Therefore, the endoscope system 1 can present detailed 3D shape information of the subject.

[0187] (Fourth modification of the first embodiment) A fourth modification of the first embodiment of the present invention will now be described. In the fourth modification of the first embodiment, the endoscope system 1 displays only the points included in the 3D data and calculates the 3D coordinates of any position on the subject.

[0188] Figure 17 shows an example of the information display processing procedure. The operation of the endoscope system 1 will be explained using Figure 17. Processes identical to those shown in Figure 2 will not be explained.

[0189] 3D data includes mesh information and polygon information. Alternatively, 3D data does not include mesh information and polygon information.

[0190] After step S100, the control unit 50 displays all three or more points included in the 3D data on the display 51. In other words, the control unit 50 displays the entire point cloud as a 3D image on the display 51 (step S140). After step S140, the control unit 50 receives position information output from the touch panel 52 in step S104.

[0191] In step S106, the control unit 50 calculates the 3D coordinates of two points on a 3D line corresponding to the position indicated by the position information. After step S106, the control unit 50 identifies a point among the three or more points included in the 3D data that is close to the 3D line and closest to the camera position (step S141).

[0192] After step S141, the control unit 50 obtains polygon information including the points identified in step S141 from the 3D data. If the 3D data does not contain polygon information, the control unit 50 generates polygons and meshes by using the point cloud around the points identified in step S141. For example, the control unit 50 identifies three or more points in the 3D data that are within a predetermined distance range from the points identified in step S141. The control unit 50 then uses the identified points to perform an algorithm that creates a mesh from the point cloud, such as Delaunay triangulation, and generates polygons and meshes (step S142).

[0193] After step S142, the control unit 50 executes steps S105 to S107 shown in Figure 2 to calculate the 3D coordinates of the intersection point between the 3D line and the polygon (step S143). After step S143, step S108 is executed.

[0194] Each aspect of the present invention may include the following modifications. After receiving position information, the control unit 50 generates polygon information.

[0195] In the fourth modification of the first embodiment, even when a 3D image consisting only of a point cloud is displayed, the endoscope system 1 can present detailed 3D shape information of the subject.

[0196] (Fifth variation of the first embodiment) A fifth modification of the first embodiment of the present invention will now be described. Figure 18 shows an example of the configuration of an endoscope system 1a according to the fifth modification of the first embodiment. Parts that are the same as those shown in Figure 1 will not be described.

[0197] The endoscopic system 1a shown in Figure 18 has an insertion section 2 and a main unit 6. The insertion section 2 and the main unit 6 constitute the endoscopic device 10a.

[0198] The insertion unit 2 shown in Figure 18 is the same as the insertion unit 2 shown in Figure 1. The main unit 6 includes an imaging drive circuit 30, an image processing unit 31, a UD drive unit 32, an RL drive unit 33, a curvature control unit 34, a light source 35, a light source control unit 36, a display 51, a touch panel 52, operation buttons 53, a communication unit 55, a volatile memory 56, a non-volatile memory 57, and a control unit 60. The same blocks as those shown in Figure 1 are assigned the same reference numerals as those shown in Figure 1.

[0199] The control unit 60 has both the functions of the control unit 40 shown in Figure 1 and the functions of the control unit 50 shown in Figure 1. The control unit 60 performs the processes shown in Figures 2, 3, 9, 15, or 17.

[0200] In the fifth modification of the first embodiment, similar to the first embodiment, the endoscope system 1 can present detailed 3D shape information of the subject.

[0201] (Sixth variation of the first embodiment) A sixth modification of the first embodiment of the present invention will now be described. Figure 19 shows an example of the configuration of an endoscope system 1b according to the sixth modification of the first embodiment. Parts that are the same as those shown in Figure 1 will not be described.

[0202] The endoscope system 1b shown in Figure 19 comprises an insertion unit 2, a scope unit 3b, and a base unit 7. The insertion unit 2 and the scope unit 3b constitute the endoscope device 10b. The scope unit 3b and the base unit 7 are connected by a cable 8.

[0203] The insertion unit 2 shown in Figure 19 is the same as the insertion unit 2 shown in Figure 1. The scope unit 3b shown in Figure 19 is the same as the scope unit 3 shown in Figure 1, except that it does not have an image processing unit 31. The base unit 7 has an image processing unit 31, a display 51, a touch panel 52, operation buttons 53, a communication unit 55, a volatile memory 56, a non-volatile memory 57, and a control unit 70. The same reference numerals as shown in Figure 1 are assigned to the same blocks as shown in Figure 1.

[0204] The control unit 70 has both the functions of the control unit 40 shown in Figure 1 and the functions of the control unit 50 shown in Figure 1. The control unit 70 executes the processes shown in Figures 2, 3, 9, 15, or 17.

[0205] In the sixth modification of the first embodiment, similar to the first embodiment, the endoscope system 1 can present detailed 3D shape information of the subject.

[0206] (Various modifications of the first embodiment) Various modifications of the first embodiment of the present invention will be described. These modifications are applicable to the first embodiment and the first to sixth modifications of the first embodiment. These modifications are also applicable to the second and third embodiments, which will be described later.

[0207] The control unit 50 may display both the edges and vertices of each polygon, or it may hide at least one of the edges and vertices of each polygon. When at least one of the edges and vertices of each polygon is not displayed, the user can easily confirm whether or not a position on the polygon's face has been specified.

[0208] The control unit 50 may change the texture resolution or sharpness according to the magnification for displaying the 3D image. For example, when a 3D image is magnified, the texture becomes blurred. Therefore, the control unit 50 may increase the texture resolution or sharpness. The user can easily specify the position on the 3D image.

[0209] As described above, in step S107, the control unit 50 calculates the 3D coordinates of the intersection point between the polygon and the 3D line. If there is no polygon that intersects the 3D line, the control unit 50 may display a warning on the display 51 indicating that it could not detect an intersection point.

[0210] When the transparency of the message is set, the control unit 50 may identify the foreground polygon or the background polygon in step S107. Alternatively, the control unit 50 may identify both the foreground polygon and the background polygon.

[0211] The control unit 50 may display information indicating the direction of the gaze along with the 3D image. Figure 20 shows an example of a 3D image displayed on the display 51. The control unit 50 displays 3D image IMG30 and 3D image IMG31 on the display 51. 3D image IMG30 is a 3D image of the subject as seen from a first viewpoint. 3D image IMG31 is a 3D image of the subject as seen from a second viewpoint different from the first viewpoint.

[0212] The user specifies a position PO1 on the 3D image IMG30. At this time, the control unit 50 displays an arrow AR1 indicating the direction of the line of sight on the 3D image IMG31. The direction indicated by arrow AR1 is parallel to the 3D line calculated in step S106. In other words, the control unit 50 displays arrow AR1 indicating the direction of the 3D line on the 3D image IMG31. The user can then confirm the direction of their line of sight.

[0213] The user uses a pointing device to specify a position on the 3D image. For example, a crosshair (pointer) is displayed on the display 51, and the user moves the crosshair using the pointing device. When the user performs an operation such as clicking, position information indicating the position of the crosshair is output. The control unit 50 may control the movement speed or distance of the crosshair according to the size of the polygon, the number of polygons, the number of points included in the 3D data, or the density of points included in the 3D data.

[0214] For example, when the points in 3D data are widely spaced and the density of those points is low, the user may select two or more points that are widely spaced. Therefore, the control unit 50 may increase the aiming speed or the distance traveled.

[0215] Each aspect of the present invention may include the following modifications. The control unit 50 displays information indicating a 3D line on the display 51.

[0216] (Second embodiment) A second embodiment of the present invention will now be described. In the second embodiment, the endoscope system 1 shown in Figure 1 is used. The endoscope system 1a shown in Figure 18 or the endoscope system 1b shown in Figure 19 may be used instead.

[0217] In the second embodiment, when a vertex or edge of a high-priority polygon exists within a predetermined distance from a position specified by the user on the 3D image, the control unit 50 snaps the point at the specified position to that vertex or edge.

[0218] Figure 21 shows the point snapping method. Polygons POL10 and POL11 exist, and point PT1 at a location specified by the user is located within polygon POL10.

[0219] In the first example, when vertex VT1 has a high priority, the control unit 50 snaps point PT1 to vertex VT1. Vertex VT1 is a vertex of polygon POL10 that covers point PT1.

[0220] In the second example, when vertex VT2 has a higher priority, the control unit 50 snaps point PT1 to vertex VT2. Vertex VT2 is a vertex of polygon POL10 that covers point PT1 and adjacent polygon POL11.

[0221] In the third example, when edge SD1 has a higher priority, the control unit 50 causes point PT1 to snap to edge SD1. Edge SD1 is an edge of polygon POL10 that covers point PT1.

[0222] The control unit 50 sets the priority of polygon vertices or edges according to the polygon's position or size. For example, when a polygon vertex or edge lies on the boundary of the 3D shape of the subject, the control unit 50 sets a high priority for that vertex or edge. For example, the control unit 50 checks the number of faces connected to each edge of each polygon. When only one face is connected to an edge, the control unit 50 can determine that the edge lies on the boundary. Alternatively, the control unit 50 sets a high priority for vertices or edges of larger polygons and a low priority for vertices or edges of smaller polygons.

[0223] Alternatively, the control unit 50 calculates the priority of a polygon based on the texture information associated with the polygon. For example, the control unit 50 calculates the maximum edge intensity of the texture corresponding to each polygon. When that edge intensity is greater than a reference value, the control unit 50 sets a high priority for that polygon. Alternatively, the control unit 50 applies a feature extraction method such as FAST (Features from Accelerated Segment Test) or ORB (Oriented FAST and Rotated BRIEF) to the texture corresponding to each polygon. When the extracted feature quantity is greater than a reference value, the control unit 50 sets a high priority for that polygon.

[0224] When specifying points on a scaled-down 3D image, precisely defining the location of those points can be difficult. In such cases, snapping the user-specified points to other points may streamline the point selection process.

[0225] Either a first mode or a second mode is set in the endoscope system 1, and the endoscope system 1 operates in either the first mode or the second mode. The first and second modes are switchable. For example, the control unit 50 switches the mode of the endoscope system 1 based on information output from the touch panel 52 or the like. The control unit 50 may also determine the shooting scene and switch the mode of the endoscope system 1 according to the shooting scene.

[0226] When the first mode is set for the endoscope system 1, the control unit 50 uses 3D coordinates calculated by the method shown in the first embodiment. When the second mode is set for the endoscope system 1, the control unit 50 snaps a point corresponding to a position on the 3D image to another point.

[0227] The control unit 50 may switch between the first mode and the second mode depending on the magnification of the 3D image displayed on the display 51. For example, when the magnification is high, the control unit 50 may set the mode of the endoscope system 1 to the first mode. When the magnification is low, the control unit 50 may set the mode of the endoscope system 1 to the second mode.

[0228] Figures 22 and 23 show examples of information display processing procedures. Figures 22 and 23 are used to explain the operation of the endoscope system 1. Processes identical to those shown in Figure 2 will not be explained.

[0229] After step S107, the control unit 50 determines whether the point suction function is enabled or not (step S150). When the first mode is set to the endoscope system 1, the point suction function is not enabled. When the second mode is set to the endoscope system 1, the point suction function is enabled.

[0230] If the control unit 50 determines in step S150 that the point snapping function is not enabled, step S108 is executed. If the control unit 50 determines in step S150 that the point snapping function is enabled, the control unit 50 refers to setting information that indicates whether to snap a point (specified point) corresponding to a position on the 3D image to a polygon vertex or edge. The setting information is pre-stored in the volatile memory 56. The control unit 50 determines whether or not to snap the specified point to a polygon vertex (step S151).

[0231] When the setting information indicates that the specified point should be attached to a polygon vertex, the control unit 50 determines that the specified point should be attached to a polygon vertex. When the setting information indicates that the specified point should be attached to an edge of the polygon, the control unit 50 determines that the specified point should not be attached to a polygon vertex.

[0232] When the control unit 50 determines in step S151 that the specified point should be snapped to a polygon vertex, the control unit 50 determines whether or not there are high-priority polygon vertices within a predetermined distance range from the specified point (step S152). If there are no high-priority polygon vertices within a predetermined distance range from the specified point, step S108 is executed. At this time, the 3D coordinates (second 3D coordinates) calculated in step S107 are used.

[0233] When a polygon vertex with higher priority exists within a predetermined distance from the specified point, the control unit 50 snaps the specified point to that vertex and obtains the 3D coordinates of that vertex from the 3D data. When two or more vertices have the same priority, the control unit 50 snaps the specified point to the vertex closest to it (step S153). After step S153, step S108 is executed. At this time, the 3D coordinates obtained in step S153 (third 3D coordinates) are used.

[0234] If the control unit 50 determines in step S151 that the specified point will not be snapped to a polygon vertex, the control unit 50 determines whether or not there is a high-priority polygon edge within a predetermined distance from the specified point (step S154). If there is no high-priority polygon edge within a predetermined distance from the specified point, step S108 is executed. At this time, the 3D coordinates (second 3D coordinates) calculated in step S107 are used.

[0235] When a polygon edge with a higher priority exists within a predetermined distance from the specified point, the control unit 50 snaps the specified point to a point on that edge. When two or more edges have the same priority, the control unit 50 snaps the specified point to the edge closest to it. The control unit 50 calculates the 3D coordinates of the point corresponding to the specified point by using the 3D coordinates of two vertices on that edge (step S155). After step S155, step S108 is executed. At this time, the 3D coordinates calculated in step S155 (third 3D coordinates) are used.

[0236] Step S155 is described in detail. The 3D coordinates of two vertices on an edge are defined as P0 and P1, and the 2D coordinates of those two vertices projected onto the plane corresponding to the screen of the display 51 are defined as Q0 and Q1. The control unit 50 calculates the distance D from a designated point S to the point closest to it using the following equations (10), (11), and (12). The point closest to the designated point S lies on the line segment Q0Q1.

[0237]

number

[0238]

number

[0239]

number

[0240] The control unit 50 selects an edge with a high priority where the distance D is within a predetermined distance. The control unit 50 snaps the designated point to a point on that edge. That point is the closest to the designated point. The control unit 50 executes steps S105 to S107 shown in Figure 2 and calculates the 3D coordinates of that point.

[0241] The control unit 50 may analyze the polygon containing the specified point and the corresponding texture information. The texture information is associated with the polygon. The control unit 50 may snap the specified point to a point in the characteristic texture. For example, in the example shown in Figure 24, the specified point DP1 is included in the polygon POL20. The control unit 50 analyzes the polygon POL20 and the corresponding texture information and detects point PT2 in the characteristic texture. The control unit 50 snaps the specified point DP1 to point PT2. The control unit 50 performs steps S105 to S107 shown in Figure 2 and calculates the 3D coordinates of point PT2.

[0242] Each aspect of the present invention may include the following modifications. When a first mode is set, the control unit 50 generates 3D shape information based on a second 3D coordinate. When a second mode different from the first mode is set, the control unit 50 generates 3D shape information based on a third 3D coordinate calculated based on the second 3D coordinate. The third 3D coordinate is different from the second 3D coordinate.

[0243] Each aspect of the present invention may include the following modifications: The third 3D coordinate is the first 3D coordinate of one of three or more points included in the 3D data, or the 3D coordinate of a point included in the edge of the figure indicated by the polygon information.

[0244] Each aspect of the present invention may include the following modifications. The control unit 50 calculates a third 3D coordinate based on the texture information of the figure at the position indicated by the position information. The texture information is associated with polygon information.

[0245] Each aspect of the present invention may include the following modifications. When a first mode is set, the control unit 50 generates 3D shape information based on a second 3D coordinate. When a second mode different from the first mode is set, the control unit 50 calculates a second position (point PT2) on the 3D image based on the texture information of the figure at the position indicated by the position information (designated point DP1). The texture information is associated with polygon information. The second position is different from the position indicated by the position information. The control unit 50 calculates a third 3D coordinate based on the line of sight information at the second position and the polygon information at the second position. The third 3D coordinate is different from the first 3D coordinate of each of the three or more points. The control unit 50 generates 3D shape information based on the third 3D coordinate.

[0246] In the second embodiment, the control unit 50 snaps the point at the location indicated by the position information to a polygon vertex or edge, etc. The endoscope system 1 can efficiently specify the point.

[0247] (Third embodiment) A third embodiment of the present invention will now be described. In the third embodiment, the endoscope system 1 shown in Figure 1 is used. The endoscope system 1a shown in Figure 18 or the endoscope system 1b shown in Figure 19 may be used instead.

[0248] In a third embodiment, the control unit 50 pre-detects one or more objects based on texture information. For example, the control unit 50 detects objects by performing segmentation using a convolutional neural network. Alternatively, the control unit 50 detects objects by performing segmentation using thresholding or clustering such as k-means. The control unit 50 may also detect objects based on the 3D shape of the subject as indicated by the 3D data.

[0249] First, the touch panel 52 outputs first position information indicating a first position on the 3D image. The control unit 50 identifies the object containing the first position indicated by the first position information. Next, the touch panel 52 outputs second position information indicating a second position on the 3D image. The control unit 50 identifies the object containing the second position indicated by the second position information. When the object containing the second position is the same as the object containing the first position, the control unit 50 accepts the second position information. When the object containing the second position is different from the object containing the first position, the control unit 50 does not accept the second position information.

[0250] If the user specifies three or more points on the 3D image, the control unit 50 performs the same processing as described above. When two or more points on the 3D image are all included in the same object, the control unit 50 generates 3D shape information by using the 3D coordinates of those two or more points.

[0251] Figures 25(a) and 25(b) show examples of object detection processes. Figure 25(a) shows texture information TX1. Figure 25(b) shows objects OB1 and OB2 detected from texture information TX1.

[0252] Figures 26 and 27 show examples of the procedure of information display processing. Using FIGS. 26 and 27, the operation of the endoscope system 1 will be described. The same processing as that shown in FIG. 2 will not be described.

[0253] Hereinafter, the user designates two or more points on the 3D image. For example, when measuring the distance between the reference plane and the measurement point, it is necessary to set three or more points to set the reference plane.

[0254] After step S103, the control unit 50 detects one or more objects based on the texture information synthesized in the 3D image (step S160). After step S160, in step S104, the control unit 50 receives the position information output from the touch panel 52. After step S107, the control unit 50 identifies the object corresponding to the position indicated by the position information (step S161). The information of the identified object is stored in the volatile memory 56. Step S161 may be executed between step S104 and step S107.

[0255] The user operates the touch panel 52 to re-enter information indicating the position on the 3D image into the endoscope system 1. The touch panel 52 outputs position information indicating the position to the control unit 50. The control unit 50 receives the position information output from the touch panel 52 (step S162).

[0256] After step S162, the control unit 50 identifies the object corresponding to the position indicated by the position information (step S163). After step S163, the control unit 50 determines whether the object identified in step S163 is the same as the object identified in step S161 (step S164).

[0257] When the object identified in step S163 is different from the object identified in step S161, step S162 is executed. At this time, the control unit 50 may display information prompting re-setting of the point on the display unit 201.

[0258] When the object specified in step S163 is the same as the object specified in step S161, the control unit 50 calculates the 3D coordinates of two points on the 3D straight line corresponding to the position indicated by the position information (step S165). Step S165 is the same as step S106.

[0259] After step S165, the control unit 50 calculates the 3D coordinates corresponding to the position indicated by the position information (step S166). Step S166 is the same as step S107.

[0260] After step S166, the control unit 50 determines whether to end the setting of the position on the 3D image (step S167). For example, when the number of necessary points is set in advance and the number of set points is less than the number of necessary points, the control unit 50 determines not to end the setting of the position on the 3D image. At this time, step S162 is executed. When the number of set points reaches the number of necessary points, the control unit 50 determines to end the setting of the position on the 3D image. At this time, step S108 is executed.

[0261] Each aspect of the present invention may include the following modification examples. The control unit 50 detects an object in the subject. The control unit 50 receives first position information and second position information as position information. The first position information indicates a first position. The second position information indicates a second position different from the first position. When the object including the first position and the object including the second position are the same, the control unit 50 generates 3D shape information based on the second 3D coordinates corresponding to the first position and the second 3D coordinates corresponding to the second position.

[0262] Each aspect of the present invention may include the following modification examples. The control unit 50 detects an object based on texture information associated with polygon information.

[0263] In a third embodiment, the control unit 50 generates 3D shape information by using the 3D coordinates of two or more points contained within the same object. For example, in measuring the distance between a reference plane and a measurement point, the control unit 50 sets the reference plane based on three or more points on the same object. The endoscope system 1 can avoid using points set in positions unintended by the user to generate 3D shape information.

[0264] While preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their variations. Additions, omissions, substitutions, and other modifications are possible without departing from the spirit of the invention. Furthermore, the present invention is not limited by the foregoing description, but only by the scope of the appended claims. [Explanation of Symbols]

[0265] 1,1a,1b Endoscopic System 2 Insertion part 2a Tip 3,3b Scope Unit 4.7 Base Unit 5,6 Main Unit 10, 10a, 10b Endoscope equipment 11 External device 20 Imaging Department 21 Curved section 22 Lighting windows 30 Imaging drive circuit 31 Image Processing Unit 32 UD drive unit 33 RL drive unit 34 Curve Control Unit 35 Light source 36 Light source control unit 40, 50, 60, 70 Control Unit 41, 54, 55 Communications Department 42,56 Volatile memory 43,57 Non-volatile memory 51 displays 52 Touch Panel 53 Operation buttons

Claims

1. It has a control unit, and the control unit is Based on images of the subject obtained by endoscopy, three-dimensional data is acquired that includes the first three-dimensional coordinates of three or more points on the subject. A three-dimensional image is displayed on a display based on the three-dimensional data, and the three-dimensional image is an image of the three-dimensional shape of the subject, including a point having the first three-dimensional coordinates. The system receives position information output from a pointing device, and the position information indicates a position on the three-dimensional image. Based on the line-of-sight information at the aforementioned location and the polygon information at the aforementioned location, a second three-dimensional coordinate is calculated for one or more points, wherein the second three-dimensional coordinate differs from the first three-dimensional coordinate of each of the three or more points, and the polygon information represents a figure generated based on the three or more points. Based on the second three-dimensional coordinates of one or more points, three-dimensional shape information of the subject is generated. The three-dimensional shape information is displayed on the display. 3D shape information display device.

2. The control unit, A reference plane is set based on the second three-dimensional coordinates of one or more points. The three-dimensional data is divided into a first region and a second region with the aforementioned reference plane as the boundary. The three-dimensional shape information is generated showing the cross-section of the subject, which is set based on the reference plane. The three-dimensional shape information display device according to claim 1.

3. The control unit generates the three-dimensional shape information indicating the size of the subject based on the second three-dimensional coordinates of one or more points. The three-dimensional shape information display device according to claim 1.

4. The control unit, The second three-dimensional coordinates of four or more points, including the one or more points mentioned above, are calculated. A reference plane is set based on the second three-dimensional coordinates of three or more points among the four or more points mentioned above. The three-dimensional shape information is generated, which indicates the distance between one of the four or more points and the reference plane. The three-dimensional shape information display device according to claim 3.

5. The control unit, The second three-dimensional coordinates of two points, which are the one or more points mentioned above, are calculated. This generates the three-dimensional shape information that indicates the distance between the two points. The three-dimensional shape information display device according to claim 3.

6. The aforementioned three-dimensional data includes the polygon information. The three-dimensional shape information display device according to claim 1.

7. After the position information is received, the control unit generates the polygon information. The three-dimensional shape information display device according to claim 1.

8. The line of sight information indicates a three-dimensional line including two or more points corresponding to the position, The control unit calculates the second three-dimensional coordinate of the intersection point between the three-dimensional line and the figure indicated by the polygon information. The three-dimensional shape information display device according to claim 1.

9. The polygon information indicates two or more of the aforementioned figures, The aforementioned intersection point is the intersection point between the three-dimensional line and the figure among the two or more figures that is closest to the camera position. The three-dimensional shape information display device according to claim 8.

10. The control unit displays information indicating the three-dimensional line on the display. The three-dimensional shape information display device according to claim 8.

11. The line-of-sight information indicates the two-dimensional coordinates of the position on the plane corresponding to the screen of the display. The control unit calculates the second three-dimensional coordinates based on the two-dimensional coordinates and the two-dimensional figure obtained by projecting the figure indicated by the polygon information onto the plane. The three-dimensional shape information display device according to claim 1.

12. The control unit, When the first mode is set, the three-dimensional shape information is generated based on the second three-dimensional coordinates. When a second mode different from the first mode is set, the three-dimensional shape information is generated based on a third three-dimensional coordinate calculated based on the second three-dimensional coordinate, and the third three-dimensional coordinate is different from the second three-dimensional coordinate. The three-dimensional shape information display device according to claim 1.

13. The third three-dimensional coordinate is the first three-dimensional coordinate of one of the three or more points, or the three-dimensional coordinate of a point included in the edge of the figure indicated by the polygon information. The three-dimensional shape information display device according to claim 12.

14. The control unit calculates the third three-dimensional coordinates based on the texture information of the figure at the position, The aforementioned texture information is associated with the aforementioned polygon information. The three-dimensional shape information display device according to claim 12.

15. The control unit, When the first mode is set, the three-dimensional shape information is generated based on the second three-dimensional coordinates. When a second mode different from the first mode is set, a second position on the three-dimensional image is calculated based on the texture information of the figure at the position, the texture information is associated with the polygon information, and the second position is different from the first position. A third three-dimensional coordinate is calculated based on the line-of-sight information at the second position and the polygon information at the second position, and the third three-dimensional coordinate is different from the first three-dimensional coordinate of each of the three or more points. The three-dimensional shape information is generated based on the third three-dimensional coordinates. The three-dimensional shape information display device according to claim 1.

16. The control unit, The object in the subject is detected, The system accepts a first location and a second location as location information, the first location indicates a first location, and the second location indicates a second location different from the first location. When the object including the first position and the object including the second position are the same, the three-dimensional shape information is generated based on the second three-dimensional coordinates corresponding to the first position and the second three-dimensional coordinates corresponding to the second position. The three-dimensional shape information display device according to claim 1.

17. The control unit detects the object based on the texture information associated with the polygon information. The three-dimensional shape information display device according to claim 16.

18. The control unit Based on images of the subject obtained by endoscopy, three-dimensional data is acquired that includes the first three-dimensional coordinates of three or more points on the subject. A three-dimensional image is displayed on a display based on the three-dimensional data, and the three-dimensional image is an image of the three-dimensional shape of the subject, including a point having the first three-dimensional coordinates. The system receives position information output from a pointing device, and the position information indicates a position on the three-dimensional image. Based on the line-of-sight information at the aforementioned location and the polygon information at the aforementioned location, a second three-dimensional coordinate is calculated for one or more points, wherein the second three-dimensional coordinate differs from the first three-dimensional coordinate of each of the three or more points, and the polygon information represents a figure generated based on the three or more points. Based on the second three-dimensional coordinates of one or more points, three-dimensional shape information of the subject is generated. The three-dimensional shape information is displayed on the display. 3D shape information display method.

19. A step of acquiring three-dimensional data including first three-dimensional coordinates of three or more points on the subject calculated based on images of the subject obtained by endoscopy, A step of displaying a three-dimensional image on a display based on the three-dimensional data, wherein the three-dimensional image is an image of the three-dimensional shape of the subject, including a point having the first three-dimensional coordinates. A step of receiving position information output from a pointing device, wherein the position information indicates a position on the three-dimensional image, A step of calculating a second three-dimensional coordinate of one or more points based on line-of-sight information at the said position and polygon information at the said position, wherein the second three-dimensional coordinate is different from the first three-dimensional coordinate of each of the three or more points, and the polygon information indicates a figure generated based on the three or more points. A step of generating three-dimensional shape information of the subject based on the second three-dimensional coordinates of one or more points, The steps include: displaying the three-dimensional shape information on the display; A program that causes a computer to execute something.

Images