Three-dimensional shape information display device, three-dimensional shape information display method, and program

Abstract

A three-dimensional shape information display device according to the present invention comprises a control unit. The control unit acquires three-dimensional data that includes first three-dimensional coordinates at three or more points on a subject, and displays a three-dimensional image on a display on the basis of the three-dimensional data. The control unit receives position information output from a pointing device, and calculates second three-dimensional coordinates at one or more points on the basis of line-of-sight information at a position indicated by the position information and polygon information at the position. The control unit generates three-dimensional shape information of the subject on the basis of the second three-dimensional coordinates at the one or more points, and displays the three-dimensional shape information on the display.

Description

Three-dimensional shape information display device, three-dimensional shape information display method, and program

[0001] The present invention relates to a three-dimensional shape information display device, a three-dimensional shape information display method, and a program. This application claims priority to Japanese Patent Application No. 2024-155145, filed on September 9, 2024, the contents of which are incorporated herein by reference.

[0002] Industrial endoscope devices are used for inspecting (endoscopic inspection) the interior of boilers, pipes, aircraft engines, heat exchangers, etc. for abnormalities, corrosion, etc. The device disclosed in Patent Document 1 has a long, thin probe including an insertion tube that can be inserted into an object to be observed, and generates an image based on an optical image acquired via the probe. The device determines three-dimensional (3D) coordinates of points on the object to be observed by using the image of the object to be observed, and determines a reference plane by using the 3D coordinates of three or more points. The device calculates the distance between each point and the reference plane and displays a color map of each point, with each point colored according to the distance.

[0003] Patent No. 6030837

[0004] The device disclosed in Patent Document 1 calculates the 3D coordinates of multiple points and determines a reference plane by using the 3D coordinates of three or more points selected as measurement points. The 3D coordinates used to determine the reference plane are limited to the pre-calculated 3D coordinates. Since the device cannot select arbitrary positions on the object to be observed as measurement points, it cannot provide detailed information about the three-dimensional shape of the object.

[0005] An object of the present invention is to provide a three-dimensional shape information display device, a three-dimensional shape information display method, and a program that can present detailed three-dimensional shape information of a subject.

[0006] A three-dimensional shape information display device according to an aspect of the present invention includes a control unit. The control unit acquires three-dimensional data including first three-dimensional coordinates of three or more points on a subject, the first three-dimensional coordinates being calculated based on an image of the subject acquired by an endoscope. The control unit displays a three-dimensional image on a display based on the three-dimensional data. The three-dimensional image is an image of the three-dimensional shape of the subject including points having the first three-dimensional coordinates. The control unit accepts position information output from a pointing device. The position information indicates a position on the three-dimensional image. The control unit calculates second three-dimensional coordinates of one or more points based on line-of-sight information at the position and polygon information at the position. The second three-dimensional coordinates are different from the first three-dimensional coordinates of each of the three or more points. The polygon information indicates a figure generated based on the three or more points. The control unit generates three-dimensional shape information of the subject based on the second three-dimensional coordinates of the one or more points. The control unit displays the three-dimensional shape information on the display.

[0007] In the three-dimensional shape information display device according to an aspect of the present invention, the control unit may set a reference plane based on the second three-dimensional coordinates of the one or more points. The control unit may divide the three-dimensional data into a first region and a second region using the reference plane as a boundary. The control unit may generate the three-dimensional shape information indicating a cross section of the subject set based on the reference plane.

[0008] In the three-dimensional shape information display device according to this aspect of the present invention, the control unit may generate the three-dimensional shape information indicating the size of the subject based on the second three-dimensional coordinates of the one or more points.

[0009] In the three-dimensional shape information display device according to an aspect of the present invention, the control unit may calculate second three-dimensional coordinates of four or more points including the one or more points. The control unit may set a reference plane based on the second three-dimensional coordinates of three or more of the four or more points. The control unit may generate the three-dimensional shape information indicating a distance between one of the four or more points and the reference plane.

[0010] In the three-dimensional shape information display device according to an aspect of the present invention, the control unit may calculate second three-dimensional coordinates of two points as the one or more points, and the control unit may generate the three-dimensional shape information indicating the distance between the two points.

[0011] In the three-dimensional shape information display device according to this aspect of the present invention, the three-dimensional data may include the polygon information.

[0012] In the three-dimensional shape information display device according to this aspect of the present invention, after the position information is received, the control unit may generate the polygon information.

[0013] In the three-dimensional shape information display device according to an aspect of the present invention, the line of sight information may indicate a three-dimensional line including two or more points corresponding to the position, and the control unit may calculate the second three-dimensional coordinates of an intersection between the three-dimensional line and the shape indicated by the polygon information.

[0014] In the three-dimensional shape information display device according to an aspect of the present invention, the polygon information may represent two or more of the figures, and the intersection may be an intersection between the three-dimensional line and a figure closest to a camera position among the two or more figures.

[0015] In the three-dimensional shape information display device according to this aspect of the present invention, the control unit may display information indicating the three-dimensional line on the display.

[0016] In the three-dimensional shape information display device according to an aspect of the present invention, the line-of-sight information may indicate two-dimensional coordinates of the position on a plane corresponding to a screen of the display, and the control unit may calculate the second three-dimensional coordinates based on the two-dimensional coordinates and a two-dimensional figure obtained by projecting the figure indicated by the polygon information onto the plane.

[0017] In the three-dimensional shape information display device according to this aspect of the present invention, the control unit may generate the three-dimensional shape information based on the second three-dimensional coordinates when a first mode is set. When a second mode different from the first mode is set, the control unit may generate the three-dimensional shape information based on third three-dimensional coordinates calculated based on the second three-dimensional coordinates. The third three-dimensional coordinates may be different from the second three-dimensional coordinates.

[0018] In a three-dimensional shape information display device according to this aspect of the present invention, the third three-dimensional coordinates may be the first three-dimensional coordinates of one of the three or more points or the three-dimensional coordinates of a point included in an edge of the figure indicated by the polygon information.

[0019] In the three-dimensional shape information display device according to this aspect of the present invention, the control unit may calculate the third three-dimensional coordinates based on texture information of the figure at the position. The texture information may be associated with the polygon information.

[0020] In the three-dimensional shape information display device according to this aspect of the present invention, when a first mode is set, the control unit may generate the three-dimensional shape information based on the second three-dimensional coordinates. When a second mode different from the first mode is set, the control unit may calculate a second position on the three-dimensional image based on texture information of the figure at the position. The texture information may be associated with the polygon information. The second position may be different from the position. The control unit may calculate third three-dimensional coordinates based on line-of-sight information at the second position and the polygon information at the second position. The third three-dimensional coordinates may be different from the first three-dimensional coordinates of each of the three or more points. The control unit may generate the three-dimensional shape information based on the third three-dimensional coordinates.

[0021] In the three-dimensional shape information display device according to this aspect of the present invention, the control unit may detect an object on the subject. The control unit may accept first position information and second position information as the position information. The first position information may indicate a first position. The second position information may indicate a second position different from the first position. When the object including the first position and the object including the second position are the same, the control unit may generate the three-dimensional shape information based on the second three-dimensional coordinates corresponding to the first position and the second three-dimensional coordinates corresponding to the second position.

[0022] In the three-dimensional shape information display device according to this aspect of the present invention, the control unit may detect the object based on texture information associated with the polygon information.

[0023] A three-dimensional shape information display method according to an aspect of the present invention includes the following processes: a control unit acquires three-dimensional data including first three-dimensional coordinates of three or more points on a subject, the first three-dimensional coordinates being calculated based on an image of the subject acquired by an endoscope; the control unit displays a three-dimensional image on a display based on the three-dimensional data; the three-dimensional image is an image of the three-dimensional shape of the subject including points having the first three-dimensional coordinates; the control unit accepts position information output from a pointing device; the position information indicates a position on the three-dimensional image; the control unit calculates second three-dimensional coordinates of one or more points based on line-of-sight information at the position and polygon information at the position; the second three-dimensional coordinates are different from the first three-dimensional coordinates of each of the three or more points; the polygon information indicates a figure generated based on the three or more points; the control unit generates three-dimensional shape information of the subject based on the second three-dimensional coordinates of the one or more points; the control unit displays the three-dimensional shape information on the display.

[0024] A program according to an aspect of the present invention causes a computer to execute the following processes. The computer acquires three-dimensional data including first three-dimensional coordinates of three or more points on a subject, the first three-dimensional coordinates being calculated based on an image of the subject acquired by an endoscope. The computer displays a three-dimensional image on a display based on the three-dimensional data. The three-dimensional image is an image of the three-dimensional shape of the subject including points having the first three-dimensional coordinates. The computer accepts position information output from a pointing device. The position information indicates a position on the three-dimensional image. The computer calculates second three-dimensional coordinates of one or more points based on line-of-sight information at the position and polygon information at the position. The second three-dimensional coordinates are different from the first three-dimensional coordinates of each of the three or more points. The polygon information indicates a figure generated based on the three or more points. The computer generates three-dimensional shape information of the subject based on the second three-dimensional coordinates of the one or more points. The computer displays the three-dimensional shape information on the display.

[0025] According to the above aspects, the three-dimensional shape information display device, the three-dimensional shape information display method, and the program can present detailed three-dimensional shape information of the subject.

[0026] FIG. 1 is a block diagram showing an example of the configuration of an endoscopic system according to a first embodiment of the present invention. FIG. 2 is a flowchart showing an example of the procedure of information display processing in a first modified example of the first embodiment of the present invention. FIG. 3 is a diagram showing an example of an image displayed on a display included in an endoscopic system according to a first modified example of the first embodiment of the present invention. FIG. 4 is a diagram showing an example of an image displayed on a display included in an endoscopic system according to a first modified example of the first embodiment of the present invention. FIG. 5 is a diagram showing a 3D space defined in 3D data in a first modified example of the first embodiment of the present invention. FIG. 6 is a diagram showing an example of an image displayed on a display included in an endoscopic system according to a first modified example of the first embodiment of the present invention. FIG. 7 is a flowchart showing an example of the procedure of information display processing in a second modified example of the first embodiment of the present invention. FIG. 8 is a flowchart showing an example of the procedure of processing for setting a reference plane in a second modified example of the first embodiment of the present invention. FIG. 9 is a flowchart showing an example of the procedure of processing for measuring the size of a subject in a second modified example of the first embodiment of the present invention. FIG. 10 is a diagram showing an example of an abnormal part that occurs in a subject in a second modified example of the first embodiment of the present invention. FIG. 11 is a diagram showing an example of an abnormal part that occurs in a subject in a second modified example of the first embodiment of the present invention. FIG. 12 is a flowchart showing an example of a procedure for information display processing in a third modified example of the first embodiment of the present invention. FIG. 13 is a diagram showing an example of a relationship between polygons and pixels of a 3D image in a third modified example of the first embodiment of the present invention. FIG. 14 is a flowchart showing an example of a procedure for information display processing in a fourth modified example of the first embodiment of the present invention. FIG. 15 is a block diagram showing an example of the configuration of an endoscope system according to a fifth modified example of the first embodiment of the present invention. FIG. 16 is a block diagram showing an example of the configuration of an endoscope system according to a sixth modified example of the first embodiment of the present invention.FIG. 10 is a diagram showing an example of an image displayed on a display of an endoscope system according to various modified examples of the first embodiment of the present invention. FIG. 11 is a diagram showing a point adsorption method according to a second embodiment of the present invention. FIG. 12 is a flowchart showing an example of a procedure for information display processing according to the second embodiment of the present invention. FIG. 13 is a diagram showing an example of a procedure for information display processing according to the second embodiment of the present invention. FIG. 14 is a diagram showing an example of a procedure for information display processing according to the third embodiment of the present invention. FIG. 15 is a diagram showing an example of a procedure for information display processing according to the third embodiment of the present invention.

[0027] An embodiment of the present invention will be described below with reference to the drawings. An endoscope system will be described below as an example of a three-dimensional (3D) shape information display device.

[0028] First Embodiment Fig. 1 shows an example of the configuration of an endoscope system 1 according to a first embodiment of the present invention. The endoscope system 1 shown in Fig. 1 has an insertion section 2, a scope unit 3, a base unit 4, and a main unit 5. The insertion section 2, the scope unit 3, and the base unit 4 constitute an endoscope device 10. The main unit 5 is an operation device.

[0029] The insertion section 2 is inserted into the inside of the subject to be observed. The subject is an industrial product. The insertion section 2 is a long, thin tube that is bendable. A user performs an insertion operation to insert the insertion section 2 into the subject. An optical adapter is attached to the tip of the insertion section 2. The insertion section 2 acquires an optical image of the inside of the subject. The insertion section 2 has an imaging section 20, a bending section 21, and an illumination window 22.

[0030] The imaging unit 20 is disposed in the distal end portion 2a including the distal end of the insertion portion 2. The imaging unit 20 is an image sensor such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor. The imaging unit 20 generates an image based on an optical image acquired by the insertion portion 2. The image generated by the imaging unit 20 is output to the scope unit 3.

[0031] The bending section 21 bends the insertion section 2 in the upward (U), downward (D), leftward (L), or rightward (R) direction. Alternatively, the bending section 21 bends the insertion section 2 in the upward-left (UL), upward-right (UR), downward-left (DL), or downward-right (DR) direction.

[0032] Illumination light is generated by a light source 35 included in the scope unit 3 and output to the distal end portion 2a through a light guide (not shown) arranged inside the insertion portion 2. The illumination light is irradiated from the illumination window 22 into the inside of the subject.

[0033] The scope unit 3 has an imaging drive circuit 30, an image processing unit 31, a UD drive unit 32, an RL drive unit 33, a bending control unit 34, a light source 35, and a light source control unit 36. The base unit 4 has a control unit 40, a communication unit 41, a volatile memory 42, and a non-volatile memory 43.

[0034] The imaging drive circuit 30 controls the imaging unit 20 and outputs the image output from the imaging unit 20 to the image processing unit 31. The image processing unit 31 performs image processing such as noise reduction on the image output from the imaging unit 20 and outputs the image to the control unit 40.

[0035] The UD driving unit 32 is connected to a UD bending wire for bending the bending portion 21 in the U direction or the D direction. The UD driving unit 32 has a motor, and bends the bending portion 21 in the U direction or the D direction by pulling the UD bending wire. The RL driving unit 33 is connected to an RL bending wire for bending the bending portion 21 in the R direction or the L direction. The RL driving unit 33 has a motor, and bends the bending portion 21 in the R direction or the L direction by pulling the RL bending wire. The bending control unit 34 controls the UD driving unit 32 and the RL driving unit 33.

[0036] The UD drive unit 32 and the RL drive unit 33 can operate simultaneously. For example, the UD drive unit 32 and the RL drive unit 33 can bend the bending portion 21 in the UL direction.

[0037] The light source 35 is a light-emitting diode (LED) or the like, and generates illumination light. The illumination light is output from the light source 35 to a light guide (not shown). The light source control unit 36 ​​controls the light source 35.

[0038] The control unit 40 controls each unit of the scope unit 3 and the base unit 4. At least one of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36 ​​may be configured with at least one of a processor and a logic circuit. For example, the processor is at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). At least one of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36 ​​may include one or more processors. At least one of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36 ​​may include one or more logic circuits.

[0039] The computer of the endoscope system 1 may load a program and execute the loaded program. The program includes instructions that define the operation of at least one of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36. In other words, at least one function of the control unit 40, the image processing unit 31, the bending control unit 34, and the light source control unit 36 ​​may be realized by software.

[0040] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer storing the program to the endoscope system 1 via a transmission medium or by transmission waves in the transmission medium. The "transmission medium" that transmits the program is a medium that has the function of transmitting information. Media that have the function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the functions described above. Furthermore, the above program may be a difference file (difference program). The functions described above may be realized by combining a program already recorded on the computer with the difference program.

[0041] The communication unit 41 has a communication circuit and performs wired or wireless communication with the main unit 5 for curvature control and the like. The communication unit 41 transmits images generated by the imaging unit 20 to the main unit 5. The volatile memory 42 is a RAM (Random Access Memory) or a DRAM (Dynamic RAM), for example. The volatile memory 42 stores various information processed by the control unit 40. The non-volatile memory 43 is a SRAM (Static RAM), an EPROM (Erasable Programmable Read-Only Memory), an EEPROM (Electrically Erasable Programmable ROM), or a flash memory. The non-volatile memory 43 may be detachable from the base unit 4. The non-volatile memory 43 stores images generated by the imaging unit 20 and various information processed by the control unit 40 .

[0042] The main unit 5 has a control unit 50, a display 51, a touch panel 52, operation buttons 53, a communication unit 54, a communication unit 55, a volatile memory 56, and a non-volatile memory 57. The main unit 5 may be an information terminal such as a smartphone or a tablet terminal.

[0043] The control unit 50 controls each part of the main unit 5. The control unit 50 may be composed of at least one of a processor and a logic circuit. The control unit 50 may include one or more processors. The control unit 50 may include one or more logic circuits. The computer of the endoscope system 1 may load a program and execute the loaded program. The program includes instructions that define the operation of the control unit 50. In other words, the functions of the control unit 50 may be realized by software. The program that realizes the functions of the control unit 50 may be realized in the same way as the program that realizes the functions of the control unit 40, etc.

[0044] The display 51 is a monitor such as an LCD (Liquid Crystal Display). The display 51 displays an image generated by the imaging unit 20. The touch panel 52 accepts operations for inputting information necessary for controlling the endoscope system 1. The touch panel 52 is arranged on the screen of the display 51. By operating the touch panel 52, the user can input to the endoscope system 1 instructions for changing the settings of the endoscope system 1, instructions necessary for operating the endoscope system 1, and the like.

[0045] The operation button 53 accepts various instructions from the user. By pressing the operation button 53, the user can input instructions regarding power or lighting to the endoscope system 1. The communication unit 54 performs wired or wireless communication with the base unit 4 for bending control and the like. The communication unit 54 receives images generated by the imaging unit 20 from the base unit 4. The communication unit 55 performs wired or wireless communication with the external device 11. The external device 11 is a remote control, a keyboard, a mouse, or the like.

[0046] The control unit 50 executes information display processing for displaying 3D shape information of the subject. The 3D shape information relates to the characteristics of the 3D shape of the subject. For example, the 3D shape information indicates the size or characteristic shape of the subject. An overview of the information display processing will be described below.

[0047] The control unit 50 acquires 3D data including first 3D coordinates of three or more points on the subject calculated based on an image of the subject acquired by the endoscope device 10. The three or more points form a point cloud. The first 3D coordinates are defined in a 3D space corresponding to real space. The control unit 50 displays an image of the 3D shape of the subject (a 3D image) on the display 51 based on the 3D data. The user operates the touch panel 52 or the like to specify a position on the 3D image. In the following example, the touch panel 52 is used as a pointing device. The operation button 53 or the external device 11 may also be used as a pointing device. The main unit 5 may have a joystick or the like as a pointing device.

[0048] When the user specifies a position on the 3D image, the touch panel 52 outputs position information indicating the position. The control unit 50 accepts the position information and calculates second 3D coordinates of a point corresponding to the position indicated by the position information. The second 3D coordinates are different from the first 3D coordinates of each of the three or more points included in the 3D data. The control unit 50 generates 3D shape information of the subject based on the second 3D coordinates and displays the 3D shape information on the display 51.

[0049] 2 shows an example of the procedure for information display processing. The operation of the endoscope system 1 will be described with reference to FIG. 2. In the first embodiment, the endoscope system 1 calculates the 3D coordinates of an arbitrary position on the subject by using a ray casting method.

[0050] The control unit 50 acquires 3D data (step S100).

[0051] The control unit 50 executes the following process in step S100. First to third examples will be described below.

[0052] First, a first example will be described. The optical adapter in the first example is a stereo optical adapter with two fields of view. The optical adapter has a first optical system and a second optical system corresponding to the two fields of view. The first optical system and the second optical system form two optical images of the subject on the imaging unit 20. The imaging unit 20 generates a stereo image corresponding to the first optical image and the second optical image. The stereo image includes a pair of two images (a first image and a second image). That is, the stereo image includes an image of the subject viewed from a first viewpoint and an image of the subject viewed from a second viewpoint. The control unit 50 calculates the 3D coordinates of three or more points on the subject by using one or more stereo images generated by the imaging unit 20, and generates 3D data including the 3D coordinates.

[0053] Next, a second example will be described. The optical adapter in the second example is a monocular optical adapter with one field of view. While the optical adapter in the first example forms two optical images of the subject, the optical adapter in the second example forms one optical image of the subject. The imaging unit 20 generates an image corresponding to the optical image formed by the optical adapter. The imaging unit 20 performs imaging from two or more different viewpoints and generates two or more images. The control unit 50 calculates the 3D coordinates of three or more points on the subject by using the two or more images generated by the imaging unit 20, and generates 3D data including the 3D coordinates.

[0054] Next, a third example will be described. The 3D data generated in the first or second example is stored in advance in the non-volatile memory 57. The control unit 50 acquires the 3D data from the non-volatile memory 57. When two or more sets of 3D data are generated in the first or second example, the control unit 50 may align the two or more sets of 3D data and convert the coordinate systems of the two or more sets of 3D data into a common coordinate system. In this way, the control unit 50 may use the two or more sets of 3D data as one large piece of 3D data.

[0055] As described above, the 3D data includes 3D coordinates of three or more points on the object. The 3D data also includes mesh information. A mesh is generated by combining two or more polygons. A polygon represents a basic shape composed of points, lines, or surfaces in 3D space. If the polygon is a triangle and the mesh information includes information on two or more polygons, the 3D data includes 3D coordinates of four or more points.

[0056] After step S100, the control unit 50 acquires texture information corresponding to each polygon (step S101).

[0057] The texture information is information about a texture image in which the part of the subject in each polygon is captured. The texture image is generated by the imaging unit 20 and is a part of the image used to generate the 3D data.

[0058] After step S101, the control unit 50 generates a 3D image by combining the texture image indicated by the texture information with the 3D data (step S102). After step S102, the control unit 50 outputs the 3D image to the display 51 and displays the 3D image on the display 51 (step S103).

[0059] After the 3D image is displayed, the user operates the touch panel 52 to input information indicating a position on the 3D image to the endoscope system 1. The touch panel 52 outputs position information indicating that position to the control unit 50. The control unit 50 receives the position information output from the touch panel 52 (step S104).

[0060] After step S104, the control unit 50 performs projective transformation of the point cloud coordinates. That is, the control unit 50 transforms the 3D coordinates of all three or more points included in the 3D data into two-dimensional (2D) coordinates (viewport coordinates) on a plane (viewport) corresponding to the screen of the display 51 (step S105). Step S105 may be performed before step S104 is performed.

[0061] Step S105 will be described in detail. A matrix storing the 3D coordinates of N points is defined as X. Matrix X has 3 rows and N columns. The view matrix in the current 3D display is defined as Tview, the projection matrix is ​​defined as Tproj, and the viewport transformation matrix is ​​defined as Tviewport. The 2D coordinates projected onto the viewport (homogeneous coordinates) are represented by the x and y components in the top two rows of the following equation (1):

[0062]

[0063] In equation (1), homogeneous(X) represents the homogeneous coordinate matrix of X, and hnormalized(X) in equation (1) represents the normal coordinate matrix corresponding to the homogeneous coordinate matrix X.

[0064] The screen size of the display 51 is defined as (W, H), and the depth range is defined as Dmin to Dmax. Here, the viewport transformation matrix Tviewport is expressed by the following equation (2).

[0065]

[0066] After step S105, the control unit 50 calculates the 3D coordinates of two points on the 3D line corresponding to the position indicated by the position information. The 3D line corresponds to the camera position and the line of sight passing through that position (step S106). The camera position indicates the position of the camera that captured the image used to generate the 3D data. For example, the camera corresponds to the endoscope device 10.

[0067] Step S106 will now be described in detail. When the position S indicated by the position information is defined as S = (Sx, Sy), the control unit 50 calculates the 3D coordinates of two points La and Lb on a 3D line corresponding to the position S. The 3D coordinates of the two points La and Lb are expressed by the following equation (3).

[0068]

[0069] The matrix T in equation (3) is expressed by the following equation (4).

[0070]

[0071] After step S106, the control unit 50 calculates the 3D coordinates corresponding to the position S. The 3D coordinates are different from the 3D coordinates of each of the three or more points included in the 3D data (step S107).

[0072] Details of step S107 will be described. The control unit 50 identifies one or more polygons among all polygons that intersect with the 3D line calculated in step S106. The control unit 50 identifies the polygon among the one or more polygons that is closest to the camera position, and calculates the 3D coordinates of the intersection between the polygon and the 3D line.

[0073] An example of step S107 when the polygon is a triangle will be described below. The control unit 50 determines whether the position S is within each triangle. The control unit 50 performs this determination for all triangles. Specifically, the control unit 50 performs the process described below.

[0074] The 3D coordinates of the three vertices of the triangle are defined as P0, P1, and P2, respectively. The 2D coordinates of the three vertices projected onto a plane corresponding to the screen of the display 51 are defined as Q0, Q1, and Q2. The control unit 50 converts the 2D coordinates of the three vertices into 3D vectors whose Z components are 0 according to the following equation (5), and calculates the cross product of each 3D vector.

[0075]

[0076] The control unit 50 determines whether the position S is on a triangle having vertices Q0, Q1, and Q2. Specifically, when the signs of the Z components of the three cross products calculated in equation (5) are the same, the control unit 50 determines that the position S is on a triangle.

[0077] In a special condition where the three vertices of a triangle are on the same straight line, the control unit 50 may determine that the position S is on the triangle even if the position S is not on the triangle. Therefore, the control unit 50 executes the determination described below.

[0078] When the control unit 50 determines that the position S is on the triangle, the control unit 50 determines whether a 3D line including two points La and Lb overlaps with the triangle. Specifically, the following equation (6) is defined, and when the condition shown in the following equation (7) is satisfied, the control unit 50 determines that the 3D line overlaps with the triangle. The control unit 50 calculates the 3D coordinate of the intersection of the 3D line and the triangle as the 3D coordinate corresponding to the position S. The 3D coordinate is expressed as La + tLab.

[0079]

[0080]

[0081] After step S107, the control unit 50 generates 3D shape information of the subject by using the 3D coordinates calculated in step S107 (step S108). In a first modified example of the first embodiment and a second modified example of the first embodiment described later, specific examples of how the control unit 50 generates 3D shape information will be described.

[0082] After step S108, the control unit 50 displays the 3D shape information on the display 51 (step S109). When step S109 is executed, the information display process shown in FIG.

[0083] The 3D shape information display device according to each aspect of the present invention includes a control unit 50. The control unit 50 acquires 3D data including first 3D coordinates of three or more points on the subject, calculated based on an image of the subject acquired by the endoscope device 10 (endoscope). The control unit 50 displays a 3D image on a display 51 based on the 3D data. The 3D image is an image of the 3D shape of the subject including points having first 3D coordinates. The control unit 50 receives position information output from a pointing device (such as a touch panel 52). The position information indicates a position on the 3D image. The control unit 50 calculates second 3D coordinates of one or more points based on line-of-sight information (a 3D line) at a position indicated by the position information and polygon information at that position. The second 3D coordinates are different from the first 3D coordinates of each of the three or more points. The polygon information indicates a shape generated based on the three or more points. The control unit 50 generates 3D shape information of the subject based on the second 3D coordinates of the one or more points, and displays the 3D shape information on the display 51.

[0084] A 3D shape information display method according to each aspect of the present invention includes first to sixth steps. In a first step (step S100), the control unit 50 acquires 3D data including first 3D coordinates of three or more points on the subject calculated based on an image of the subject acquired by the endoscope device 10 (endoscope). In a second step (step S103), the control unit 50 displays a 3D image on the display 51 based on the 3D data. In a third step (step S104), the control unit 50 accepts position information output from a pointing device (such as a touch panel 52). In a fourth step (step S107), the control unit 50 calculates second 3D coordinates of one or more points based on line-of-sight information (a 3D line) at a position indicated by the position information and polygon information at that position. In a fifth step (step S108), the control unit 50 generates 3D shape information of the subject based on the second 3D coordinates of one or more points, and in a sixth step (step S109), displays the 3D shape information on the display 51.

[0085] The program according to each aspect of the present invention causes a computer to execute the first to sixth steps described above.

[0086] Each aspect of the present invention may include the following modifications: The 3D data includes polygon information.

[0087] Each aspect of the present invention may include the following modifications: The line of sight information indicates a 3D line including two or more points corresponding to the positions indicated by the position information, and the control unit 50 calculates second 3D coordinates of the intersection between the 3D line and the shape indicated by the polygon information.

[0088] Each aspect of the present invention may include the following modifications: The polygon information indicates two or more figures, and the intersection point is an intersection point between a 3D line and a figure that is closest to the camera position among the two or more figures.

[0089] In the first embodiment, the control unit 50 calculates second 3D coordinates based on line-of-sight information at a position indicated by the position information and polygon information at that position. The control unit 50 also generates 3D shape information of the subject based on the second 3D coordinates and displays the 3D shape information on the display 51. The control unit 50 can generate the 3D shape information without being bound by the first 3D coordinates included in the 3D data. Therefore, the endoscope system 1 can present detailed 3D shape information of the subject. In the first embodiment, detailed 3D shape information of the subject can be presented even when the number of vertices and polygons is reduced to handle a wide range of 3D data, thereby reducing the size of the 3D data and the computational load.

[0090] (First Modification of First Embodiment) A first modification of the first embodiment of the present invention will be described. In the first modification of the first embodiment, the control unit 50 generates 3D shape information indicating a characteristic shape of the subject. Specifically, the control unit 50 generates 3D shape information indicating a cross section of the subject.

[0091] After the 3D image is displayed on the display 51, the control unit 50 accepts the designation of one or more reference points on the subject. In the following example, the control unit 50 accepts the designation of two reference points.

[0092] The control unit 50 sets a reference plane (cutting reference) in the 3D space based on one or more reference points. Specifically, the control unit 50 sets a reference figure based on the one or more reference points, and sets a reference plane based on the reference figure. The reference figure is a line, a plane, or a curved surface. An example in which a boundary line is used as the reference figure will be described below.

[0093] The control unit 50 divides the entire area of ​​the subject into two areas by the reference plane. The cross section of the subject is formed at the intersection of the reference plane and the subject. The control unit 50 changes the display state of one of the two areas. In the first embodiment, the control unit 50 hides one of the two areas. As a result, the cross section of the subject becomes visible.

[0094] 3 shows an example of the procedure of information display processing. The operation of the endoscope system 1 will be described with reference to FIG. 2. Processing that is the same as the processing shown in FIG. 2 will not be described.

[0095] In step S103, the control unit 50 displays a 3D image on the display 51. FIG. 4 shows an example of a 3D image displayed on the display 51 in step S103. The control unit 50 displays the 3D image IMG10 shown in FIG. 4 on the display 51. The 3D image IMG10 includes a 3D shape SH10 of the subject. The control unit 50 displays an icon IC10 on the 3D image IMG10. The icon IC10 includes an illustration of the subject and prompts the user to set a first reference point. The icon IC10 also notifies the user of areas that will be hidden. For example, the icon IC10 notifies the user that an area in front of a reference plane that passes through two reference points will be hidden.

[0096] The control unit 50 displays a cursor CS10 on the 3D image IMG10. The user moves the cursor CS10 on the 3D image IMG10 by operating the touch panel 52. The user may also move the cursor CS10 on the 3D image IMG10 by operating the operation button 53, the external device 11, or the like.

[0097] The user operates the touch panel 52 or the like to input an instruction to set a first reference point to the endoscope system 1. The control unit 50 determines whether or not the instruction has been input (step S110). If the control unit 50 determines in step S110 that the instruction has not been input, the control unit 50 repeats step S110.

[0098] When the control unit 50 determines in step S110 that the instruction has been input, the control unit 50 accepts the instruction and sets a first reference point at the position of the cursor CS10. The control unit 50 also executes steps S105 to S107 shown in FIG. 2 to calculate the 3D coordinates of the first reference point (step S111). Information about the first reference point is stored in the volatile memory 56.

[0099] 5 shows an example of a 3D image displayed on the display 51 when the first reference point is set in step S111. Portions that are the same as those shown in FIG.

[0100] The control unit 50 displays an icon IC11 on the 3D image IMG10. The icon IC11 includes an illustration of the subject and prompts the user to set the second reference point. The icon IC11 also notifies the user of the area that will be hidden.

[0101] The user operates the touch panel 52 or the like to move the cursor CS10 on the 3D image IMG10. The user operates the touch panel 52 or the like to input an instruction to set a second reference point to the endoscope system 1. The control unit 50 determines whether or not the instruction has been input (step S112).

[0102] If the control unit 50 determines in step S112 that the instruction has not been input, the control unit 50 highlights the line connecting the first reference point and the point on the cursor on the 3D image (step S117). After step S117, step S112 is executed.

[0103] 6 shows an example of a 3D image displayed on the display 51 in step S117. Portions that are the same as those shown in FIG.

[0104] The control unit 50 displays a line L10 on the 3D image IMG10. The line L10 is a straight line connecting the first reference point set in step S111 and the point indicated by the cursor CS10. The line L10 is displayed as a thick line. When the user moves the cursor CS10, the line L10 moves.

[0105] When the control unit 50 determines in step S112 that an instruction to set a second reference point has been input, the control unit 50 accepts the instruction and sets the second reference point at the position of the cursor CS10. The control unit 50 also executes steps S106 and S107 shown in FIG. 2 to calculate the 3D coordinates of the second reference point (step S113). The second reference point is different from the first reference point. Information about the second reference point is stored in the volatile memory 56.

[0106] After step S113, the control unit 50 sets a reference plane that passes through the first reference point and the second reference point (step S114).

[0107] Step S114 will now be described in detail. Fig. 7 shows a 3D space defined in 3D data. The 3D data includes 3D coordinates of three or more points constituting the 3D shape SH11 of the subject. The 3D coordinates of the three or more points are composed of coordinate values ​​on the mutually orthogonal X-axis, Y-axis, and Z-axis.

[0108] The control unit 50 sets a first reference point RP10 in step S111, and sets a second reference point RP11 in step S113. The control unit 50 sets a boundary line BL10 that passes through the first reference point RP10 and the second reference point RP11 in step S114. The control unit 50 also sets a plane PL10 that includes the boundary line BL10 and is parallel to the Z axis in step S114. The plane PL10 is a reference plane.

[0109] After step S114, the control unit 50 divides the entire region of the 3D data into a first region and a second region (step S115). Dividing the entire region into two regions means defining two regions within the entire region. In other words, dividing the entire region into two regions means assigning each point included in the entire region to one of the two regions. The boundary between the first region and the second region is the reference plane set in step S114. Each of the three or more points included in the 3D data is included in the first region or the second region.

[0110] After step S115, the control unit 50 changes the display state of the 3D image displayed on the display 51. Specifically, the control unit 50 changes the display state of one of the first region and the second region. For example, the control unit 50 hides one of the first region and the second region. As a result, the control unit 50 displays 3D shape information indicating a cross section of the subject (step S116). When step S116 is executed, the information display process shown in FIG. 3 ends.

[0111] Instead of hiding one of the first region and the second region, the control unit 50 may increase the transparency of one of the first region and the second region. For example, before step S116 is executed, the transparency of the first region and the second region is 0%. In step S116, the control unit 50 sets the transparency of one of the first region and the second region to a value greater than 0% and less than or equal to 100%. The control unit 50 may also set the transparency of one of the first region and the second region to 50% or more.

[0112] When the first region or the second region is hidden, the cross section hidden by the first region or the second region is displayed. When the transparency of one of the first region or the second region is increased, the cross section hidden by the first region or the second region becomes visible. This allows the user to easily understand the 3D shape of the cross section. Since the 3D shape of the cross section as well as the 3D shape of the surrounding area of ​​the cross section are displayed, the user can easily intuitively understand the 3D shape of the subject.

[0113] 8 shows an example of a 3D image displayed on the display 51 in step S116. The same parts as those shown in FIG. 6 will not be described.

[0114] For example, the first region is located on the positive side of the Y axis from the reference plane set in step S114. For example, the second region is located on the negative side of the Y axis from the reference plane. The control unit 50 hides one or more points included in the second region. As a result, the cross section of the 3D shape SH10 is visualized. The control unit 50 also displays a line L11 on the 3D image IMG10 to highlight the cross section. For example, the line L11 is displayed with a specific thickness and a specific color.

[0115] The control unit 50 may set a reference plane based on three reference points. For example, the control unit 50 sets a plane passing through the first reference point, the second reference point, and the origin of the 3D data as the reference plane. The origin of the 3D data corresponds to the camera position when the imaging unit 20 generated the image. The origin of the 3D data may coincide with the center of the tip of the insertion unit 2. In this method, the reference plane corresponds to a straight line on a distortion-corrected image generated by correcting optical distortion of the image generated by the imaging unit 20. Therefore, the user can easily understand the position through which the reference plane passes on the distortion-corrected image.

[0116] The control unit 50 may set the reference plane based on one reference point. For example, the control unit 50 sets, as the reference plane, a plane that passes through the first reference point and is perpendicular to the line of sight from the origin of the 3D view camera to the first reference point. The origin of the 3D view camera corresponds to the viewpoint of the perspective projection transformation when a 3D image is generated from 3D data and displayed on the display 51.

[0117] Each aspect of the present invention may include the following modifications: The control unit 50 sets a reference plane based on second 3D coordinates of one or more points, and divides the 3D data into a first region and a second region using the reference plane as a boundary. The control unit 50 generates 3D shape information indicating a cross section of the subject set based on the reference plane.

[0118] In the first modification of the first embodiment, the control unit 50 generates 3D shape information indicating a cross section of the subject. The endoscope system 1 can display a detailed 3D shape of the cross section of the subject.

[0119] (Second Modification of First Embodiment) A second modification of the first embodiment of the present invention will be described. In the second modification of the first embodiment, the control unit 50 generates 3D shape information indicating the size of the subject.

[0120] A monocular optical adapter used for normal observation is attached to the tip of the insertion section 2. The imaging section 20 generates two or more images based on an optical image formed through the monocular optical adapter. The monocular optical adapter and the imaging section 20 form a monocular camera with one field of view.

[0121] At least a portion of the surface of the object is curved. For example, the object is a pipe having a cylindrical surface. The control unit 50 measures the size of the object by using the 3D data in the information display process.

[0122] 9 shows an example of the procedure of information display processing. The operation of the endoscope system 1 will be described with reference to FIG. 9. Processing that is the same as the processing shown in FIG. 2 will not be described.

[0123] In step S100, the control unit 50 performs 3D reconstruction processing using two or more images to generate 3D data. For example, the control unit 50 generates 3D data using the method disclosed in Japanese Patent Application Laid-Open No. 2020-12635.

[0124] The 3D data includes 3D coordinates of three or more points (3D point cloud) of the subject, camera coordinates, and posture information. The 3D data may also include meshes, which are surfaces having the 3D point cloud as vertices, and mesh polygon data, which is a collection of texture information associated with the meshes.

[0125] The 3D coordinates are defined in a 3D space corresponding to the real space. The camera coordinates indicate the 3D coordinates of the camera that captured each of the two or more images, and are associated with each of the two or more images. The camera coordinates are the 3D coordinates of the viewpoint when each image was captured, and indicate the position of the camera. For example, the camera coordinates indicate the 3D coordinates of the observation optical system of the camera. The attitude information indicates the attitude of the camera that captured each of the two or more images, and is associated with each of the two or more images. For example, the attitude information indicates the attitude of the observation optical system of the camera.

[0126] 3D data indicates a relative shape without the dimension of length. Although the 3D data includes the 3D coordinates of each point, the relative distance calculated using the 3D coordinates is different from the absolute size of the object.

[0127] The 3D data generated in step S100 is stored in the non-volatile memory 57. The control unit 50 may obtain from the non-volatile memory 57 3D data generated in an examination that was previously performed.

[0128] After step S103, the control unit 50 sets a reference plane in the 3D space based on three or more reference points (step S120).

[0129] 10 shows an example of the procedure of the process executed in step S120. The operation of the endoscope system 1 will be described with reference to FIG.

[0130] The control unit 50 sets a variable n for managing the number of reference points to 1 (step S200).

[0131] The user operates the touch panel 52 to set a reference point in the 3D image displayed on the display 51, and inputs the reference point to the endoscope system 1. For example, the user touches the position of the reference point. The control unit 50 accepts the reference point input by the user and sets the reference point on the 3D image. The control unit 50 also executes steps S105 to S107 shown in FIG. 2 to calculate the 3D coordinates of the reference point (step S201). If step S105 has already been executed, execution of step S105 may be omitted. The information about the reference point is stored in the volatile memory 56. The reference point set in step S201 is treated as the nth reference point.

[0132] After step S201, the control unit 50 selects an area near the nth reference point in the 3D image displayed on the display 51 (step S202). For example, the control unit 50 selects a spherical area centered on the reference point in step S202. After step S202, the control unit 50 displays the area selected in step S202 on the 3D image (step S203).

[0133] After step S203, the control unit 50 estimates a curved surface that approximates the surface of the 3D shape of the subject by using the 3D coordinates of the points included in the region selected in step S202 (step S204), and displays the curved surface on the display 51 (step S205).

[0134] After step S205, the control unit 50 determines whether the curved surface has been accurately estimated (step S206).

[0135] Step S206 will now be described in detail. For example, the user checks the curved surface displayed on the display 51. The user determines the degree of coincidence between the surface of the subject and the curved surface, and inputs information indicating the determination result to the endoscope system 1. The control unit 50 determines whether the curved surface has been accurately estimated based on the information input by the user.

[0136] When the control unit 50 determines in step S206 that the curved surface has been accurately estimated, the control unit 50 sets the curved surface as a reference surface (step S207). Information about the reference surface is stored in the volatile memory 56. When step S207 is executed, the process shown in Fig. 10, i.e., step S120 shown in Fig. 9, ends.

[0137] If the control unit 50 determines in step S206 that the curved surface has not been accurately estimated, the control unit 50 increments the variable n by 1 (step S208). After step S208, step S201 is executed. After step S208, the control unit 50 may change the position of the reference point that has already been set.

[0138] When steps S201 to S204 are executed two or more times, the control unit 50 sets two or more reference points and selects two or more regions. The control unit 50 estimates a curved surface based on the 3D coordinates of three or more points, including one or more points included in each of the two or more regions. The control unit 50 may estimate a curved surface based on the 3D coordinates of four or more points, including two or more points included in each of the two or more regions.

[0139] When step S201 is executed three times and three reference points are set, control unit 50 displays 3D image IMG20 shown in FIG. 11 on display 51.

[0140] The control unit 50 displays regions R1 to R3 corresponding to the three reference points. Region R1 includes the first reference point, region R2 includes the second reference point, and region R3 includes the third reference point. In the example shown in Fig. 11, the control unit 50 estimates the curved surface based on the 3D coordinates of six or more points, including two or more points in each region.

[0141] 9 again, the operation of the endoscope system 1 will be described. After the reference plane is set, the user operates the touch panel 52 or the like to input a known reference length of the 3D shape of the object in the 3D data to the endoscope system 1. The control unit 50 accepts the reference length input by the user and sets the reference length (step S121). The reference length information is stored in the volatile memory 56. For example, if the object is a pipe, the reference length is the radius or diameter of the cylindrical surface that constitutes the pipe.

[0142] After step S121, the control unit 50 displays the reference plane set in step S207 and the reference length set in step S121 on the display 51 (step S122).

[0143] After step S122, the control unit 50 measures the length on the 3D data corresponding to the reference length. For example, if the object is a pipe, the control unit 50 calculates the radius or diameter of the reference surface set in step S120 (step S123).

[0144] After step S123, the control unit 50 calculates the ratio between the reference length set in step S121 and the length calculated in step S123. The control unit 50 uses the ratio as a scale factor to convert the scale of the 3D data (step S124).

[0145] For example, if the reference length is X and the length in the 3D data corresponding to the reference length is Y, the scale factor is X / Y. In step S124, the control unit 50 converts the 3D data into 3D data having an absolute length dimension. After step S124 is executed, the 3D coordinates in the 3D data are converted into absolute 3D coordinates. The 3D data with the converted scale is stored in the non-volatile memory 57.

[0146] After step S124, the control unit 50 measures the size of the subject based on one or more points in the 3D data (step S125). When step S125 is executed, the information display process shown in FIG.

[0147] 12 shows an example of the procedure of the process executed in step S125. The operation of the endoscope system 1 will be described with reference to FIG.

[0148] The control unit 50 refers to the measurement mode stored in the volatile memory 56. The measurement mode indicates curved surface-based measurement, plane-based measurement, or two-point distance measurement. The curved surface-based measurement is a mode for measuring the 3D distance from a measurement point to a point where a straight line that includes the measurement point and is perpendicular to the reference curved surface intersects with the reference curved surface. The plane-based measurement is a mode for measuring the 3D distance from a measurement point to a point where a straight line that includes the measurement point and is perpendicular to the reference plane intersects with the reference plane. The two-point distance measurement is a mode for measuring the 3D distance between two measurement points. The control unit 50 determines whether the measurement mode is curved surface-based measurement (step S300).

[0149] When the control unit 50 determines in step S300 that the measurement mode is curved surface reference measurement, the control unit 50 sets a measurement point. At this time, the control unit 50 executes steps S104 to S107 shown in FIG. 2 to calculate the 3D coordinates of the measurement point (step S301). If step S105 has already been executed, execution of step S105 may be omitted. Information about the measurement point is stored in the volatile memory 56.

[0150] For example, the user operates the touch panel 52 to set a measurement point and inputs the measurement point to the endoscope system 1. For example, the user touches the position of the measurement point on the 3D image displayed on the display 51. The control unit 50 accepts the measurement point input by the user and sets the measurement point on the 3D image.

[0151] After step S301, the control unit 50 calculates the 3D distance from the reference curved surface to the measurement point (step S302). The reference curved surface is the reference surface set in step S207. Because the reference surface for converting the scale of the 3D data is used as the reference curved surface in the curved surface-reference measurement, the endoscope system 1 does not need to calculate a new reference curved surface.

[0152] After step S302, the control unit 50 displays the measurement results on the display 51 (step S303).

[0153] After step S303, the control unit 50 determines whether to continue the measurement (step S304). The control unit 50 may determine to continue the measurement until the user inputs information indicating that the measurement is to be ended into the endoscope system 1. When the user inputs information indicating that the measurement is to be ended into the endoscope system 1, the control unit 50 may determine not to continue the measurement.

[0154] When the control unit 50 determines in step S304 that the measurement is to be continued, step S300 is executed. When the control unit 50 determines in step S304 that the measurement is not to be continued, the process shown in FIG. 12 ends.

[0155] When the control unit 50 determines in step S300 that the measurement mode is not curved surface reference measurement, the control unit 50 determines whether the measurement mode is flat surface reference measurement (step S305).

[0156] When the control unit 50 determines in step S305 that the measurement mode is plane-reference measurement, the control unit 50 sets three reference points (step S306). The three reference points are included in three or more points in the 3D data. Information on the three reference points is stored in the volatile memory 56.

[0157] After step S306, the control unit 50 calculates a reference plane that passes through the three reference points set in step S306 (step S307), and then displays the reference plane on the display 51 (step S308).

[0158] After step S308, the control unit 50 sets measurement points (step S309). Information about the measurement points is stored in the volatile memory 56. The method for setting the measurement points in step S309 is the same as the method for setting the measurement points in step S301.

[0159] After step S309, the control unit 50 calculates the 3D distance from the reference plane to the measurement point (step S310). After step S310, step S303 is executed.

[0160] When the control unit 50 determines in step S305 that the measurement mode is not plane-reference measurement, the control unit 50 sets two measurement points to perform two-point distance measurement (steps S311 and S312). Information about the two measurement points is stored in the volatile memory 56. The method for setting the measurement points in steps S311 and S312 is the same as the method for setting the measurement points in step S301.

[0161] After step S312, the control unit 50 calculates the 3D distance between the two measurement points (step S313). After step S313, step S303 is executed.

[0162] Fig. 13 shows an example of an image displayed on the display 51 in the process shown in Fig. 12. The control unit 50 displays a 3D image IMG20 on the display 51. When the curved surface reference measurement is performed, the control unit 50 sets a measurement point MP1 shown in Fig. 13 on the 3D image IMG20 and displays the measurement point MP1 on the 3D image IMG20. The control unit 50 calculates the 3D distance from the reference curved surface to a point in 3D space corresponding to the measurement point MP1, and displays a measurement result MR1 indicating the 3D distance on the display 51.

[0163] The measurement points in the curved surface reference measurement or the flat surface reference measurement are set, for example, at abnormal parts in the object to be measured. The abnormal parts are recesses or protrusions.

[0164] 14A and 14B show examples of an abnormality occurring in a test specimen. The test specimen SB1 shown in FIGS. 14A and 14B is a pipe. The test specimen SB1 has an abnormality AP1. The abnormality AP1 is a convex portion occurring on the inner surface of the test specimen SB1. The control unit 50 sets a reference plane that approximates the inner surface of the test specimen SB1, and also sets a measurement point MP3. For example, the measurement point MP3 is the highest point of the abnormality AP1. The control unit 50 calculates the distance D1 between the reference plane and the measurement point MP3.

[0165] As described above, the control unit 50 estimates the curved surface of the object and sets a reference length on that curved surface. The control unit 50 calculates a length on the 3D data that corresponds to the reference length, and converts the scale of the 3D data based on that length and the reference length. The control unit 50 performs measurement by using the 3D data with the converted scale. The length corresponding to the size of the curved surface in the 3D data (e.g., the radius or diameter of a cylindrical surface) is reflected in the scale of the 3D data. In other words, the scale of the 3D data is converted according to the shape of the object. This improves the accuracy of the scale conversion and the accuracy of the measurement results.

[0166] The curved surface of the object may be a cylindrical surface, a spherical surface, a torus surface, an elliptical surface, a paraboloid surface, a free-form surface, etc. The object does not need to have the entirety of such a curved surface. For example, the object may have a part of a spherical surface.

[0167] The control unit 50 may use 3D data including 3D coordinates calculated by using stereo images. In this case, the 3D data has a dimension of length. Therefore, the control unit 50 does not need to convert the scale of the 3D data.

[0168] Each aspect of the present invention may include the following modifications: The control unit 50 generates 3D shape information indicating the size of the subject based on the second 3D coordinates of the one or more points.

[0169] Each aspect of the present invention may include the following modifications: The control unit 50 calculates second 3D coordinates of four or more points. The control unit 50 sets a reference plane based on the second 3D coordinates of three or more of the four or more points. The control unit 50 generates 3D shape information indicating the distance between one of the four or more points and the reference plane.

[0170] Each aspect of the present invention may include the following modifications: The control unit 50 calculates second 3D coordinates of two points, and generates 3D shape information indicating the distance between the two points.

[0171] In the second modification of the first embodiment, the control unit 50 generates 3D shape information indicating the size of the subject. The endoscope system 1 can acquire detailed size information of the subject.

[0172] (Third Modification of First Embodiment) A third modification of the first embodiment of the present invention will be described. In the third modification of the first embodiment, the endoscope system 1 calculates the 3D coordinates of an arbitrary position on the subject by using a depth buffer method.

[0173] 15 shows an example of the procedure for information display processing. The operation of the endoscope system 1 will be described using Fig. 15. Processing that is the same as the processing shown in Fig. 2 will not be described.

[0174] After step S103, step S105 is executed. After step S105, the control unit 50 determines whether each polygon projected onto the plane corresponding to the screen of the display 51 covers each pixel of the 3D image displayed on the display (step S130).

[0175] 16 shows an example of the relationship between polygons and pixels of a 3D image. Polygons POL1 to POL6 are shown. Polygon POL1 covers pixel PIX1. On the other hand, polygons POL2 to POL6 do not cover pixel PIX1.

[0176] After step S130, the control unit 50 calculates the depth value of each pixel by using the information on the vertices of the polygon that covers each pixel, and stores the calculated depth value in a buffer (volatile memory 56) (step S131).

[0177] Step S131 will be described in detail. The control unit 50 performs projection transformation and calculates the depth value "depth" by using the following equation (8): depth=(w / (x*px+y*py+z*pz+w)) (8)

[0178] In equation (8), w is the W component of the homogeneous coordinates of each vertex of the triangles that make up each polygon, and is basically 1. In equation (8), x, y, and z are the 3D coordinates of each vertex in the world coordinate system. In equation (8), px, py, and pz represent the components of the viewpoint vector.

[0179] The control unit 50 calculates the depth value of each vertex according to equation (8). The control unit 50 interpolates the depth value of a pixel included in a triangle by using the depth values ​​of the three vertices of the triangle. If two or more polygons cover a pixel, the control unit 50 calculates the depth value of the pixel for each polygon according to equation (8). The control unit 50 stores the depth value corresponding to the shortest distance among the two or more depth values ​​in a buffer.

[0180] After step S131, in step S104, the control unit 50 receives position information output from the touch panel 52. After step S104, the control unit 50 obtains a depth value in 2D coordinates of the position indicated by the position information from the buffer (step S132).

[0181] After step S132, the control unit 50 converts the 2D coordinates in the camera coordinate system into 3D coordinates in the world coordinate system by using the depth values ​​acquired in step S132 (step S133). After step S133, step S108 is executed.

[0182] Step S133 will now be described in detail. The control unit 50 calculates the 3D coordinate P corresponding to the position indicated by the position information by using the following equation (9).

[0183]

[0184] In equation (9), (Sx, Sy) indicates the 2D coordinates of the position indicated by the position information and corresponds to the line of sight. D in equation (9) indicates a depth value. The matrix T in equation (9) is expressed by equation (4) described above.

[0185] Each aspect of the present invention may include the following modifications: The line-of-sight information indicates 2D coordinates of a position on a plane corresponding to the screen of the display 51. The control unit 50 calculates second 3D coordinates based on the 2D coordinates and a 2D figure obtained by projecting onto the plane the figure indicated by the polygon information.

[0186] In a third modification of the first embodiment, the control unit 50 calculates second 3D coordinates by using depth values ​​of the positions indicated by the position information. The control unit 50 also generates 3D shape information of the subject based on the second 3D coordinates and displays the 3D shape information on the display 51. The control unit 50 can generate the 3D shape information without being bound by the first 3D coordinates included in the 3D data. Therefore, the endoscope system 1 can present detailed 3D shape information of the subject.

[0187] (Fourth Modification of First Embodiment) A fourth modification of the first embodiment of the present invention will be described. In the fourth modification of the first embodiment, the endoscope system 1 displays only points included in the 3D data and calculates the 3D coordinates of any position on the subject.

[0188] 17 shows an example of the procedure for information display processing. The operation of the endoscope system 1 will be described using FIG. 17. Processing that is the same as the processing shown in FIG. 2 will not be described.

[0189] The 3D data includes mesh information and polygon information, or alternatively, the 3D data does not include mesh information or polygon information.

[0190] After step S100, the control unit 50 displays all of the three or more points included in the 3D data on the display 51. That is, the control unit 50 displays the entire point cloud as a 3D image on the display 51 (step S140). After step S140, the control unit 50 receives position information output from the touch panel 52 in step S104.

[0191] In step S106, the control unit 50 calculates the 3D coordinates of two points on the 3D line corresponding to the position indicated by the position information. After step S106, the control unit 50 identifies a point that is closest to the 3D line and closest to the camera position among the three or more points included in the 3D data (step S141).

[0192] After step S141, the control unit 50 acquires information about a polygon that includes the point identified in step S141 from the 3D data. When the 3D data does not include polygon information, the control unit 50 generates a polygon and a mesh by using a point cloud surrounding the point identified in step S141. For example, the control unit 50 identifies three or more points in the 3D data that are within a predetermined distance from the point identified in step S141. The control unit 50 executes an algorithm for creating a mesh from the point cloud using the identified points, such as Delaunay triangulation, to generate a polygon and a mesh (step S142).

[0193] After step S142, the control unit 50 executes steps S105 to S107 shown in Fig. 2 to calculate the 3D coordinates of the intersection between the 3D line and the polygon (step S143). After step S143, step S108 is executed.

[0194] Each aspect of the present invention may include the following modifications: After the position information is received, the control unit 50 generates polygon information.

[0195] In the fourth modification of the first embodiment, even when a 3D image of only a point cloud is displayed, the endoscope system 1 can present detailed 3D shape information of the subject.

[0196] (Fifth Modification of First Embodiment) A fifth modification of the first embodiment of the present invention will be described. Fig. 18 shows an example of the configuration of an endoscope system 1a according to the fifth modification of the first embodiment. Portions that are the same as those shown in Fig. 1 will not be described.

[0197] 18 includes an insertion portion 2 and a main unit 6. The insertion portion 2 and the main unit 6 constitute an endoscope device 10a.

[0198] The insertion section 2 shown in Fig. 18 is the same as the insertion section 2 shown in Fig. 1. The main unit 6 has an imaging drive circuit 30, an image processing section 31, a UD drive section 32, an RL drive section 33, a bending control section 34, a light source 35, a light source control section 36, a display 51, a touch panel 52, operation buttons 53, a communication section 55, a volatile memory 56, a non-volatile memory 57, and a control section 60. The same reference symbols as those shown in Fig. 1 are assigned to blocks that are the same as those shown in Fig. 1.

[0199] The control unit 60 has both the functions of the control unit 40 shown in Fig. 1 and the functions of the control unit 50 shown in Fig. 1. The control unit 60 executes the processing shown in Fig. 2, Fig. 3, Fig. 9, Fig. 15, or Fig. 17.

[0200] In the fifth modification of the first embodiment, similarly to the first embodiment, the endoscope system 1 can present detailed 3D shape information of the subject.

[0201] (Sixth Modification of First Embodiment) A sixth modification of the first embodiment of the present invention will be described. Fig. 19 shows an example of the configuration of an endoscope system 1b according to the sixth modification of the first embodiment. Portions that are the same as those shown in Fig. 1 will not be described.

[0202] 19 includes an insertion portion 2, a scope unit 3b, and a base unit 7. The insertion portion 2 and the scope unit 3b constitute an endoscope device 10b. The scope unit 3b and the base unit 7 are connected by a cable 8.

[0203] The insertion section 2 shown in Fig. 19 is the same as the insertion section 2 shown in Fig. 1. The scope unit 3b shown in Fig. 19 is the same as the scope unit 3 shown in Fig. 1 except that it does not have the image processing section 31. The base unit 7 has the image processing section 31, a display 51, a touch panel 52, operation buttons 53, a communication section 55, a volatile memory 56, a non-volatile memory 57, and a control section 70. The same reference symbols as those in Fig. 1 are assigned to blocks that are the same as those in Fig. 1.

[0204] The control unit 70 has both the functions of the control unit 40 shown in Fig. 1 and the functions of the control unit 50 shown in Fig. 1. The control unit 70 executes the processing shown in Fig. 2, Fig. 3, Fig. 9, Fig. 15, or Fig. 17.

[0205] In the sixth modification of the first embodiment, similarly to the first embodiment, the endoscope system 1 can present detailed 3D shape information of the subject.

[0206] (Various Modifications of the First Embodiment) Various modifications of the first embodiment of the present invention will be described. These various modifications are applicable to the first embodiment and the first to sixth modifications of the first embodiment. These various modifications are also applicable to the second and third embodiments described below.

[0207] The control unit 50 may display both the sides and vertices of each polygon, or may hide at least one of the sides and vertices of each polygon. When at least one of the sides and vertices of each polygon is not displayed, the user can easily confirm whether or not a position on the surface of the polygon has been specified.

[0208] The control unit 50 may change the resolution or sharpness of the texture depending on the magnification ratio for displaying the 3D image. For example, when a 3D image is magnified, the texture becomes blurred. Therefore, the control unit 50 may increase the resolution or sharpness of the texture. The user can easily specify a position on the 3D image.

[0209] As described above, the control unit 50 calculates the 3D coordinates of the intersection between the polygon and the 3D line in step S107. If there is no polygon intersecting with the 3D line, the control unit 50 may display a warning on the display 51 indicating that the intersection could not be detected.

[0210] When the transparency of the message is set, the control unit 50 may identify the foremost polygon or the backmost polygon in step S107. Alternatively, the control unit 50 may identify the foremost polygon and the backmost polygon.

[0211] The control unit 50 may display information indicating the line of sight along with the 3D image. Fig. 20 shows an example of a 3D image displayed on the display 51. The control unit 50 displays a 3D image IMG30 and a 3D image IMG31 on the display 51. The 3D image IMG30 is a 3D image of the subject viewed from a first viewpoint. The 3D image IMG31 is a 3D image of the subject viewed from a second viewpoint different from the first viewpoint.

[0212] The user specifies a position PO1 on the 3D image IMG30. At this time, the control unit 50 displays an arrow AR1 indicating the line of sight on the 3D image IMG31. The direction indicated by the arrow AR1 is parallel to the 3D line calculated in step S106. In other words, the control unit 50 displays the arrow AR1 indicating the direction of the 3D line on the 3D image IMG31. The user can confirm the line of sight direction.

[0213] The user uses a pointing device to specify a position on the 3D image. For example, a crosshair (pointer) is displayed on the display 51, and the user moves the crosshair by using the pointing device. When the user performs an operation such as clicking, position information indicating the position of the crosshair is output. The control unit 50 may control the speed or distance of movement of the crosshair according to the size of the polygon, the number of polygons, the number of points included in the 3D data, the density of points included in the 3D data, or the like.

[0214] For example, when the 3D data includes points spaced widely apart and at low density, the user may specify two or more widely spaced points, and therefore the control unit 50 may increase the speed or distance of the movement of the aim.

[0215] Each aspect of the present invention may include the following modifications: The control unit 50 displays information indicating a 3D line on the display 51 .

[0216] Second Embodiment A second embodiment of the present invention will be described. In the second embodiment, the endoscope system 1 shown in Fig. 1 is used. An endoscope system 1a shown in Fig. 18 or an endoscope system 1b shown in Fig. 19 may also be used.

[0217] In the second embodiment, when a vertex or edge of a polygon with a high priority exists within a predetermined distance from a position specified by a user on a 3D image, the control unit 50 adsorbs the point at the specified position to that vertex or edge.

[0218] 21 shows a method for attaching a point. A polygon POL10 and a polygon POL11 exist, and a point PT1 at a position designated by the user is within the polygon POL10.

[0219] In the first example, when the priority of the vertex VT1 is high, the control unit 50 adsorbs the point PT1 to the vertex VT1. The vertex VT1 is a vertex of the polygon POL10 that covers the point PT1.

[0220] In the second example, when the priority of the vertex VT2 is high, the control unit 50 attaches the point PT1 to the vertex VT2. The vertex VT2 is a vertex of a polygon POL11 that is adjacent to the polygon POL10 that covers the point PT1.

[0221] In the third example, when the priority of the side SD1 is high, the control unit 50 adsorbs the point PT1 to the side SD1. The side SD1 is a side of the polygon POL10 that covers the point PT1.

[0222] The control unit 50 sets the priority of the vertices or edges of a polygon according to the position or size of the polygon. For example, when a vertex or edge of a polygon is on the boundary of the 3D shape of the subject, the control unit 50 sets a high priority to that vertex or edge. For example, the control unit 50 checks the number of faces connected to each edge of each polygon. When only one face is connected to an edge, the control unit 50 can determine that the edge is on the boundary. Alternatively, the control unit 50 sets a high priority to the vertices or edges of larger polygons and a low priority to the vertices or edges of smaller polygons.

[0223] Alternatively, the control unit 50 calculates the priority of a polygon based on texture information associated with the polygon. For example, the control unit 50 calculates the maximum edge strength of the texture corresponding to each polygon. If the edge strength is greater than a reference value, the control unit 50 assigns a high priority to the polygon. Alternatively, the control unit 50 applies a feature extraction method such as FAST (Features from Accelerated Segment Test) or ORB (Oriented FAST and Rotated BRIEF) to the texture corresponding to each polygon. If the extracted feature amount is greater than a reference value, the control unit 50 assigns a high priority to the polygon.

[0224] When a point is specified on a reduced 3D image, it is difficult to specify the precise position of the point. In such cases, it may be possible to make point specification more efficient by attaching the point specified by the user to another point.

[0225] The first mode or the second mode is set in the endoscope system 1, and the endoscope system 1 operates in the first mode or the second mode. The first mode and the second mode are switchable. For example, the control unit 50 switches the mode of the endoscope system 1 based on information output from the touch panel 52 or the like. The control unit 50 may determine the photographing scene and may switch the mode of the endoscope system 1 depending on the photographing scene.

[0226] When the first mode is set in the endoscope system 1, the control unit 50 uses the 3D coordinates calculated by the method described in the first embodiment. When the second mode is set in the endoscope system 1, the control unit 50 adsorbs a point corresponding to a position on the 3D image to another point.

[0227] The control unit 50 may switch between the first mode and the second mode depending on the magnification rate of the 3D image displayed on the display 51. For example, when the magnification rate is high, the control unit 50 may set the mode of the endoscope system 1 to the first mode. When the magnification rate is low, the control unit 50 may set the mode of the endoscope system 1 to the second mode.

[0228] 22 and 23 show an example of the procedure of information display processing. The operation of the endoscope system 1 will be described using Fig. 22 and 23. Processing that is the same as the processing shown in Fig. 2 will not be described.

[0229] After step S107, the control unit 50 determines whether the point adsorption function is enabled (step S150). When the first mode is set to the endoscope system 1, the point adsorption function is disabled. When the second mode is set to the endoscope system 1, the point adsorption function is enabled.

[0230] When the control unit 50 determines in step S150 that the point adsorption function is not enabled, step S108 is executed. When the control unit 50 determines in step S150 that the point adsorption function is enabled, the control unit 50 refers to setting information indicating that a point (specified point) corresponding to a position on the 3D image is to be adsorbed to a vertex or edge of a polygon. The setting information is stored in advance in the volatile memory 56. The control unit 50 determines whether or not to adsorb the specified point to a vertex of a polygon (step S151).

[0231] When the setting information indicates that the designated point should be attached to a vertex of a polygon, the control unit 50 determines that the designated point should be attached to a vertex of the polygon. When the setting information indicates that the designated point should be attached to an edge of the polygon, the control unit 50 determines that the designated point should not be attached to a vertex of the polygon.

[0232] When the control unit 50 determines in step S151 that the specified point should be attached to a vertex of a polygon, the control unit 50 determines whether a vertex of a polygon with a high priority exists within a predetermined distance from the specified point (step S152). If a vertex of a polygon with a high priority does not exist within the predetermined distance from the specified point, step S108 is executed. At this time, the 3D coordinates (second 3D coordinates) calculated in step S107 are used.

[0233] If a vertex of a polygon with a high priority exists within a predetermined distance from the specified point, the control unit 50 attaches the specified point to that vertex and obtains the 3D coordinates of that vertex from the 3D data. If two or more vertices have the same priority, the control unit 50 attaches the specified point to the vertex closest to the specified point (step S153). After step S153, step S108 is executed. At this time, the 3D coordinates obtained in step S153 (third 3D coordinates) are used.

[0234] When the control unit 50 determines in step S151 that the specified point should not be attached to a polygon vertex, the control unit 50 determines whether or not a side of a polygon with a high priority exists within a predetermined distance from the specified point (step S154). When no side of a polygon with a high priority exists within the predetermined distance from the specified point, step S108 is executed. At this time, the 3D coordinates (second 3D coordinates) calculated in step S107 are used.

[0235] When a polygon edge with a high priority exists within a predetermined distance from the designated point, the control unit 50 attaches the designated point to a point on that edge. When two or more edges have the same priority, the control unit 50 attaches the designated point to the edge closest to the designated point. The control unit 50 calculates the 3D coordinates of the point corresponding to the designated point by using the 3D coordinates of the two vertices on that edge (step S155). After step S155, step S108 is executed. At this time, the 3D coordinates calculated in step S155 (third 3D coordinates) are used.

[0236] Step S155 will be described in detail. The 3D coordinates of two vertices on the side are defined as P0 and P1, and the 2D coordinates of the two vertices projected onto a plane corresponding to the screen of the display 51 are defined as Q0 and Q1. The control unit 50 calculates the distance D from the specified point S to the point closest to the specified point S by using the following equations (10), (11), and (12). The point closest to the specified point S is on the line segment Q0Q1.

[0237]

[0238]

[0239]

[0240] The control unit 50 selects a side with a high priority whose distance D is within a predetermined distance. The control unit 50 adsorbs the specified point to a point on that side that is closest to the specified point. The control unit 50 executes steps S105 to S107 shown in FIG. 2 to calculate the 3D coordinates of that point.

[0241] The control unit 50 may analyze texture information corresponding to a polygon that includes the designated point. The texture information is associated with the polygon. The control unit 50 may attach the designated point to a point in a characteristic texture. For example, in the example shown in FIG. 24 , the designated point DP1 is included in polygon POL20. The control unit 50 analyzes the texture information corresponding to polygon POL20 and detects point PT2 in the characteristic texture. The control unit 50 attaches the designated point DP1 to point PT2. The control unit 50 executes steps S105 to S107 shown in FIG. 2 to calculate the 3D coordinates of point PT2.

[0242] Each aspect of the present invention may include the following modifications. When a first mode is set, the control unit 50 generates 3D shape information based on second 3D coordinates. When a second mode different from the first mode is set, the control unit 50 generates 3D shape information based on third 3D coordinates calculated based on the second 3D coordinates. The third 3D coordinates are different from the second 3D coordinates.

[0243] Each aspect of the present invention may include the following modifications: The third 3D coordinates are the first 3D coordinates of one point among three or more points included in the 3D data or the 3D coordinates of a point included in a side of a figure indicated by the polygon information.

[0244] Each aspect of the present invention may include the following modifications: The control unit 50 calculates the third 3D coordinates based on texture information of the figure at the position indicated by the position information. The texture information is associated with the polygon information.

[0245] Each aspect of the present invention may include the following modifications. When a first mode is set, the control unit 50 generates 3D shape information based on second 3D coordinates. When a second mode different from the first mode is set, the control unit 50 calculates a second position (point PT2) on the 3D image based on texture information of a figure at a position (specified point DP1) indicated by the position information. The texture information is associated with polygon information. The second position differs from the position indicated by the position information. The control unit 50 calculates third 3D coordinates based on line-of-sight information at the second position and polygon information at the second position. The third 3D coordinates differ from the first 3D coordinates of each of the three or more points. The control unit 50 generates 3D shape information based on the third 3D coordinates.

[0246] In the second embodiment, the control unit 50 adsorbs a point at a position indicated by the position information onto a vertex or an edge of a polygon, etc. The endoscope system 1 can make point specification more efficient.

[0247] Third Embodiment A third embodiment of the present invention will be described. In the third embodiment, the endoscope system 1 shown in Fig. 1 is used. An endoscope system 1a shown in Fig. 18 or an endoscope system 1b shown in Fig. 19 may also be used.

[0248] In the third embodiment, the control unit 50 detects one or more objects in advance based on texture information. For example, the control unit 50 detects the objects by performing segmentation using a convolutional neural network. Alternatively, the control unit 50 detects the objects by performing segmentation using threshold processing or clustering using the k-means method or the like. The control unit 50 may detect the objects based on the 3D shape of the subject indicated by the 3D data.

[0249] First, the touch panel 52 outputs first position information indicating a first position on the 3D image. The control unit 50 identifies an object including the first position indicated by the first position information. Next, the touch panel 52 outputs second position information indicating a second position on the 3D image. The control unit 50 identifies an object including the second position indicated by the second position information. When the object including the second position is the same as the object including the first position, the control unit 50 accepts the second position information. When the object including the second position is different from the object including the first position, the control unit 50 does not accept the second position information.

[0250] If the user specifies three or more points on the 3D image, the control unit 50 performs the same process as described above. When two or more points on the 3D image are all included in the same object, the control unit 50 generates 3D shape information by using the 3D coordinates of the two or more points.

[0251] 25A and 25B show an example of a process for detecting objects. Fig. 25A shows texture information TX1. Fig. 25B shows objects OB1 and OB2 detected from the texture information TX1.

[0252] 26 and 27 show an example of the procedure of information display processing. The operation of the endoscope system 1 will be described using Fig. 26 and 27. Processing that is the same as the processing shown in Fig. 2 will not be described.

[0253] In the following, the user specifies two or more points on the 3D image. For example, when measuring the distance between a reference plane and a measurement point, three or more points must be specified to set the reference plane.

[0254] After step S103, the control unit 50 detects one or more objects based on the texture information synthesized with the 3D image (step S160). After step S160, the control unit 50 receives position information output from the touch panel 52 in step S104. After step S107, the control unit 50 identifies an object corresponding to the position indicated by the position information (step S161). Information about the identified object is stored in the volatile memory 56. Step S161 may be executed between step S104 and step S107.

[0255] The user operates the touch panel 52 to again input information indicating a position on the 3D image to the endoscope system 1. The touch panel 52 outputs the position information indicating that position to the control unit 50. The control unit 50 receives the position information output from the touch panel 52 (step S162).

[0256] After step S162, the control unit 50 identifies an object corresponding to the position indicated by the position information (step S163). After step S163, the control unit 50 determines whether the object identified in step S163 is the same as the object identified in step S161 (step S164).

[0257] If the object identified in step S163 is different from the object identified in step S161, step S162 is executed. At this time, the control unit 50 may display information on the display unit 201 prompting the user to reset the point.

[0258] If the object identified in step S163 is the same as the object identified in step S161, the control unit 50 calculates the 3D coordinates of two points on the 3D line corresponding to the position indicated by the position information (step S165). Step S165 is the same as step S106.

[0259] After step S165, the control unit 50 calculates 3D coordinates corresponding to the position indicated by the position information (step S166). Step S166 is the same as step S107.

[0260] After step S166, the control unit 50 determines whether or not to end the setting of positions on the 3D image (step S167). For example, if the number of required points is set in advance and the number of set points is less than the required number of points, the control unit 50 determines not to end the setting of positions on the 3D image. At this time, step S162 is executed. When the number of set points reaches the required number of points, the control unit 50 determines to end the setting of positions on the 3D image. At this time, step S108 is executed.

[0261] Each aspect of the present invention may include the following modifications. The control unit 50 detects an object in a subject. The control unit 50 receives first position information and second position information as position information. The first position information indicates a first position. The second position information indicates a second position different from the first position. When the object including the first position and the object including the second position are the same, the control unit 50 generates 3D shape information based on second 3D coordinates corresponding to the first position and second 3D coordinates corresponding to the second position.

[0262] Each aspect of the present invention may include the following modifications: The control unit 50 detects an object based on texture information associated with polygon information.

[0263] In the third embodiment, the control unit 50 generates 3D shape information by using the 3D coordinates of two or more points included in the same object. For example, the control unit 50 sets a reference plane based on three or more points on the same object when measuring the distance between the reference plane and a measurement point. The endoscope system 1 can avoid using points set at positions unintended by the user in generating 3D shape information.

[0264] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Addition, omission, substitution, and other modifications of the configuration are possible within the scope of the spirit of the present invention. Furthermore, the present invention is not limited by the above description, but is limited only by the scope of the appended claims.

[0265] According to each embodiment of the present invention, the three-dimensional shape information display device, the three-dimensional shape information display method, and the program can present detailed three-dimensional shape information of a subject.

[0266] REFERENCE SIGNS LIST 1, 1a, 1b Endoscope system 2 Insertion section 2a Tip section 3, 3b Scope unit 4, 7 Base unit 5, 6 Main unit 10, 10a, 10b Endoscope device 11 External device 20 Imaging section 21 Bending section 22 Illumination window 30 Imaging drive circuit 31 Image processing section 32 UD drive section 33 RL drive section 34 Bending control section 35 Light source 36 Light source control section 40, 50, 60, 70 Control section 41, 54, 55 Communication section 42, 56 Volatile memory 43, 57 Non-volatile memory 51 Display 52 Touch panel 53 Operation button

Claims

1. A three-dimensional shape information display device having a control unit, which: acquires three-dimensional data including first three-dimensional coordinates of three or more points on a subject calculated based on an image of the subject acquired by an endoscope; displays a three-dimensional image on a display based on the three-dimensional data, the three-dimensional image being an image of the three-dimensional shape of the subject including points having the first three-dimensional coordinates; accepts position information output from a pointing device, the position information indicating a position on the three-dimensional image; calculates second three-dimensional coordinates of one or more points based on line-of-sight information at the position and polygon information at the position, the second three-dimensional coordinates being different from the first three-dimensional coordinates of each of the three or more points, the polygon information indicating a figure generated based on the three or more points; generates three-dimensional shape information of the subject based on the second three-dimensional coordinates of the one or more points; and displays the three-dimensional shape information on the display.

2. The three-dimensional shape information display device according to claim 1, wherein the control unit sets a reference plane based on the second three-dimensional coordinates of the one or more points, divides the three-dimensional data into a first region and a second region using the reference plane as a boundary, and generates the three-dimensional shape information showing a cross-section of the subject set based on the reference plane.

3. The three-dimensional shape information display device according to claim 1, wherein the control unit generates the three-dimensional shape information indicating the size of the subject based on the second three-dimensional coordinates of the one or more points.

4. The three-dimensional shape information display device described in claim 3, wherein the control unit calculates second three-dimensional coordinates of four or more points including the one or more points, sets a reference plane based on the second three-dimensional coordinates of three or more points among the four or more points, and generates the three-dimensional shape information indicating the distance between one of the four or more points and the reference plane.

5. The three-dimensional shape information display device according to claim 3, wherein the control unit calculates second three-dimensional coordinates of two points as the one or more points, and generates the three-dimensional shape information indicating the distance between the two points.

6. The three-dimensional shape information display device according to claim 1, wherein the three-dimensional data includes the polygon information.

7. The three-dimensional shape information display device according to claim 1, wherein after the position information is received, the control unit generates the polygon information.

8. A three-dimensional shape information display device as described in claim 1, wherein the line of sight information indicates a three-dimensional line including two or more points corresponding to the position, and the control unit calculates the second three-dimensional coordinates of the intersection between the three-dimensional line and the figure indicated by the polygon information.

9. A three-dimensional shape information display device as described in claim 8, wherein the polygon information indicates two or more of the figures, and the intersection is the intersection between the three-dimensional line and the figure closest to the camera position among the two or more figures.

10. The three-dimensional shape information display device according to claim 8, wherein the control unit displays information indicating the three-dimensional line on the display.

11. A three-dimensional shape information display device as described in claim 1, wherein the line of sight information indicates two-dimensional coordinates of the position on a plane corresponding to the screen of the display, and the control unit calculates the second three-dimensional coordinates based on the two-dimensional coordinates and a two-dimensional figure obtained by projecting the figure indicated by the polygon information onto the plane.

12. The three-dimensional shape information display device of claim 1, wherein the control unit generates the three-dimensional shape information based on the second three-dimensional coordinates when a first mode is set, and generates the three-dimensional shape information based on third three-dimensional coordinates calculated based on the second three-dimensional coordinates when a second mode different from the first mode is set, the third three-dimensional coordinates being different from the second three-dimensional coordinates.

13. A three-dimensional shape information display device as described in claim 12, wherein the third three-dimensional coordinates are the first three-dimensional coordinates of one of the three or more points or the three-dimensional coordinates of a point included in the edge of the figure indicated by the polygon information.

14. A three-dimensional shape information display device as described in claim 12, wherein the control unit calculates the third three-dimensional coordinates based on texture information of the figure at the position, and the texture information is associated with the polygon information.

15. The three-dimensional shape information display device of claim 1, wherein the control unit: when a first mode is set, generates the three-dimensional shape information based on the second three-dimensional coordinates; when a second mode different from the first mode is set, calculates a second position on the three-dimensional image based on texture information of the figure at the position, the texture information being associated with the polygon information and the second position being different from the position; calculates third three-dimensional coordinates based on line-of-sight information at the second position and the polygon information at the second position, the third three-dimensional coordinates being different from the first three-dimensional coordinates of each of the three or more points; and generates the three-dimensional shape information based on the third three-dimensional coordinates.

16. The three-dimensional shape information display device of claim 1, wherein the control unit detects an object in the subject, receives first position information and second position information as the position information, the first position information indicates a first position, and the second position information indicates a second position different from the first position, and when the object including the first position and the object including the second position are the same, generates the three-dimensional shape information based on the second three-dimensional coordinates corresponding to the first position and the second three-dimensional coordinates corresponding to the second position.

17. The three-dimensional shape information display device according to claim 16, wherein the control unit detects the object based on texture information associated with the polygon information.

18. A three-dimensional shape information display method, in which a control unit acquires three-dimensional data including first three-dimensional coordinates of three or more points on a subject, the first three-dimensional coordinates being calculated based on an image of the subject acquired by an endoscope; displays a three-dimensional image on a display based on the three-dimensional data, the three-dimensional image being an image of the three-dimensional shape of the subject including points having the first three-dimensional coordinates; accepts position information output from a pointing device, the position information indicating a position on the three-dimensional image; calculates second three-dimensional coordinates of one or more points based on line-of-sight information at the position and polygon information at the position, the second three-dimensional coordinates being different from the first three-dimensional coordinates of each of the three or more points, the polygon information indicating a figure generated based on the three or more points; generates three-dimensional shape information of the subject based on the second three-dimensional coordinates of the one or more points; and displays the three-dimensional shape information on the display.

19. A program for causing a computer to execute the following steps: acquiring three-dimensional data including first three-dimensional coordinates of three or more points on a subject calculated based on an image of the subject acquired by an endoscope; displaying a three-dimensional image on a display based on the three-dimensional data, wherein the three-dimensional image is an image of the three-dimensional shape of the subject including points having the first three-dimensional coordinates; accepting position information output from a pointing device, wherein the position information indicates a position on the three-dimensional image; calculating second three-dimensional coordinates of one or more points based on line-of-sight information at the position and polygon information at the position, wherein the second three-dimensional coordinates are different from the first three-dimensional coordinates of each of the three or more points, and the polygon information indicates a figure generated based on the three or more points; generating three-dimensional shape information of the subject based on the second three-dimensional coordinates of the one or more points; and displaying the three-dimensional shape information on the display.

Images