Measuring device, measuring method, and program

Abstract

The present invention provides a measuring device, a measuring method, and a program that enable efficient measurement. [Solution] The measuring device has a control unit. The control unit acquires three-dimensional data including the three-dimensional coordinates of two or more points on the subject and sets a measurement reference. The control unit detects one or more feature regions on the subject and outputs an image of the three-dimensional shape of the subject to a display. The control unit selects at least one of the one or more feature regions as the measurement target region. The control unit generates distance information indicating the distance between the measurement reference and the measurement point, superimposes the measurement result generated based on the distance information onto the image of the measurement target region, and outputs the image with the measurement result superimposed onto the display.

Description

Technical Field

[0001] The present invention relates to a measuring device, a measuring method, and a program.

Background Art

[0002] Industrial endoscope devices are used for inspections (endoscopic inspections) of abnormalities and corrosion inside boilers, pipes, aircraft engines, heat exchangers, etc. The device disclosed in Patent Document 1 has an elongated probe including an insertion tube that can be inserted into an observation target, and generates an image based on an optical image acquired via the probe. The device determines the three-dimensional (3D) coordinates of points in the observation target by using an image of the observation target, and determines a reference plane by using the 3D coordinates of three or more points. The device calculates the distance between the reference plane and each point, and displays a color map of each point colored according to the distance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique disclosed in Patent Document 1, the user needs to specify three or more points by using an input device such as a joystick. Therefore, it is troublesome for the user and the measurement is inefficient.

[0005] An object of the present invention is to provide a measuring device, a measuring method, and a program that can efficiently perform measurement.

Means for Solving the Problems

[0006] The present invention relates to a measuring device having a control unit, the control unit acquires three-dimensional data including the three-dimensional coordinates of two or more points on a subject calculated based on an endoscopic image of the subject acquired by an endoscope, sets a measurement reference indicating a reference position for measurement, detects one or more feature regions on the subject based on the three-dimensional data or the endoscopic image, determines that the three-dimensional shape of the subject has common features in each of the one or more feature regions, selects at least one of the one or more feature regions as a measurement target region, sets one or more points included in the measurement target region and included in the two or more points as measurement points, generates distance information indicating the distance between the measurement reference and the measurement points, superimposes the measurement results generated based on the distance information onto an image of the measurement target region, and outputs the superimposed image to a display.

[0007] In the measuring device of the present invention, the control unit superimposes a graphic having a display state set based on the distance information onto the image of the area to be measured as the measurement result.

[0008] In the measuring device of the present invention, the control unit selects at least one of the two or more measurement points based on the distance indicated by the distance information of each of the two or more measurement points included in the measurement target area, and superimposes the information of the at least one measurement point onto the image of the measurement target area as the measurement result.

[0009] In the measuring device of the present invention, the control unit selects the measurement point with the largest or smallest distance as the at least one measurement point.

[0010] In the measuring device of the present invention, the control unit calculates a reference value based on the distance and selects at least one measurement point based on the result of comparing the distance with the reference value.

[0011] In the measuring device of the present invention, the control unit sets all points included in the measurement target area and included in the two or more points as the measurement points.

[0012] In the measuring device of the present invention, the control unit detects one or more feature regions based on the shape features of two or more points included in the three-dimensional data.

[0013] In the measuring device of the present invention, the control unit detects one or more feature regions based on the image features in the endoscopic image.

[0014] In the measuring device of the present invention, when only one feature region is detected on the subject, the control unit selects the one feature region as the measurement target region.

[0015] In the measuring device of the present invention, when two or more feature regions are detected on the subject, the control unit selects at least one of the two or more feature regions as the measurement target region.

[0016] In the measuring device of the present invention, the control unit selects the feature region that has the smallest distance between each of the two or more feature regions and the position of the tip of the endoscope as the measurement target region.

[0017] In the measuring device of the present invention, the control unit selects at least one feature region as the measurement target region based on the shape characteristics of the two or more feature regions.

[0018] In the measuring device of the present invention, the control unit selects the at least one feature region as the measurement target region based on the information input to the input device.

[0019] In the measuring device of the present invention, the control unit sets a point at the tip of the endoscope as the measurement reference.

[0020] In the measuring device of the present invention, the control unit sets a point at the tip of the endoscope as a first measurement reference, sets a second measurement reference including one or more points from the two or more points, generates first distance information indicating the distance between the first measurement reference and the measurement point, generates second distance information indicating the distance between the second measurement reference and the measurement point, superimposes the first measurement result generated based on the first distance information onto an image of the measurement target area, superimposes the second measurement result generated based on the second distance information onto an image of the measurement target area, and outputs the image with the first measurement result superimposed and the image with the second measurement result superimposed to the display.

[0021] In the measuring device of the present invention, the control unit outputs either the image on which the first measurement result is superimposed or the image on which the second measurement result is superimposed to the display, and the display of the image on which the first measurement result is superimposed and the display of the image on which the second measurement result is superimposed are switchable.

[0022] In the measuring device of the present invention, the control unit calculates the distance between the measurement reference and each of the two or more points, selects at least one of the two or more points based on the distance between the measurement reference and each of the two or more points, and outputs information of the feature region that is included in the one or more feature regions and includes at least one of the points to the display.

[0023] The present invention acquires three-dimensional data including three-dimensional coordinates of two or more points on a subject calculated based on an endoscopic image of the subject acquired by an endoscope, sets a measurement reference indicating a reference position for measurement, detects one or more characteristic regions on the subject based on the three-dimensional data or the endoscopic image, the three-dimensional shape of the subject has a common characteristic in each of the one or more characteristic regions, selects at least one of the one or more characteristic regions as a measurement target region, sets one or more points included in the measurement target region and also included in the two or more points as measurement points, generates distance information indicating the distance between the measurement reference and the measurement points, superimposes a measurement result generated based on the distance information on an image of the measurement target region, and outputs the image with the measurement result superimposed on a display.

[0024] The present invention includes steps of acquiring three-dimensional data including three-dimensional coordinates of two or more points on a subject calculated based on an endoscopic image of the subject acquired by an endoscope, setting a measurement reference indicating a reference position for measurement, detecting one or more characteristic regions on the subject based on the three-dimensional data or the endoscopic image, the three-dimensional shape of the subject having a common characteristic in each of the one or more characteristic regions, selecting at least one of the one or more characteristic regions as a measurement target region, setting one or more points included in the measurement target region and also included in the two or more points as measurement points, generating distance information indicating the distance between the measurement reference and the measurement points, generating a graphic to be superimposed on an image of the measurement target region based on a measurement result generated based on the distance information, and outputting the image and the graphic of the measurement target region, and is a program for causing a computer to execute these steps.

Advantages of the Invention

[0025] According to the present invention, a measuring device, a measuring method, and a program can efficiently execute measurement.

Brief Description of the Drawings

[0026] [Figure 1] It is a block diagram showing an example of the configuration of an endoscope system according to the first embodiment of the present invention. [Figure 2] It is a flowchart showing an example of the procedure of measurement processing in the first embodiment of the present invention. [Figure 3] It is a diagram showing an example of an image displayed on a display included in the endoscope system according to the first embodiment of the present invention. [Figure 4] It is a diagram showing an example of an image displayed on a display included in the endoscope system according to the first embodiment of the present invention. [Figure 5] It is a diagram showing an example of an image displayed on a display included in the endoscope system according to the first embodiment of the present invention. [Figure 6] It is a diagram showing an example of an image displayed on a display included in the endoscope system according to the first embodiment of the present invention. [Figure 7] It is a block diagram showing an example of the configuration of an endoscope system according to the second modification of the first embodiment of the present invention. [Figure 8] It is a block diagram showing an example of the configuration of an endoscope system according to the third modification of the first embodiment of the present invention. [Figure 9] It is a flowchart showing an example of the procedure of measurement processing in the second embodiment of the present invention. [Figure 10] It is a diagram showing an example of an image displayed on a display included in the endoscope system according to the second embodiment of the present invention. [Figure 11] It is a flowchart showing an example of the procedure of measurement processing in the third embodiment of the present invention. [Figure 12] It is a diagram showing an example of an image displayed on a display included in the endoscope system according to the third embodiment of the present invention. [Figure 13] It is a flowchart showing an example of the procedure of measurement processing in a modification of the third embodiment of the present invention. [Figure 14]This figure shows an example of an image displayed on a display of an endoscope system according to a modification of the third embodiment of the present invention. [Figure 15] This figure shows an example of an image displayed on a display of an endoscope system according to a modification of the third embodiment of the present invention. [Figure 16] This figure shows an example of an image displayed on a display of an endoscope system according to a modification of the third embodiment of the present invention. [Figure 17] This flowchart shows an example of the measurement process procedure in the fourth embodiment of the present invention. [Figure 18] This figure shows an example of an image displayed on the display of an endoscope system according to a fourth embodiment of the present invention. [Figure 19] This figure shows an example of an image displayed on the display of an endoscope system according to a fourth embodiment of the present invention. [Modes for carrying out the invention]

[0027] Embodiments of the present invention will be described below with reference to the drawings. In the following, an endoscope system will be described as an example of a measuring device.

[0028] (First embodiment) Figure 1 shows an example of the configuration of an endoscope system 1 according to a first embodiment of the present invention. The endoscope system 1 shown in Figure 1 has an insertion unit 2, a scope unit 3, a base unit 4, and a main unit 5. The insertion unit 2, the scope unit 3, and the base unit 4 constitute an endoscope device 10. The main unit 5 is an operating device.

[0029] The insertion unit 2 is inserted into the body of the subject being observed. The subject is an industrial product. The insertion unit 2 is a long, slender tube and is bendable. The user performs the insertion procedure and inserts the insertion unit 2 into the subject. An optical adapter is attached to the tip of the insertion unit 2. The insertion unit 2 acquires an optical image of the inside of the subject. The insertion unit 2 has an imaging unit 20, a bending unit 21, and an illumination window 22.

[0030] The imaging unit 20 is located at the tip portion 2a, which includes the tip of the insertion portion 2. The imaging unit 20 is an image sensor such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor. The imaging unit 20 generates an image based on the optical image acquired by the insertion portion 2. The image generated by the imaging unit 20 is output to the scope unit 3.

[0031] The curved portion 21 curves the insertion portion 2 upward (U), downward (D), left (L), or right (R). Alternatively, the curved portion 21 curves the insertion portion 2 upward left (UL), upward right (UR), downward left (DL), or downward right (DR).

[0032] Illumination light is generated by the light source 35 of the scope unit 3 and output to the tip 2a through a light guide (not shown) located inside the insertion section 2. The illumination light is shone into the inside of the subject through the illumination window 22.

[0033] The scope unit 3 includes an imaging drive circuit 30, an image processing unit 31, an UD drive unit 32, an RL drive unit 33, a curvature control unit 34, a light source 35, and a light source control unit 36. The base unit 4 includes a control unit 40, a communication unit 41, a volatile memory 42, and a non-volatile memory 43.

[0034] The imaging drive circuit 30 controls the imaging unit 20 and outputs the image output from the imaging unit 20 to the image processing unit 31. The image processing unit 31 performs image processing such as noise reduction on the image output from the imaging unit 20 and outputs the image to the control unit 40.

[0035] The UD drive unit 32 is connected to a UD bending wire for bending the curved section 21 in the U or D direction. The UD drive unit 32 has a motor and bends the curved section 21 in the U or D direction by pulling the UD bending wire. The RL drive unit 33 is connected to an RL bending wire for bending the curved section 21 in the R or L direction. The RL drive unit 33 has a motor and bends the curved section 21 in the R or L direction by pulling the RL bending wire. The bending control unit 34 controls the UD drive unit 32 and the RL drive unit 33.

[0036] The UD drive unit 32 and the RL drive unit 33 can operate simultaneously. For example, the UD drive unit 32 and the RL drive unit 33 can bend the curved section 21 in the UL direction.

[0037] The light source 35 is an LED (Light-Emitting Diode) or the like, and generates illumination light. The illumination light is output from the light source 35 to a light guide (not shown). The light source control unit 36 ​​controls the light source 35.

[0038] The control unit 40 controls the parts of the scope unit 3 and the base unit 4. At least one of the control unit 40, the image processing unit 31, the curvature control unit 34, and the light source control unit 36 ​​may consist of at least one processor and logic circuit. For example, the processor is at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). At least one of the control unit 40, the image processing unit 31, the curvature control unit 34, and the light source control unit 36 ​​may include one or more processors. At least one of the control unit 40, the image processing unit 31, the curvature control unit 34, and the light source control unit 36 ​​may include one or more logic circuits.

[0039] The computer of the endoscope system 1 may load a program and execute the loaded program. The program includes instructions that define the operation of at least one of the control unit 40, image processing unit 31, bending control unit 34, and light source control unit 36. In other words, at least one function of the control unit 40, image processing unit 31, bending control unit 34, and light source control unit 36 ​​may be implemented by software.

[0040] The above program may be provided on a "computer-readable recording medium," such as flash memory. The program may be transmitted from the computer holding the program to the endoscope system 1 via a transmission medium or by transmission waves within the transmission medium. The "transmission medium" for transmitting the program is a medium that has the function of transmitting information. A medium that has the function of transmitting information includes networks such as the Internet and communication lines such as telephone lines. The above program may implement some of the functions described above. Furthermore, the above program may be a differential file (differential program). The functions described above may be implemented by a combination of a program already recorded on the computer and a differential program.

[0041] The communication unit 41 has a communication circuit and performs wired or wireless communication with the main unit 5 for purposes such as curvature control. The communication unit 41 transmits images generated by the imaging unit 20 to the main unit 5. The volatile memory 42 is a RAM (Random Access Memory) or DRAM (Dynamic RAM), etc. The volatile memory 42 stores various information processed by the control unit 40. The non-volatile memory 43 is an SRAM (Static RAM), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable ROM), or flash memory. The non-volatile memory 43 may be detachable from the base unit 4. The non-volatile memory 43 stores images generated by the imaging unit 20 and various information processed by the control unit 40.

[0042] The main unit 5 includes a control unit 50, a display 51, a touch panel 52, operation buttons 53, a communication unit 54, a communication unit 55, a volatile memory 56, and a non-volatile memory 57. The main unit 5 may also be an information terminal such as a smartphone or tablet.

[0043] The control unit 50 controls each part of the main unit 5. The control unit 50 may consist of at least one processor and one logic circuit. The control unit 50 may include one or more processors or one or more logic circuits. The computer of the endoscope system 1 may read and execute a program. The program includes instructions that define the operation of the control unit 50. In other words, the functions of the control unit 50 may be implemented by software. The program that implements the functions of the control unit 50 may be implemented in the same way as the program that implements the functions of the control unit 40 and other units.

[0044] The display 51 is a monitor such as an LCD (Liquid Crystal Display). The display 51 displays images generated by the imaging unit 20. The touch panel 52 accepts input for information necessary to control the endoscope system 1. The touch panel 52 is located on the screen of the display 51. By operating the touch panel 52, the user can input instructions to change the settings of the endoscope system 1 and instructions necessary to operate the endoscope system 1 into the endoscope system 1.

[0045] The operation button 53 receives various instructions from the user. By pressing the operation button 53, the user can input instructions regarding power or illumination to the endoscope system 1. The communication unit 54 performs wired or wireless communication with the base unit 4 for purposes such as bending control. The communication unit 54 receives images generated by the imaging unit 20 from the base unit 4. The communication unit 55 performs wired or wireless communication with the external device 11. The external device 11 may be a remote control, keyboard, or mouse.

[0046] The volatile memory 56 is a memory similar to the volatile memory 42 and stores various information processed by the control unit 50. The non-volatile memory 57 is a memory similar to the non-volatile memory 43 and stores images generated by the imaging unit 20 and various information processed by the control unit 50. The non-volatile memory 57 may be detachable from the main unit 5.

[0047] Figure 2 shows an example of the measurement process performed by the endoscope system 1. Figure 2 will be used to explain the operation of the endoscope system 1.

[0048] The control unit 50 acquires 3D data (step S100).

[0049] The control unit 50 performs the following processing in step S100. The first to third examples are described below.

[0050] First, let's describe the first example. In the first example, the optical adapter is a stereo optical adapter with two fields of view. The optical adapter has a first optical system and a second optical system corresponding to the two fields of view. The first and second optical systems form two optical images of the subject on the imaging unit 20. The imaging unit 20 generates a stereo image corresponding to the first and second optical images. The stereo image includes a pair of two images (the first image and the second image). That is, the stereo image includes an image of the subject as seen from a first viewpoint and an image of the subject as seen from a second viewpoint. The control unit 50 calculates the 3D coordinates of two or more points on the subject using one or more stereo images generated by the imaging unit 20 and generates 3D data including those 3D coordinates.

[0051] Next, a second example will be described. In the second example, the optical adapter is a monocular optical adapter with a single field of view. In the first example, the optical adapter forms two optical images of the subject, but in the second example, the optical adapter forms one optical image of the subject. The imaging unit 20 generates an image corresponding to the optical image formed by the optical adapter. The imaging unit 20 performs imaging from two or more different viewpoints and generates two or more images. The control unit 50 uses the two or more images generated by the imaging unit 20 to calculate the 3D coordinates of two or more points on the subject and generates 3D data including those 3D coordinates.

[0052] Next, a third example will be described. The 3D data generated in the first or second example is pre-stored in the non-volatile memory 57. The control unit 50 retrieves the 3D data from the non-volatile memory 57.

[0053] The 3D coordinates included in the 3D data are defined in a 3D space corresponding to real space. Hereinafter, points with 3D coordinates included in the 3D data will be referred to as points included in the 3D data. The control unit 50 may generate 3D data that includes the 3D coordinates of three or more points.

[0054] After step S100, the control unit 50 sets a measurement reference in a space containing two or more points included in the 3D data. For example, information indicating the measurement reference to be used is recorded in the non-volatile memory 57, and the control unit 50 sets the measurement reference according to that information (step S101). The measurement reference indicates a reference position for calculating the 3D distance, which will be described later. In the following example, the measurement reference indicates the position of the tip of the endoscope. The tip of the endoscope corresponds to the tip of the insertion section 2.

[0055] After step S101, the control unit 50 calculates the 3D distance between the measurement reference and each of the two or more points included in the 3D data. In other words, the control unit 50 calculates the 3D distance between the point at the tip of the endoscope and each point included in the 3D data. The control unit 50 generates distance information indicating the calculated 3D distance (step S102). The distance information is stored in the volatile memory 56.

[0056] After step S102, the control unit 50 performs segmentation. In segmentation, the control unit 50 detects one or more segments (feature regions) on the subject by using 3D data. The control unit 50 may also detect one or more segments by using one or more two-dimensional (2D) images used to generate the 3D data. The one or more 2D images are one or more stereo images acquired using a stereo optical adapter, or two or more images acquired using a monocular optical adapter (step S103). Information on each segment is stored in the volatile memory 56.

[0057] Step S103 is described in detail. The control unit 50 extracts the features of the 3D shape of the object using 3D data. The control unit 50 assigns each point corresponding to the 3D coordinates in the 3D data to one of two or more feature regions according to the extracted features. The 3D shape of the object has common features within one feature region. The features of the 3D shape of the object are different between two or more different feature regions. Only one point may be assigned to one feature region. If two or more points are assigned to one feature region, those two or more points satisfy common conditions that indicate the features of the 3D shape of the object. The conditions that a point in the first feature region satisfies are different from the conditions that a point in the second feature region, which is different from the first feature region, satisfies.

[0058] Segmentation is a simple method for classifying each point corresponding to 3D data. For example, the control unit 50 can use Euclidean Cluster Extraction for segmentation. This is a function included in the open-source PCL (Point Cloud Library).

[0059] The control unit 50 uses this function to determine that points within a predetermined distance from each point included in the 3D data are neighboring points. A point and its neighboring points are located on the same object. For example, if the subject has a first object and a second object that are separated from each other, each point included in the 3D data is classified as either a point on the first object or a point on the second object. Each of the first and second objects is a feature region (segment). In this case, two or more points within a single feature region are characterized as being located on the same object.

[0060] The control unit 50 may use the Watershed algorithm or Deep Learning, etc., for segmentation. The control unit 50 may calculate normals perpendicular to the surface of the subject based on 3D data and detect edges or steps of the subject as feature regions based on changes in the direction of the normals. For example, the control unit 50 may detect a first feature region consisting of edges or steps and a second feature region consisting of parts other than those edges or steps.

[0061] The control unit 50 may detect edges of the subject by performing image processing on a 2D image of the subject. The control unit 50 may detect feature regions corresponding to edges on the 2D image from the 3D shape of the subject shown in the 3D data. The control unit 50 may detect feature regions based on the brightness or color of the 2D image. The control unit 50 may perform matching processing on a stereo image of the subject and detect feature regions based on the correlation values ​​obtained in the matching processing.

[0062] After step S103, the control unit 50 generates a 3D image based on the 3D data and outputs the 3D image to the display 51. The display 51 displays the 3D image (step S104). The 3D image is an image of the 3D shape of the subject.

[0063] The control unit 50 may superimpose information from one or more feature regions onto the 3D image and output the 3D image with the superimposed information to the display 51. For example, the control unit 50 may superimpose graphics (computer graphics) having colors corresponding to each feature region onto the 3D image. Alternatively, the control unit 50 may superimpose graphics showing frames surrounding each feature region onto the 3D image.

[0064] Figure 3 shows an example of a 3D image displayed on the display 51 in step S104. The control unit 50 displays the 3D image IMG10 shown in Figure 3 on the display 51. The 3D image IMG10 includes a first region RG10 and a second region RG11 of the subject.

[0065] After the 3D image is displayed, the user operates the touch panel 52 or the like to select the measurement target area. This allows the user to input position information indicating their location on the 3D image displayed on the display 51. The control unit 50 receives the position information and selects the feature area at the location indicated by that position information as the measurement target area. The number of feature areas selected as measurement target areas is less than or equal to the number of feature areas detected in step S103. If only one feature area is detected in step S103, the control unit 50 may select that feature area as the measurement target area without receiving position information. The control unit 50 may also select two or more feature areas as measurement target areas (step S105).

[0066] Figure 4 shows an example of a 3D image displayed on the display 51 in step S105. The control unit 50 displays the 3D image IMG10 shown in Figure 4 on the display 51. The 3D image IMG10 shown in Figure 4 is the same as the 3D image IMG10 shown in Figure 3 and includes a first region RG10 and a second region RG11.

[0067] In step S103, the control unit 50 detects the first region RG10 and the second region RG11 as feature regions. For example, the user inputs position information indicating their location in the first region RG10. The control unit 50 selects the first region RG10 as the region to be measured.

[0068] After step S105, the control unit 50 sets one or more points included in the measurement target area selected in step S105 as measurement points. These one or more points are included in the 3D data. For example, the control unit 50 sets the point at the position indicated by the position information received in step S105 as a measurement point. In the following example, the control unit 50 sets all points included in the measurement target area as measurement points (step S106). The measurement point information is stored in the volatile memory 56.

[0069] After step S106, the control unit 50 obtains distance information for each measurement point from the volatile memory 56 (step S107).

[0070] After step S107, the control unit 50 sets the display state of the graphic at each measurement point and its corresponding position according to the 3D distance indicated by the distance information. For example, the control unit 50 sets the color of the graphic to the color corresponding to the 3D distance. The control unit 50 superimposes the graphic onto the 3D image (step S108). Hereinafter, the 3D distance indicated by the distance information of each measurement point will be referred to as the 3D distance at each measurement point.

[0071] After step S108, the control unit 50 outputs a 3D image with superimposed graphics to the display 51. The display 51 displays the 3D image. The user may input information to instruct the display of the 3D image by operating the touch panel 52 or the like. When such information is input, the control unit 50 may output a 3D image with superimposed graphics to the display 51 (step S109). When step S109 is executed, the measurement process shown in Figure 2 is completed.

[0072] At least one of steps S103 to S106 may be performed before step S101 or S102 is performed. Step S104 may be performed before one of steps S101 to S103 is performed.

[0073] Before step S105 is executed, the control unit 50 may calculate the 3D distance between the measurement reference and each of the points on the subject, and may display a 3D image on the display 51 with a graphic superimposed according to that 3D distance. If only one feature region is detected, the user can recognize that selecting a feature region is unnecessary by checking the 3D image with the superimposed graphic. The user may also input information to instruct the display of the 3D image without selecting a feature region.

[0074] In the measurement process shown in Figure 2, the control unit 50 automatically sets the measurement reference. When the user inputs instructions to the endoscope system 1 to select the measurement target area, the control unit 50 automatically sets the measurement points. This reduces the effort required from the user and increases the efficiency of the measurement.

[0075] Figure 5 shows a first example of a 3D image displayed on the display 51 in step S109. The control unit 50 displays the 3D image IMG11 shown in Figure 5 on the display 51. The 3D image IMG11 includes a first region RG10 and a second region RG11, similar to the 3D image IMG10 shown in Figure 3.

[0076] The control unit 50 superimposes a graphic on the 3D image IMG11 that shows the measurement point MP10 located at the position indicated by the position information received in step S105. The control unit 50 superimposes a graphic on the 3D image IMG11 that has a color corresponding to the 3D distance at each measurement point in the first region RG10 selected as the measurement target area. In other words, the control unit 50 superimposes a graphic for displaying a color map on the 3D image IMG11. This graphic shows the measurement results of the 3D distance. The image of the first region RG10 has a color corresponding to the 3D distance between the position of the tip of the endoscope and each measurement point. The control unit 50 may also superimpose a graphic such as a numerical value indicating the 3D distance on the 3D image IMG11.

[0077] In each embodiment of the present invention, the area of ​​focus is an abnormal area such as a recess, protrusion, or scratch. A color map is displayed with colors set according to the 3D distance at each measurement point, allowing the user to easily check the condition of the abnormal area.

[0078] The control unit 50 superimposes graphics onto the 3D image IMG11 to make the images of feature regions other than the measurement target region less prominent. For example, the image of the second region RG11 is grayed out. The control unit 50 may also hide the image of the second region RG11. The user can then focus on checking the inspection results in the measurement target region, which is the area of ​​interest. The control unit 50 may also redisplay the images of the hidden feature regions on the display 51.

[0079] Figure 6 shows a second example of a 3D image displayed on the display 51 in step S109. The control unit 50 displays the 3D image IMG12 shown in Figure 6 on the display 51. The 3D image IMG12 includes a first region RG10 and a second region RG11, similar to the 3D image IMG10 shown in Figure 3.

[0080] The control unit 50 superimposes a graphic indicating the measurement point MP10 onto the 3D image IMG12, similar to the 3D image IMG11 shown in Figure 5. The control unit 50 also superimposes a graphic for displaying a color map onto the 3D image IMG12, similar to the 3D image IMG11 shown in Figure 5.

[0081] The control unit 50 identifies the maximum and minimum values ​​of the 3D distance at each measurement point in the first region RG10. The measurement point where the maximum value is measured corresponds to the deepest point of the recess formed in the specimen. The measurement point where the minimum value is measured corresponds to the highest point of the protrusion formed in the specimen.

[0082] The control unit 50 superimposes a graphic icon representing the highest point HP10 onto the 3D image IMG12. This graphic is superimposed at the position of the highest point HP10. The control unit 50 also superimposes a graphic icon representing the deepest point DP10 onto the 3D image IMG12. This graphic is superimposed at the position of the deepest point DP10. The graphic icon representing the highest point HP10 or the deepest point DP10 indicates the measurement result of the 3D distance. The control unit 50 may superimpose the graphic icon representing the highest point HP10 or the deepest point DP10 onto the 3D image IMG12 without superimposing a graphic for displaying a color map.

[0083] The measurement reference may be a point, line, or surface set in a 3D space that defines the 3D coordinates included in the 3D data. For example, the user inputs position information indicating the location of one or more reference points by operating a touch panel 52 or the like. The control unit 50 receives the position information and sets a reference point at the position indicated by that position information. The reference point is included in the 3D data.

[0084] For example, the control unit 50 sets a straight line containing two reference points. In step S102, the control unit 50 calculates the 3D distance between that straight line and each measurement point. Alternatively, the control unit 50 sets a plane containing three reference points. In step S102, the control unit 50 calculates the 3D distance between that plane and each measurement point.

[0085] After the measurement criteria have been established, they may be changed. For example, after the position of the tip of the endoscope has been established as the measurement criteria, the measurement criteria may be changed to a point, line, or plane. Details of how to change the measurement criteria will be described in the fourth embodiment.

[0086] The control unit 40 may perform the measurement process shown in Figure 2. Alternatively, the control unit 40 and the control unit 50 may perform the measurement process jointly.

[0087] Each embodiment of the present invention includes a control unit 50. The control unit 50 acquires 3D data including the 3D coordinates of two or more points on a subject calculated based on an endoscopic image (2D image) of the subject acquired by an endoscope (insertion unit 2). The control unit 50 sets a measurement reference indicating the reference position for measurement. The control unit 50 detects one or more feature regions on the subject based on the 3D data or endoscopic image. The 3D shape of the subject has common features in each of the one or more feature regions. The control unit 50 selects at least one of the one or more feature regions as the measurement target region. The control unit 50 sets one or more points included in the measurement target region and included in the two or more points as measurement points. The control unit 50 generates distance information indicating the distance between the measurement reference and the measurement points. The control unit 50 superimposes the measurement results generated based on the distance information onto the image of the measurement target region and outputs the superimposed image to the display 51. The image of the measurement target region is at least a part of the image of the 3D shape generated based on the 3D data.

[0088] The control unit 50 may generate a graphic to be superimposed on the image of the measurement target area based on the measurement results generated based on the distance information. The control unit 50 may output the image of the measurement target area and the graphic. The control unit 50 may superimpose the graphic on the image of the measurement target area and output the image with the superimposed graphic to a processing device or recording medium. For example, the control unit 50 may save the image of the measurement target area and the graphic as data such as CSV in memory. The data saved in memory may be output to another device and processed by software running on that device. For example, the software may superimpose the graphic output from memory onto the image output from memory and display the image on a display. The software may perform various processes on the data output from memory.

[0089] Each aspect of the present invention's measurement method comprises eight steps. In the first step (step S100), the control unit 50 acquires 3D data. In the second step (step S101), the control unit 50 sets a measurement reference. In the third step (step S103), the control unit 50 detects one or more feature regions on the subject. In the fourth step (step S105), the control unit 50 selects at least one feature region as the measurement target region. In the fifth step (step S106), the control unit 50 sets one or more points as measurement points. In the sixth step (step S107), the control unit 50 generates distance information indicating the distance between the measurement reference and the measurement points. In the seventh step (step S108), the control unit 50 superimposes the measurement results generated based on the distance information onto an image of the measurement target region. In the eighth step (step S109), the control unit 50 outputs an image with the measurement results superimposed on it to the display 51.

[0090] A program according to each aspect of the present invention causes a computer to execute the first to eighth steps described above.

[0091] Each aspect of the present invention may include the following modifications. The control unit 50 superimposes a graphic having a display state set based on distance information onto the image of the area to be measured as a measurement result.

[0092] Each aspect of the present invention may include the following modifications. The control unit 50 selects at least one measurement point from among two or more measurement points included in the measurement target area based on the distance indicated by the distance information of each of those two or more measurement points. The control unit 50 superimposes the information of at least one measurement point onto the image of the measurement target area as a measurement result.

[0093] Each aspect of the present invention may include the following modifications: The control unit 50 selects at least one measurement point that has the largest or smallest distance indicated by the distance information.

[0094] Each aspect of the present invention may include the following modifications. The control unit 50 sets all points included in two or more points that are included in the measurement target area and included in the 3D data as measurement points.

[0095] Each aspect of the present invention may include the following modifications. The control unit 50 detects one or more feature regions based on the shape features of two or more points included in the 3D data.

[0096] Each aspect of the present invention may include the following modifications. The control unit 50 detects one or more feature regions based on image features in the endoscopic image.

[0097] Each aspect of the present invention may include the following modifications. When only one feature region is detected on the subject, the control unit 50 selects that one feature region as the measurement target region.

[0098] Each aspect of the present invention may include the following modifications. When two or more feature regions are detected on the subject, the control unit 50 selects at least one of the two or more feature regions as the measurement target region.

[0099] Each aspect of the present invention may include the following modifications. The control unit 50 selects at least one feature region as the measurement target region based on information input to the input device (touch panel 52, etc.).

[0100] Each aspect of the present invention may include the following modifications. The control unit 50 sets a point at the tip of the endoscope as the measurement reference.

[0101] In the first embodiment, the control unit 50 sets a measurement criterion and selects at least one feature region as the measurement target region. The control unit 50 also sets one or more points included in the measurement target region as measurement points. Because the effort required for user operations is reduced, the endoscope system 1 can perform measurements efficiently.

[0102] When the control unit 50 selects a portion of the area of ​​the subject as the measurement target area, the user can concentrate on checking the measurement results in the measurement target area.

[0103] (First modification of the first embodiment) A first modification of the first embodiment of the present invention will now be described. In the first modification of the first embodiment, the control unit 50 automatically selects the feature region detected through segmentation as the measurement target region.

[0104] The endoscope system 1 performs the measurement process shown in Figure 2. The control unit 50 performs the following process in step S105.

[0105] If only one feature region is detected in step S103, the control unit 50 selects that feature region as the measurement target region. If two or more feature regions are detected in step S103, the control unit 50 selects at least one of those two or more feature regions as the measurement target region.

[0106] The following describes examples 1 through 4.

[0107] First, let's explain the first example. The control unit 50 calculates the 3D distance between each of two or more feature regions and the position of the tip of the endoscope, and selects the feature region with the smallest 3D distance as the measurement target region.

[0108] Next, a second example will be described. The control unit 50 selects the measurement target area based on the shape characteristics of two or more feature areas. Specifically, the control unit 50 selects the feature area with the largest size as the measurement target area. The control unit 50 may also determine the size of the feature area based on the number of points contained in the feature area.

[0109] Next, a third example will be described. In this third example as well, the control unit 50 selects the measurement target area based on the shape characteristics of two or more feature regions. Specifically, the control unit 50 calculates the 3D distance between the position of the tip of the endoscope and each point in the feature region. The control unit 50 identifies the maximum and minimum values ​​of the 3D distance. The difference between the maximum and minimum values ​​indicates the degree of unevenness in the feature region. The control unit 50 selects the feature region with the greatest degree of unevenness as the measurement target area.

[0110] Next, a fourth example will be described. The control unit 50 selects the feature area displayed in the center of the display 51 screen as the measurement target area.

[0111] After the measurement target area has been selected as described above, the control unit 50 may select other measurement target areas in response to instructions from the user.

[0112] Each aspect of the present invention may include the following modifications. The control unit 50 selects the feature region that has the smallest distance between each of two or more feature regions and the position of the tip of the endoscope as the measurement target region.

[0113] Each aspect of the present invention may include the following modifications. The control unit 50 selects at least one feature region as the measurement target region based on the shape features of two or more feature regions.

[0114] In the first modified example of the first embodiment, the endoscope system 1 can perform measurements efficiently, similar to the first embodiment.

[0115] (Second modification of the first embodiment) A second modification of the first embodiment of the present invention will now be described. Figure 7 shows an example of the configuration of the endoscope system 1a according to the second modification of the first embodiment. Parts that are the same as those shown in Figure 1 will not be described.

[0116] The endoscopic system 1a shown in Figure 7 has an insertion section 2 and a main unit 6. The insertion section 2 and the main unit 6 constitute the endoscopic device 10a.

[0117] The insertion unit 2 shown in Figure 7 is the same as the insertion unit 2 shown in Figure 1. The main unit 6 includes an imaging drive circuit 30, an image processing unit 31, an UD drive unit 32, an RL drive unit 33, a curvature control unit 34, a light source 35, a light source control unit 36, a display 51, a touch panel 52, operation buttons 53, a communication unit 55, a volatile memory 56, a non-volatile memory 57, and a control unit 60. Blocks that are the same as those shown in Figure 1 are assigned the same reference numerals as those shown in Figure 1.

[0118] The control unit 60 has both the functions of the control unit 40 shown in Figure 1 and the functions of the control unit 50 shown in Figure 1. The control unit 60 executes the process shown in Figure 2.

[0119] In the second modification of the first embodiment, the endoscope system 1 can perform measurements efficiently, similar to the first embodiment.

[0120] (Third modification of the first embodiment) A third modification of the first embodiment of the present invention will now be described. Figure 8 shows an example of the configuration of the endoscope system 1b according to the third modification of the first embodiment. Parts that are the same as those shown in Figure 1 will not be described.

[0121] The endoscope system 1b shown in Figure 8 comprises an insertion unit 2, a scope unit 3b, and a base unit 7. The insertion unit 2 and the scope unit 3b constitute the endoscope device 10b. The scope unit 3b and the base unit 7 are connected by a cable 8.

[0122] The insertion unit 2 shown in Figure 8 is the same as the insertion unit 2 shown in Figure 1. The scope unit 3b shown in Figure 8 is the same as the scope unit 3 shown in Figure 1, except that it does not have an image processing unit 31. The base unit 7 has an image processing unit 31, a display 51, a touch panel 52, operation buttons 53, a communication unit 55, a volatile memory 56, a non-volatile memory 57, and a control unit 70. The same blocks as shown in Figure 1 are assigned the same reference numerals as shown in Figure 1.

[0123] The control unit 70 has both the functions of the control unit 40 shown in Figure 1 and the functions of the control unit 50 shown in Figure 1. The control unit 70 executes the process shown in Figure 2.

[0124] In the third modification of the first embodiment, the endoscope system 1 can perform measurements efficiently, similar to the first embodiment.

[0125] (Second embodiment) A second embodiment of the present invention will now be described. In the second embodiment, the endoscope system 1 shown in Figure 1 is used. The endoscope system 1a shown in Figure 7 or the endoscope system 1b shown in Figure 8 may be used instead.

[0126] The control unit 50 detects abnormal areas in the measurement target area and changes the display state of the abnormal areas in the 3D image of the subject. Abnormal areas include areas with recesses, protrusions, or scratches.

[0127] Figure 9 shows an example of the measurement process performed by the endoscope system 1. The operation of the endoscope system 1 will be explained using Figure 9. Processes identical to those shown in Figure 2 will not be explained.

[0128] After step S108, the control unit 50 calculates a measurement reference and a reference value for the 3D distance to each measurement point in the measurement target area. For example, the control unit 50 calculates the average value of the 3D distance at each measurement point as the reference value (step S110).

[0129] After step S110, the control unit 50 compares the reference value with the 3D distance at each measurement point and selects one or more measurement points in the measurement target area. These one or more measurement points constitute an abnormal area. For example, the control unit 50 extracts 3D distances that are not included in a predetermined range centered on the reference value and selects the measurement point with that 3D distance. For example, the predetermined range is set based on the standard deviation of the 3D distance at each measurement point (step S111).

[0130] After step S111, the control unit 50 sets the display state of the graphic at the position corresponding to the measurement point selected in step S111. For example, the control unit 50 sets the color of the graphic to a predetermined color. The control unit 50 superimposes the graphic onto the 3D image (step S112). After step S112, step S109 is executed.

[0131] Figure 10 shows an example of a 3D image displayed on the display 51 in step S109. The control unit 50 displays the 3D image IMG13 shown in Figure 10 on the display 51. The 3D image IMG13 includes a first region RG10 and a second region RG11, similar to the 3D image IMG10 shown in Figure 3.

[0132] The control unit 50 superimposes a graphic indicating the measurement point MP10 onto the 3D image IMG13, similar to the 3D image IMG11 shown in Figure 5. The control unit 50 superimposes a graphic for displaying a color map onto the 3D image IMG13, similar to the 3D image IMG11 shown in Figure 5. The control unit 50 superimposes a graphic indicating the abnormal region AR10 and a graphic indicating the abnormal region AR11 onto the 3D image IMG13.

[0133] Each aspect of the present invention may include the following modifications. The control unit 50 calculates a reference value based on the 3D distance indicated by the distance information of each of two or more measurement points included in the measurement target area. The control unit 50 selects at least one measurement point based on the result of comparing the 3D distance with the reference value.

[0134] In the second embodiment, the endoscope system 1 can highlight abnormal areas in the 3D image displayed on the display 51, allowing the user to focus on and examine the condition of the abnormal areas.

[0135] (Third embodiment) A third embodiment of the present invention will now be described. In the third embodiment, the endoscope system 1 shown in Figure 1 is used. The endoscope system 1a shown in Figure 7 or the endoscope system 1b shown in Figure 8 may be used instead.

[0136] Before the user inputs location information for the area to be measured, the control unit 50 detects a feature area that includes an abnormal area and displays information about that feature area on the display 51. The user selects the area to be measured by referring to the information displayed on the display 51.

[0137] Figure 11 shows an example of the measurement process performed by the endoscope system 1. The operation of the endoscope system 1 will be explained using Figure 11. Processes identical to those shown in Figure 2 will not be explained.

[0138] After step S104, the control unit 50 compares the reference value with the 3D distance at each point included in the 3D data and selects one or more points. These one or more points constitute an abnormal region. For example, the control unit 50 extracts 3D distances that do not fall within a predetermined range centered on the reference value and selects points with those 3D distances. For example, the predetermined range is set based on the standard deviation of the 3D distance at each point (step S120).

[0139] After step S120, the control unit 50 outputs information about the feature region, which includes an abnormal region containing one or more points selected in step S120, to the display 51. The display 51 displays this information (step S121). After step S121, step S105 is executed.

[0140] The control unit 50 may calculate the number of points included in the abnormal region for each feature region. The control unit 50 may output information about the feature region containing the abnormal region with the most points to the display 51.

[0141] Figure 12 shows an example of a 3D image displayed on the display 51 in step S121. The control unit 50 displays the 3D image IMG14 shown in Figure 12 on the display 51. The 3D image IMG14 includes a first region RG10 and a second region RG11, similar to the 3D image IMG10 shown in Figure 3.

[0142] If the second region RG11 contains an abnormal region, the control unit 50 superimposes a graphic of the frame FR10 surrounding the second region RG11 onto the 3D image IMG14. The control unit 50 also superimposes a message MS10 on the 3D image IMG14 indicating that a feature region containing an abnormal region has been detected. The user can then determine that the second region RG11 is suitable for measurement.

[0143] Each aspect of the present invention may include the following modifications. The control unit 50 calculates the distance between a measurement reference and each of two or more points included in the 3D data. Based on that distance, the control unit 50 selects at least one of the two or more points. The control unit 50 outputs information of a feature region that is included in one or more feature regions and includes the selected at least one point to the display 51.

[0144] In the third embodiment, the endoscope system 1 can notify the user of information about a characteristic region including an abnormal area, and can prompt the user to select that characteristic region as the measurement target area.

[0145] (Modified version of the third embodiment) A modified example of the third embodiment of the present invention will now be described. In the modified example of the third embodiment, after the user inputs location information of the area to be measured, the control unit 50 determines whether or not the area to be measured includes an abnormal area. That is, the control unit 50 determines whether or not the area to be measured is suitable for measurement. When the area to be measured is suitable for measurement, the control unit 50 displays information on the display 51 indicating that re-selection of the area to be measured is unnecessary. When the area to be measured is not suitable for measurement, the control unit 50 displays information on the display 51 indicating that re-selection of the area to be measured is necessary.

[0146] Figure 13 shows an example of the measurement process performed by the endoscope system 1. The operation of the endoscope system 1 will be explained using Figure 13. Processes identical to those shown in Figure 12 will not be explained.

[0147] After step S120, step S105 is executed. After step S105, the control unit 50 determines whether the measurement target area selected in step S105 includes the points of the abnormal area selected in step S120. Based on this, the control unit 50 determines whether the measurement target area is suitable for measurement (step S130).

[0148] If the measurement target area does not contain any points in the abnormal area, the control unit 50 determines in step S130 that the measurement target area is not suitable for measurement. In step S105, the control unit 50 outputs information to the display 51 to prompt the user to select a feature area different from the feature area selected as the measurement target area. The display 51 displays this information (step S131). After step S131, step S105 is executed.

[0149] When the measurement target area includes points in the abnormal area, the control unit 50 determines in step S130 that the measurement target area is suitable for measurement. In step S105, the control unit 50 outputs information to the display 51 indicating that it is unnecessary to select a feature area different from the feature area selected as the measurement target area. The display 51 displays this information (step S132). After step S132, step S106 is executed.

[0150] The control unit 50 may calculate the number of points included in the abnormal region for each feature region. The control unit 50 may determine whether or not the region to be measured includes the abnormal region containing the most points. If the region to be measured includes the abnormal region, the control unit 50 may determine that the region to be measured is suitable for measurement. If the region to be measured does not include the abnormal region, the control unit 50 may determine that the region to be measured is not suitable for measurement.

[0151] Figure 14 shows an example of a 3D image displayed on the display 51 in step S132. The control unit 50 displays the 3D image IMG15 shown in Figure 14 on the display 51. The 3D image IMG15 includes a first region RG10 and a second region RG11, similar to the 3D image IMG10 shown in Figure 3.

[0152] The control unit 50 superimposes a message MS11 on the 3D image IMG15 indicating that the feature region selected as the measurement target area is suitable for measurement. The user can then determine that re-selection of the feature region is unnecessary.

[0153] Figure 15 shows a first example of a 3D image displayed on the display 51 in step S131. The control unit 50 displays the 3D image IMG16 shown in Figure 15 on the display 51. The 3D image IMG16 includes a first region RG10 and a second region RG11, similar to the 3D image IMG10 shown in Figure 3.

[0154] The control unit 50 superimposes a message MS12 on the 3D image IMG16 indicating that the feature region selected as the measurement target area is unsuitable for measurement. The user can then determine that re-selection of the feature region is necessary.

[0155] The control unit 50 may identify the maximum and minimum values ​​of the 3D distance at each of two or more points included in the 3D data. The point where the maximum value is measured corresponds to the deepest point of a concave part formed in the subject. The point where the minimum value is measured corresponds to the highest point of a convex part formed in the subject. When the measurement target area includes either the highest point or the deepest point, the control unit 50 may determine that the measurement target area is suitable for measurement. When the measurement target area does not include either the highest point or the deepest point, the control unit 50 may determine that the measurement target area is not suitable for measurement.

[0156] Figure 16 shows a second example of a 3D image displayed on the display 51 in step S131. The control unit 50 displays the 3D image IMG17 shown in Figure 16 on the display 51. The 3D image IMG17 includes a first region RG10 and a second region RG11, similar to the 3D image IMG10 shown in Figure 3.

[0157] For example, the user selects the first region RG10 as the measurement target region. If the first region RG10 does not include either the highest point or the deepest point, and the second region RG11 includes either the highest point or the deepest point, the control unit 50 superimposes a graphic of the frame FR11 surrounding the second region RG11 onto the 3D image IMG17. The frame FR11 is displayed in a predetermined color. The user can then determine that the second region RG11 is suitable for measurement.

[0158] In a modified version of the third embodiment, the endoscope system 1 can notify the user whether the measurement target area selected by the user is suitable for measurement. If the measurement target area selected by the user is not suitable for measurement, the endoscope system 1 can prompt the user to re-select the measurement target area.

[0159] (Fourth embodiment) A fourth embodiment of the present invention will now be described. In the fourth embodiment, the endoscope system 1 shown in Figure 1 is used. The endoscope system 1a shown in Figure 7 or the endoscope system 1b shown in Figure 8 may be used instead.

[0160] After the 3D image with the measurement results superimposed is displayed on the display 51, the user can change the measurement criteria. After the measurement criteria are changed, the control unit 50 calculates the 3D distance between the measurement criteria and each measurement point included in the measurement target area, and superimposes a graphic set according to that 3D distance onto the 3D image.

[0161] Figure 17 shows an example of the measurement process performed by the endoscope system 1. The operation of the endoscope system 1 will be explained using Figure 17. Processes identical to those shown in Figure 2 will not be explained.

[0162] After step S109, the control unit 50 outputs a message to the display 51 to prompt the user to confirm whether or not to change the measurement standard. The user inputs information indicating whether or not to change the measurement standard by operating the touch panel 52 or the like. Based on this information, the control unit 50 determines whether or not to change the measurement standard (step S140).

[0163] If the control unit 50 determines in step S140 that the measurement criteria will not be changed, the measurement process shown in Figure 17 ends. If the control unit 50 determines in step S140 that the measurement criteria will be changed, the control unit 50 changes the measurement criteria. In this case, the control unit 50 sets a measurement criterion different from the one set in step S101 (step S141).

[0164] For example, a measurement reference indicating the position of the tip of the endoscope is set in step S101, and a measurement reference indicating a point, line, or surface is set in step S141. The user may input information indicating the measurement reference by operating the touch panel 52 or the like. The control unit 50 may change the measurement reference to the measurement reference indicated by that information.

[0165] After step S141, the control unit 50 calculates the 3D distance between the measurement reference set in step S141 and each measurement point set in step S106, and generates distance information indicating the calculated 3D distance (step S142). The distance information is stored in the volatile memory 56. After step S142, step S107 is executed.

[0166] The user may input position information indicating the location of one or more reference points by operating the touch panel 52 or the like in order to change the measurement standard. The control unit 50 may set a reference point at the location indicated by the position information. The reference point is included in two or more points included in the 3D data.

[0167] When the user inputs location information indicating the position of one reference point, the control unit 50 sets that reference point as the measurement reference. When the user inputs location information indicating the positions of two reference points, the control unit 50 sets the line containing those two reference points as the measurement reference. When the user inputs location information indicating the positions of three reference points, the control unit 50 sets the plane containing those three reference points as the measurement reference.

[0168] When a user inputs location information indicating the positions of two or more reference points, the control unit 50 may determine whether those two or more reference points are included in the same feature region. If those two or more reference points are included in the same feature region, the control unit 50 may change the measurement criteria. If those two or more reference points are not included in the same feature region, the control unit 50 may display information on the display 51 prompting the user to reset the reference points.

[0169] Figure 18 shows an example of a 3D image displayed on the display 51 in step S109 after the measurement criteria have been changed. The control unit 50 displays the 3D image IMG18 shown in Figure 18 on the display 51. The 3D image IMG18 includes a first region RG10 and a second region RG11, similar to the 3D image IMG10 shown in Figure 3.

[0170] For example, the user inputs location information indicating the positions of three reference points in order to change the measurement standard. The control unit 50 superimposes graphics showing the reference points RP10, RP11, and RP12 indicated by the location information onto the 3D image IMG18. The control unit 50 also superimposes graphics showing a plane PL10 containing the reference points RP10, RP11, and RP12 onto the 3D image IMG18. Plane PL10 is the measurement standard.

[0171] The control unit 50 superimposes a graphic onto the 3D image IMG18 that has the 3D distance at each measurement point in the first region RG10 selected as the measurement target area, and the corresponding color. In other words, the control unit 50 superimposes a graphic for displaying a color map onto the 3D image IMG18.

[0172] Figure 19 shows an example of a 3D image displayed on the display 51 after the viewpoint for displaying the 3D image IMG18 shown in Figure 18 has been changed. The control unit 50 displays the 3D image IMG19 shown in Figure 19 on the display 51. The 3D image IMG19 includes a first region RG10 and a second region RG11, similar to the 3D image IMG18 shown in Figure 18. The 3D image IMG19 includes a plane PL10, similar to the 3D image IMG18 shown in Figure 18. The user can check the results of measurements using different measurement criteria than the initially set ones. By using different measurement criteria and the position of the endoscope tip, fine undulations on the surface of the subject may appear in the 3D image.

[0173] The control unit 50 may set two different measurement criteria. The control unit 50 may generate first distance information in step S107 by using the first measurement criterion, superimpose a first graphic onto the 3D image based on the first distance information in step S108, and display the 3D image with the first graphic superimposed on it on the display 51 in S109.

[0174] The control unit 50 may generate second distance information in step S107 by using a second measurement criterion different from the first measurement criterion, superimpose a second graphic onto the 3D image based on the second distance information in step S108, or display the 3D image with the second graphic superimposed on it on the display 51 in S109.

[0175] The control unit 50 may display on the display 51 either a 3D image with a first graphic superimposed or a 3D image with a second graphic superimposed. The control unit 50 may switch between the first state and the second state. In the first state, the display 51 displays a 3D image with a first graphic superimposed. In the second state, the display 51 displays a 3D image with a second graphic superimposed.

[0176] In this embodiment using the first graphic and the second graphic, the method of setting the area or the order in which the processing is performed is not particularly limited. For example, the control unit 50 may set a measurement target area, set a second measurement reference in that measurement target area, and generate second distance information. Alternatively, the control unit 50 may set a second measurement reference at an arbitrary position, set the area related to the set position as the measurement target area, and generate second distance information.

[0177] Each aspect of the present invention may include the following modifications. The control unit 50 sets a point at the tip of the endoscope as a first measurement reference. The control unit 50 sets a second measurement reference that includes one or more points from two or more points included in the 3D data. The control unit 50 generates first distance information indicating the distance between the first measurement reference and the measurement point, and generates second distance information indicating the distance between the second measurement reference and the measurement point. The control unit 50 superimposes the first measurement result generated based on the first distance information onto the image of the area to be measured, and superimposes the second measurement result generated based on the second distance information onto the image of the area to be measured. The control unit 50 outputs the image with the first measurement result superimposed and the image with the second measurement result superimposed to the display.

[0178] Each aspect of the present invention may include the following modifications. The control unit 50 outputs either an image with the first measurement result superimposed or an image with the second measurement result superimposed to the display 51. The display of the image with the first measurement result superimposed and the display of the image with the second measurement result superimposed are switchable.

[0179] In the fourth embodiment, the control unit 50 performs the measurement using each of two or more measurement criteria. The endoscope system 1 can notify the user of the measurement results obtained using each of the two or more measurement criteria.

[0180] While preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their variations. Additions, omissions, substitutions, and other modifications are possible without departing from the spirit of the invention. Furthermore, the present invention is not limited by the foregoing description, but only by the scope of the appended claims. [Explanation of Symbols]

[0181] 1,1a,1b Endoscopic System 2 Insertion part 2a Tip 3,3b Scope Unit 4.7 Base Unit 5,6 Main Unit 10, 10a, 10b Endoscope equipment 11 External device 20 Imaging Department 21 Curved section 22 Lighting windows 30 Imaging drive circuit 31 Image Processing Unit 32 UD drive unit 33 RL drive unit 34 Curve Control Unit 35 Light source 36 Light source control unit 40, 50, 60, 70 Control Unit 41, 54, 55 Communications Department 42,56 Volatile memory 43,57 Non-volatile memory 51 displays 52 Touch Panel 53 Operation buttons

Claims

1. It has a control unit, and the control unit is Based on endoscopic images of the subject obtained by the endoscope, three-dimensional data is acquired that includes the three-dimensional coordinates of two or more points on the subject. A measurement standard is established to indicate the reference position for measurement. Based on the three-dimensional data or the endoscopic image, one or more feature regions are detected on the subject, and the three-dimensional shape of the subject has common features in each of the one or more feature regions. Select at least one feature region from the one or more feature regions mentioned above as the measurement target region. One or more points included in the measurement target area and included in the two or more points are set as measurement points. Distance information indicating the distance between the measurement reference and the measurement point is generated. The measurement results generated based on the distance information are superimposed onto the image of the measurement target area. The image with the measurement results superimposed is output to the display. Measuring device.

2. The control unit superimposes a graphic having a display state set based on the distance information onto the image of the measurement target area as the measurement result. The measuring device according to claim 1.

3. The control unit, Based on the distance indicated by the distance information of each of the two or more measurement points included in the measurement target area, at least one measurement point is selected from the two or more measurement points. The information of the at least one measurement point is superimposed on the image of the measurement target region as the measurement result. The measuring device according to claim 1.

4. The control unit selects the measurement point with the largest or smallest distance as the at least one measurement point. The measuring device according to claim 3.

5. The control unit, Based on the aforementioned distance, a reference value is calculated, Select at least one measurement point based on the result of comparing the distance with the reference value. The measuring device according to claim 3.

6. The control unit sets all points included in the measurement target area and included in two or more points as measurement points. The measuring device according to claim 1.

7. The control unit detects one or more feature regions based on the shape features of the two or more points included in the three-dimensional data. The measuring device according to claim 1.

8. The control unit detects one or more feature regions based on the image features in the endoscopic image. The measuring device according to claim 1.

9. When only one feature region is detected on the subject, the control unit selects the one feature region as the measurement target region. The measuring device according to claim 1.

10. When two or more feature regions are detected on the subject, the control unit selects at least one of the two or more feature regions as the measurement target region. The measuring device according to claim 1.

11. The control unit selects the feature region that has the smallest distance between each of the two or more feature regions and the position of the tip of the endoscope as the measurement target region. The measuring device according to claim 10.

12. The control unit selects at least one feature region as the measurement target region based on the shape characteristics of the two or more feature regions. The measuring device according to claim 10.

13. The control unit selects the at least one feature region as the measurement target region based on the information input to the input device. The measuring device according to claim 10.

14. The control unit sets a point at the tip of the endoscope as the measurement reference. The measuring device according to claim 1.

15. The control unit, The point at the tip of the endoscope is set as the first measurement reference, A second measurement criterion is established that includes one or more of the two or more points mentioned above. A first distance information is generated that indicates the distance between the first measurement reference and the measurement point. A second distance information is generated that indicates the distance between the second measurement reference and the measurement point. The first measurement result generated based on the first distance information is superimposed onto the image of the measurement target area. The second measurement result generated based on the second distance information is superimposed onto the image of the measurement target area. The image with the first measurement result superimposed and the image with the second measurement result superimposed are output to the display. The measuring device according to claim 1.

16. The control unit outputs to the display either the image on which the first measurement result is superimposed or the image on which the second measurement result is superimposed. The display of the image with the first measurement result superimposed and the display of the image with the second measurement result superimposed are switchable. The measuring device according to claim 15.

17. The control unit, The distance between the measurement reference and each of the two or more points is calculated. Based on the distance between the measurement standard and each of the two or more points, select at least one of the two or more points. Information of a feature region that is included in one or more feature regions and contains at least one point is output to the display. The measuring device according to claim 1.

18. Based on endoscopic images of the subject obtained by the endoscope, three-dimensional data is acquired that includes the three-dimensional coordinates of two or more points on the subject. A measurement standard is established to indicate the reference position for measurement. Based on the three-dimensional data or the endoscopic image, one or more feature regions are detected on the subject, and the three-dimensional shape of the subject has common features in each of the one or more feature regions. Select at least one feature region from the one or more feature regions mentioned above as the measurement target region. One or more points included in the measurement target area and included in the two or more points are set as measurement points. Distance information indicating the distance between the measurement reference and the measurement point is generated. The measurement results generated based on the distance information are superimposed onto the image of the measurement target area. The image with the measurement results superimposed is output to the display. Measurement method.

19. A step of acquiring three-dimensional data including the three-dimensional coordinates of two or more points on the subject calculated based on endoscopic images of the subject acquired by an endoscope, A step of setting a measurement reference that indicates the reference position for measurement, The steps include detecting one or more feature regions on the subject based on the three-dimensional data or the endoscopic image, and the three-dimensional shape of the subject having common features in each of the one or more feature regions, The steps include selecting at least one feature region from the one or more feature regions mentioned above as the measurement target region, The steps include setting one or more points that are included in the measurement target area and are included in the two or more points as measurement points, A step of generating distance information indicating the distance between the measurement reference and the measurement point, The steps include generating a graphic to be superimposed on the image of the measurement target area based on the measurement results generated based on the distance information, The steps include outputting the image and graphic of the measurement target area, A program that causes a computer to execute something.

Images