Image processing device, image processing method, and image processing program

Abstract

To provide an image processing device, an image processing method, and an image processing program capable of measuring dimensions of a subject.SOLUTION: An image processing device processes an image acquired by an endoscope, and includes a processor. The processor calculates a subject distance on the basis of a focal position of a focus lens of an endoscope focusing on the subject in the image (S4), calculates dimensional characteristics for defining apparent dimensions of the subject in the image on the basis of optical system information on an imaging optical system of the endoscope when the focus lens is positioned at the focal position (S5), calculates scale information for defining relationships between the apparent dimensions of the subject in the image and actual dimensions on the basis of the subject distance and the dimensional characteristics (S6), and generates a display pattern indicating the actual dimensions of the subject and superimposed on the image on the basis of the scale information (S7).SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] Conventionally, an endoscope system having a function for measuring the dimensions of a subject is known (see, for example, Patent Document 1). The endoscope system described in Patent Document 1 irradiates the subject with measurement auxiliary light from an endoscope to form a spot on the subject, estimates the subject distance from the position of the spot on an imaging element, generates an index figure indicating the actual dimensions of the subject based on the subject distance, and displays the index figure together with an image of the subject. Such an endoscope system allows accurate estimation of the size of the subject without relying on the subjective judgment or skill of the doctor, and also allows the doctor to select a treatment procedure in endoscopic examination and treatment according to the size of the subject, such as a lesion. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6692440 Summary of the Invention [Problem to be solved by the invention]

[0004] The endoscope system of Patent Document 1 requires auxiliary measurement light to measure the dimensions of the subject. Therefore, it is necessary to use a special endoscope equipped with an illumination optical system for the auxiliary measurement light. In addition, such an endoscope is not suitable for making its diameter smaller.

[0005] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an image processing device, an image processing method, and an image processing program that can measure the dimensions of a subject without requiring a special endoscope. [Means for solving the problem]

[0006] One aspect of the present invention is an image processing device that processes images acquired by an endoscope, wherein the imaging optical system of the endoscope has a focus lens that moves the focal point of the endoscope and is equipped with a processor, the processor calculates a subject distance based on a focus position of the focus lens, the focus position being the position of the focus lens when the focus is on the subject in the image, calculates dimensional characteristics that define the apparent dimensions of the subject in the image based on optical system information of the imaging optical system when the focus lens is located at the focus position, calculates scale information that defines the relationship between the apparent dimensions and actual dimensions of the subject in the image based on the subject distance and the dimensional characteristics, and generates a display pattern that represents the actual dimensions of the subject and is superimposed on the image based on the scale information.

[0007] Another aspect of the present invention is an image processing device that processes images acquired by an endoscope, wherein the imaging optical system of the endoscope has a focus lens that moves the focal point of the endoscope, and is equipped with a processor, the processor calculates a subject distance based on a focus position of the focus lens, the focus position being the position of the focus lens when the focus is on the subject in the image, calculates dimensional characteristics that define the apparent dimensions of the subject in the image based on optical system information of the imaging optical system when the focus lens is located at the focus position, generates three-dimensional information of the subject using the image, calculates scale information that defines the relationship between the apparent dimensions and actual dimensions of the subject in the image based on the subject distance, the dimensional characteristics and the three-dimensional information, and generates a display pattern that represents the actual dimensions of the subject and is superimposed on the image based on the scale information.

[0008] Another aspect of the present invention is an image processing method for processing an image acquired by an endoscope, the image processing method including: an imaging optical system of the endoscope having a focus lens that moves the focal point of the endoscope; calculating a subject distance based on a focus position of the focus lens, the focus position being a position of the focus lens when the focus is on the subject in the image; calculating dimensional characteristics defining apparent dimensions of the subject in the image based on optical system information of the imaging optical system when the focus lens is located at the focus position; calculating scale information defining a relationship between the apparent dimensions and actual dimensions of the subject in the image based on the subject distance and the dimensional characteristics; and generating a display pattern representing the actual dimensions of the subject and being superimposed on the image based on the scale information.

[0009] Another aspect of the present invention is an image processing method for processing an image acquired by an endoscope, the image processing method including: an imaging optical system of the endoscope having a focus lens that moves the focal point of the endoscope; calculating a subject distance based on a focus position of the focus lens, the focus position being a position of the focus lens when the focus is on the subject in the image; calculating dimensional characteristics defining apparent dimensions of the subject in the image based on optical system information of the imaging optical system when the focus lens is located at the focus position; generating three-dimensional information of the subject using the image; calculating scale information defining a relationship between the apparent dimensions and actual dimensions of the subject in the image based on the subject distance, the dimensional characteristics and the three-dimensional information; and generating a display pattern representing the actual dimensions of the subject and being superimposed on the image based on the scale information.

[0010] Another aspect of the present invention is an image processing program that causes a computer to execute an image processing method for processing an image acquired by an endoscope, wherein the imaging optical system of the endoscope has a focus lens that moves the focal point of the endoscope, and the image processing method includes: calculating a subject distance based on a focus position of the focus lens, the focus position being a position of the focus lens when the focus is on the subject in the image, calculating dimensional characteristics that define the apparent dimensions of the subject in the image based on optical system information of the imaging optical system when the focus lens is located at the focus position, calculating scale information that defines the relationship between the apparent dimensions and actual dimensions of the subject in the image based on the subject distance and the dimensional characteristics, and generating a display pattern that represents the actual dimensions of the subject and is superimposed on the image based on the scale information.

[0011] Another aspect of the present invention is an image processing program that causes a computer to execute an image processing method for processing an image acquired by an endoscope, wherein the imaging optical system of the endoscope has a focus lens that moves the focal point of the endoscope, and the image processing method includes: calculating a subject distance based on a focal position of the focus lens, the focal position being a position of the focus lens when the focus is on the subject in the image, calculating dimensional characteristics that define the apparent dimensions of the subject in the image based on optical system information of the imaging optical system when the focus lens is located at the focal position, generating three-dimensional information of the subject using the image, calculating scale information that defines the relationship between the apparent dimensions and actual dimensions of the subject in the image based on the subject distance, the dimensional characteristics and the three-dimensional information, and generating a display pattern that represents the actual dimensions of the subject and is superimposed on the image based on the scale information. Effect of the Invention

[0012] According to the present invention, it is possible to obtain an effect that the dimensions of a subject can be measured without requiring a special endoscope. [Brief description of the drawings]

[0013] [Figure 1] 1 is an overall configuration diagram of an image processing device and an endoscope system according to a first embodiment. [Diagram 2] FIG. 2 is a functional block diagram of a processor of the image processing device. [Figure 3A] FIG. 13 is a diagram showing an example of a scale display superimposed on an image. [Figure 3B] FIG. 13 is a diagram showing another example of a scale display superimposed on an image. [Figure 3C] FIG. 13 is a diagram showing another example of a scale display superimposed on an image. [Figure 4] 1 is a flowchart of an image processing method executed by the image processing device. [Figure 5A] FIG. 13 is a diagram showing an example of a scale display in which the size of the display pattern is variable, showing the case where the subject distance is long. [Figure 5B] FIG. 13 is a diagram showing an example of a scale display in which the size of the display pattern is variable, showing the case where the subject distance is short. [Figure 6A] FIG. 13 is a diagram showing another example of a scale display in which the size of the display pattern is variable, showing the case where the subject distance is long. [Figure 6B] FIG. 13 is a diagram showing another example of a scale display in which the size of the display pattern is variable, showing the case where the subject distance is short. [Figure 7A] FIG. 13 is a diagram showing an example of a scale display in which the size of the display pattern is fixed and the numerical values ​​are variable, showing the case where the subject distance is long. [Figure 7B] FIG. 13 is a diagram showing an example of a scale display in which the size of the display pattern is fixed and the numerical values ​​are variable, and shows the case where the subject distance is short. [Figure 8A] FIG. 13 is a diagram showing another example of a scale display in which the size of the display pattern is fixed and the numerical values ​​are variable, showing the case where the subject distance is long. [Figure 8B] FIG. 13 is a diagram showing another example of a scale display in which the size of the display pattern is fixed and the numerical values ​​are variable, showing the case where the subject distance is short. [Figure 9A] FIG. 2 is a diagram illustrating a plurality of distance measurement areas in an image. [Figure 9B] FIG. 1 shows a distorted image with a scale indication superimposed on the center and periphery. [Figure 10A] FIG. 13 is a diagram showing an example of an image in which a scale display is superimposed on a subject in the foreground. [Figure 10B] FIG. 13 is a diagram showing an example of an image in which a scale display is superimposed on a subject in the background. [Figure 11A] FIG. 13 is a diagram showing another example of an image in which a scale display is superimposed on a subject in the foreground. [Figure 11B] FIG. 13 is a diagram showing another example of an image in which a scale display is superimposed on a subject in the background. [Figure 12A] FIG. 13 is a diagram showing another example of an image in which a scale display is superimposed on a subject in the foreground. [Figure 12B] FIG. 13 is a diagram showing another example of an image in which a scale display is superimposed on a subject in the background. [Figure 13A] FIG. 13 is a diagram showing another example of an image in which a scale display is superimposed on a foreground subject. [Figure 13B] FIG. 13 is a diagram showing another example of an image in which a scale display is superimposed on a subject in the background. [Figure 14A] FIG. 13 is a diagram showing another example of an image in which a scale display is superimposed on a foreground subject. [Figure 14B] FIG. 13 is a diagram showing another example of an image in which a scale display is superimposed on a subject in the background. [Figure 15] FIG. 11 is a functional block diagram of a processor of an image processing device according to a second embodiment. [Figure 16] 1 is a flowchart of an example of a method for generating 3D information. [Figure 17A] FIG. 13 is a diagram showing an example of an image focused on a subject in the foreground and with a scale display superimposed in the center. [Figure 17B] FIG. 13 is a diagram showing an example of an image in which a subject in the background is focused and a scale display is superimposed in the center. [Figure 18A] FIG. 13 is a diagram showing an example of an image on which a scale display is superimposed when a measurement point is set on a subject in the foreground. [Figure 18B] FIG. 13 is a diagram showing an example of an image on which a scale display is superimposed when a measurement point is set on a subject at the back. [Figure 19] 10 is a flowchart of an image processing method according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] (First embodiment) An image processing device, an image processing method, and an image processing program according to a first embodiment of the present invention will be described with reference to the drawings. As shown in FIG. 1, an image processing device 10 according to this embodiment is applied to an endoscope system 100. The endoscope system 100 includes an image processing device 10, an endoscope 20, and a display device 30. The image processing device 10 is connected to the endoscope 20 and the display device 30.

[0015] The endoscope 20 has an imaging optical system 21 and an imager 22. The imaging optical system 21 has an objective lens 23 at the tip of the endoscope 20, and an autofocus (AF) lens 24 between the objective lens 23 and the imager 22. The imager 22 has an image sensor such as a CMOS image sensor, and captures an image of the subject S through the imaging optical system 21 to generate an image. The image output from the imager 22 is input to the display device 30 through the image processing device 10 and is displayed on the display device 30. The display device 30 is any type of display device such as a liquid crystal display.

[0016] The AF lens 24 is movable along the optical axis A of the objective lens 23, and the movement of the AF lens 24 causes the focal point F of the objective lens 23 to move on the optical axis A. The endoscope 20 can perform an AF operation to move the AF lens 24 along the optical axis A using an actuator (not shown) to focus on the subject S. The endoscope 20 may always perform the AF operation automatically, or may perform the AF operation in accordance with an external control signal. The endoscope 20 may also perform the AF operation to focus on a position in an image designated by a user.

[0017] The image processing device 10 includes a processor 1 such as a central processing unit, a storage unit 2, a memory 3, and a user interface 4. The storage unit 2 is a computer-readable non-transitory recording medium, such as a known magnetic disk, optical disk, flash memory, etc. The storage unit 2 stores an image processing program 2a that causes the processor 1 to execute an image processing method, which will be described later.

[0018] The memory 3 is made up of a volatile storage device such as a RAM (random access memory), and is used as a working area for the processor 1 . The user interface 4 has input devices such as a mouse, a keyboard, a touch panel, a microphone, etc. A user can input various information, instructions, etc. to the image processing device 10 by operating the user interface 4.

[0019] 2 shows a functional block diagram of the processor 1. The processor 1 has, as its functions, an image processing unit 11, an evaluation value calculation unit 12, a lens position reading unit 13, a subject distance estimation unit 14, a dimensional characteristic calculation unit 15, a scale information calculation unit 16, and a scale display generation unit 17. The image processing unit 11 acquires an image input from the endoscope 20 to the image processing device 10, performs image processing such as missing pixel correction, white balance adjustment, noise reduction, and demosaicing on the image, and then outputs the image to the display device 30.

[0020] The evaluation value calculation unit 12, lens position readout unit 13, subject distance estimation unit 14, dimensional characteristic calculation unit 15, scale information calculation unit 16, and scale display generation unit 17 execute the processes described below during execution of a measurement mode for measuring the dimensions of the subject S. The processor 1 executes the measurement mode in response to, for example, a start instruction. The start instruction is input to the image processing device 10 by, for example, a user operating the user interface 4 at a desired timing.

[0021] The evaluation value calculation unit 12 sequentially acquires images during AF operation input from the endoscope 20 to the image processing device 10, and sequentially calculates an AF evaluation value indicating the degree of focus of the image using the images. For example, the AF evaluation value is the contrast of the image, and the contrast reaches its peak when the subject S is in focus. The evaluation value calculation unit 12 may calculate the AF evaluation value by using phase difference information instead of an image. In this case, the imager 22 has pixels for detecting an image plane phase difference, and the phase difference information is input from the endoscope 20 to the image processing device 10. The lens position reading unit 13 acquires driving information of the AF lens 24, including the position of the AF lens 24 during AF operation, from the endoscope 20, and reads out the position of the AF lens 24 from the driving information.

[0022] The subject distance estimation unit 14 determines the in-focus position of the AF lens 24 based on the AF evaluation value, and estimates the subject distance based on the in-focus position. The in-focus position is the position of the AF lens 24 when the focal point F is aligned with the subject S. The subject distance is the actual distance from the objective lens 23 to the subject S in the direction along the optical axis A.

[0023] Specifically, the subject distance estimation unit 14 determines the AF evaluation value at the time of focusing from among the AF evaluation values ​​during AF operation. For example, when the AF evaluation value is the contrast of an image, the subject distance estimation unit 14 determines the peak contrast. Next, the subject distance estimation unit 14 determines the position of the AF lens 24 at the time of acquiring the image for which the AF evaluation value at the time of focusing was calculated as the focusing position. Next, the subject distance estimation unit 14 estimates the subject distance from the focusing position using a LUT (Look Up Table). The LUT represents the relationship between the focusing position and the subject distance, and is stored in advance in the storage unit 2.

[0024] The dimensional characteristic calculation unit 15 acquires optical system information of the imaging optical system 21 from the endoscope 20, and calculates the dimensional characteristics of the subject S in the image based on the optical system information. The optical system information is information on the optical characteristics of the imaging optical system 21, and includes the observation magnification, the amount of distortion, the optical axis center position, etc. The optical system information changes depending on the position of the AF lens 24, so the dimensional characteristic calculation unit 15 acquires the optical system information in a state where the AF lens 24 is disposed at each position. The dimensional characteristic calculation unit 15 may automatically acquire the optical system information before the measurement mode is executed, for example, when the endoscope 20 is connected to the image processing device 10.

[0025] Next, the dimensional characteristic calculation unit 15 calculates the dimensional characteristic based on the optical system information in the in-focus state where the AF lens 24 is located at the in-focus position. For example, in the case of a wide-angle objective lens 23, the apparent dimensions of the subject S in the image vary depending on the position due to peripheral distortion occurring in the image, a shift in the optical axis center position, etc., and for example, the apparent dimensions become smaller from the center to the periphery (see FIG. 9B). The dimensional characteristic defines the apparent dimensions of the subject S at each position in such an image.

[0026] The scale information calculation unit 16 calculates scale information based on the subject distance and dimensional characteristics. The scale information is information that defines the relationship between the apparent dimensions and the actual dimensions of the subject S in the image, and is, for example, information that represents the relationship between the apparent distance from the optical axis center and the actual distance from the optical center in the image. The scale information calculation unit 16 calculates the actual dimensions of the focused subject S based on the subject distance, and calculates scale information based on the actual dimensions and dimensional characteristics of the subject S.

[0027] The scale display generating unit 17 generates a scale display 5, which is a virtual scale to be superimposed on the image, based on the scale information. Figures 3A and 3B show an example of the scale display 5. The scale display 5 includes a display pattern 6 consisting of a graphic representing the actual dimensions of the subject S, and a numerical value 7 of the actual dimensions that is added to the display pattern 6. The processor 1 outputs the image B together with the scale display 5 to the display device 30, and causes the display device 30 to display the image B with the scale display 5 superimposed thereon.

[0028] The display pattern 6 in Fig. 3A is a concentric circle made up of multiple circles, each having a different radius and a center that is located at the center of image B. The radius of each circle represents the actual size of subject S, and each circle is given a numerical value 7 of the actual size (5mm, 10mm, 15mm).

[0029] The display pattern 6 in Fig. 3B is a linear scale with an origin located at the center of image B. The distance from the origin represents the actual size of subject S, and each scale line is labeled with a numerical value 7 representing the actual size (5 mm, 10 mm, 15 mm). In Fig. 3B, the arrangement of the scale lines is symmetrical with respect to the origin. 3C, the arrangement of the scale lines may be asymmetric with respect to the origin, which allows the size of the subject S to be measured more accurately.

[0030] Next, an image processing method executed by the image processing device 10 will be described. As shown in FIG. 4, the image processing method according to this embodiment includes step S3 of acquiring information required during AF operation of the endoscope 20, step S4 of estimating the subject distance, step S5 of calculating dimensional characteristics of the subject S in the image, step S6 of calculating scale information, and step S7 of generating a scale display 5.

[0031] The processor 1 starts the measurement mode in response to, for example, a start instruction (step S1), and executes steps S3 to S7. Before step S3, the processor 1 may set a display mode based on a user's operation of the user interface 4 (step S2). The display mode setting includes the format of the scale display 5. For example, the user can select the format from concentric circles and linear scales.

[0032] Next, the processor 1 acquires information required for generating the scale display 5, specifically, images, driving information, and optical system information, from the endoscope 20 during the AF operation of the endoscope 20 (step S3). In step S3, the processor 1 may cause the endoscope 20 to perform the AF operation by transmitting a control signal to the endoscope 20. Alternatively, the processor 1 may acquire the information while the endoscope 20 is automatically performing the AF operation.

[0033] Next, the processor 1 estimates the subject distance based on the image and the driving information (step S4). Specifically, the evaluation value calculation unit 12 calculates the AF evaluation value based on the image, and at the same time, the lens position reading unit 13 reads the position of the AF lens 24 from the driving information. Next, the subject distance estimation unit 14 determines the focus position of the AF lens 24 based on the AF evaluation value, and estimates the subject distance based on the focus position.

[0034] Next, the dimensional characteristic calculation unit 15 calculates the dimensional characteristic of the subject S in the image based on the optical system information of the imaging optical system 21 in the focused state (step S5). Next, the scale information calculation unit 16 calculates the scale information based on the dimensional characteristics and the subject distance (step S6).

[0035] Next, the scale display generating unit 17 generates the scale display 5 according to the set display mode based on the scale information (step S7). The scale indication 5 is output from the image processing device 10 to the display device 30 together with the image B, and the image B with the scale indication 5 superimposed thereon is displayed on the display device 30. A user such as a doctor can measure the actual dimensions of the subject S, such as an affected area, based on the scale indication 5 on the image.

[0036] After step S7, the processor 1 may determine whether to switch the display mode (step S8) and whether to end the measurement mode (step S9). For example, when the user wants to change the display mode, the user operates the user interface 4 to select the desired display mode. The processor 1 switches the display mode based on the operation of the user interface 4 (YES in step S8, NO in step S9, step S2), and repeats steps S3 to S7. If the display mode is not to be switched (NO in step S8), the processor 1 returns to step S3. Furthermore, when the user wishes to end the measurement mode, the user operates the user interface 4 to input an end instruction to the image processing device 10. In response to the end instruction, the processor 1 ends the measurement mode (YES in step S9).

[0037] To calculate the scale information, the subject distance, which is information on the actual dimensions of the subject S in the image, is required. According to this embodiment, the subject distance is calculated from the focal position of the AF lens 24. Therefore, the image processing device 10 can be combined with a general small-diameter endoscope 20 having a focus lens such as the AF lens 24, and can measure the actual dimensions of the subject S without requiring a special endoscope such as an endoscope equipped with an illumination optical system for measurement auxiliary light.

[0038] In this embodiment, the scale indication 5 may be variable depending on the subject distance. Specifically, the processor 1 may change the size of the display pattern 6 depending on the subject distance (see Figures 5A to 6B), or may generate a display pattern 6 of a fixed size and change the numerical value 7 depending on the subject distance (see Figures 7A to 8B). In this case, in step S2, the display mode may be set to select whether the size of the display pattern 6 is variable or fixed.

[0039] 5A and 5B, the radius of each of the concentric circles changes according to the estimated subject distance. Specifically, the shorter the subject distance, the larger the radius, which allows the user to recognize that the distance to the subject S is close. 6A and 6B, the interval between the scale lines changes depending on the estimated subject distance. Specifically, the shorter the subject distance, the larger the interval between the scale lines, which allows the user to recognize that the distance to the subject S is close.

[0040] 7A to 8B, the shorter the subject distance, the smaller the numerical value 7, thereby allowing the user to recognize that the distance to the subject S is close. The radius of each circle or the interval between the scale lines is constant regardless of the subject distance.

[0041] In this embodiment, one scale indication 5 is arranged in the center of image B. However, instead, one or more scale indications 5 may be arranged in other positions within image B. For example, as shown in Figures 9A and 9B, a plurality of distance measuring areas C arranged in the horizontal and vertical directions are set in an image B, and scale displays 51 and 52 are displayed on each of one or more focused distance measuring areas C. Figure 9B shows an image B of a plurality of circles arranged at equal intervals in the horizontal and vertical directions and having the same diameter. The circles on the periphery shrink and become ellipses due to distortion.

[0042] When image B has distortion as in FIG. 9B, scale indication generation unit 17 may generate scale indications 51, 52 by taking the distortion into consideration. In the example of FIG. 9B, scale indications 51, 52 are single circles or ellipses, and the radius or the major and minor radii represent the actual dimensions of subject S. Scale indication 51 in the center, which has no distortion, is a circle. Scale indication 52 in the lower left, which has distortion, is an ellipse tilted in the direction of the distortion, due to the circle being distorted in response to the distortion. The two scale indications 51, 52 represent the same actual dimensions.

[0043] In this case, the subject distance estimation unit 14 selects at least one focused distance measurement area C, and estimates the subject distance of the selected distance measurement area C. The scale indication generation unit 17 generates scale indications 51, 52 for the selected distance measurement area C. Here, the subject distance estimation unit 14 may select the focused ranging area C based on the AF evaluation value of each ranging area C. In this case, the evaluation value calculation unit 12 calculates the AF evaluation value of each ranging area C. Alternatively, the subject distance estimation unit 14 may select a ranging area including a position specified by the user as the focused ranging area C. In this case, the processor 1 causes the endoscope 20 to perform an AF operation so as to focus on the selected ranging area C.

[0044] 10A to 13B show an image B of the inside of a lumen extending from the lower left to the upper right, such as the large intestine, in which a plurality of subjects Sa and Sb, such as tumors, are present at different positions in the depth direction. Figures 10A to 11B show the case of a concentric scale display 5, in which the size of the display pattern 6 is fixed in Figures 10A and 10B, and the size of the display pattern 6 is variable in Figures 11A and 11B. In Figures 10A to 11B, the display pattern 6 becomes an inclined concentric ellipse due to the concentric circles being distorted in accordance with the distortion of image B. 12A to 13B show the case of a linear scale display 5, where the size of the display pattern 6 is fixed in FIGS. 12A and 12B, and the size of the display pattern 6 is variable in FIGS. 13A and 13B.

[0045] 10A, 11A, 12A, and 13A, the camera focuses on a subject Sa in the foreground, either automatically or based on a user's designation, and a scale display 5 is superimposed on the subject Sa. In Fig. 10B, 11B, 12B, and 13B, the camera focuses on a subject Sb in the background, either automatically or based on a user's designation, and a scale display 5 is superimposed on the subject Sb.

[0046] In Figures 12A to 13B, the linear scale is parallel to the horizontal direction of image B. Alternatively, as in Figures 14 and 14B, the linear scale may be inclined in the direction of distortion. In the case of Figures 14A and 14B, since the actual distance from the origin is equal on the left and right, it is possible to generate a linear scale that is symmetrical with respect to the origin (i.e., the interval between the scale lines and the value 7 are the same on both sides of the origin).

[0047] Second embodiment Next, an image processing device, an image processing method, and an image processing program according to a second embodiment of the present invention will be described. This embodiment differs from the first embodiment in that scale information is calculated using three-dimensional information of the object in addition to the object distance and dimensional characteristics. In this embodiment, configurations different from the first embodiment will be described, and configurations common to the first embodiment will be denoted by the same reference numerals and description thereof will be omitted.

[0048] The image processing device 10 according to this embodiment is applied to an endoscope system 100 including an endoscope 20 and a display device 30, as in the first embodiment, and includes a processor 101, a storage unit 2, a memory 3, and a user interface 4.

[0049] FIG. 15 is a functional block diagram of the processor 101 in this embodiment. The processor 101 has as its functions an image processing unit 11, an evaluation value calculation unit 12, a lens position reading unit 13, a subject distance estimation unit 14, a dimensional characteristic calculation unit 15, a scale information calculation unit 16, and a scale display generation unit 17, as well as a three-dimensional (3D) information generation unit 18.

[0050] The 3D information generator 18 generates 3D information of the subject S using multiple images with different viewpoints and in focus. The 3D information is a three-dimensional model of the subject S and is generated using a known three-dimensional reconstruction technique. The positional relationship between the 3D information and each image used to generate the 3D information is known. The 3D information is defined by the relative dimensions of the subject S and does not include information on the actual dimensions of the subject S.

[0051] FIG. 16 shows an example of a method in which the 3D information generator 18 generates 3D information. First, a plurality of images for generating 3D information are acquired from images sequentially input to the image processing device 10 (step S101). Next, feature points are extracted from each image (step S102). It is preferable that the feature points are robust against changes in image magnification, rotation, lighting, etc., and for example, SIFT (Scale-invariant feature transform), ORB (Oriented The information is extracted using techniques such as AKAZE (Accelerated KAZE) and FAST and Rotated BRIEF.

[0052] Next, matching between the images is performed based on the feature amounts of the respective feature points (step S103). Next, mismatching feature points are removed from the matched feature points using geometric constraints (step S104). A known method such as the RANSAC (Random Sample Consensus) algorithm is used to remove the mismatching feature points. Next, a change in the camera pose is estimated based on the change in the matched feature points (step S105). For example, if the internal parameters of the camera are known, a fundamental matrix is ​​estimated using five feature points, and if the internal parameters are unknown, a fundamental matrix is ​​estimated using eight or more feature points, thereby estimating the camera pose.

[0053] Next, the three-dimensional position of each matched feature point is estimated by triangulation based on the estimated fundamental matrix or fundamental matrix, thereby generating a three-dimensional point cloud (step S106). Next, the camera pose and the 3D point cloud are adjusted (bundle adjustment is performed) by the nonlinear least squares method so as to minimize the error when the point cloud is reprojected onto the image (step S107).

[0054] The subject distance estimating unit 14 determines the in-focus position of the AF lens 24 based on the AF evaluation value, and estimates the subject distance based on the in-focus position. Here, when the subjects are distributed in the depth direction of the image, the focus is not necessarily on the center of the image, but may be on another position. For example, as shown in Fig. 17A, the focus may be on the subject Sa in the foreground, or as shown in Fig. 17B, the focus may be on the subject Sb in the background. Point P indicates the focus position where the subjects Sa and Sb are focused. The subject distance estimating unit 14 detects a focus position P in the image B based on, for example, the AF evaluation values ​​of each of a plurality of distance measuring areas C in the image B, and estimates the subject distance at the focus position P.

[0055] The scale information calculation unit 16 calculates scale information at a measurement point Q based on the subject distance, dimensional characteristics, and 3D information at the focusing position P. The measurement point Q is a predetermined arbitrary position within the image B. For example, as shown in FIGS. 17A and 17B, the measurement point Q is initially set to the center of the image B.

[0056] When measurement point Q is the center of image B, scale information calculation unit 16 estimates actual distance Z_Rd between the center of image B and focus position P from the subject distance and 3D information using the following formula (1). Here, Zo is the center position of the image in the 3D information, Zf is the focus position P in the 3D information, and Z_Rf is the subject distance at focus position P estimated by subject distance estimation unit 14. Z_Rd = |Zf - Zo| × Z_Rf / Zf (1) Next, the scale information calculation unit 16 generates scale information based on the distance Z_Rd and the dimensional characteristics. The scale display generating unit 17 generates a scale display 5 that represents the actual dimension at the measurement point Q and is superimposed on the measurement point Q (for example, the center) in the image B based on the scale information.

[0057] The scale information calculation unit 16 may set a position in image B designated by the user as the measurement point Q. In this case, as shown in Fig. 18A and Fig. 18B, the user designates an arbitrary position in image B by operating the user interface 4. In Fig. 18A and Fig. 18B, a subject Sa in the foreground and a subject Sb in the background are designated, respectively. In this case, the scale information calculation unit 16 estimates the actual distance between the measurement point Q and the focusing position P by using the position of the measurement point Q in the 3D information instead of the center position Zo in equation (1). Next, the scale information calculation unit 16 generates scale information that defines the relationship between the apparent distance from the measurement point Q and the actual distance from the measurement point Q based on the estimated actual distance and dimensional characteristics.

[0058] Based on the scale information, the scale indication generating unit 17 generates a scale indication 5 indicating an actual dimension at the measurement point Q. The generated scale indication 5 is superimposed on the measurement point Q, for example, the subject Sa or the subject Sb. The scale mark 5 is not limited to concentric circles, and may be other scale mark 5 described in the first embodiment (see Figs. 3A, 3B, and 10A to 14B). The size of the display pattern 6 may change according to the subject distance (see Figs. 5A to 6B), or the size of the display pattern 6 may be fixed while the numerical value 7 changes according to the subject distance (see Figs. 7A to 8B). Furthermore, the scale mark 5 may be distorted according to the distortion of the image B (see Figs. 10A to 11B, 14A, and 14B).

[0059] Next, an image processing method executed by the image processing device 10 of this embodiment will be described. As shown in FIG. 19, the image processing method according to this embodiment includes, in addition to steps S3 to S7, step S10 of generating 3D information. After step S3, the subject distance estimation unit 14 estimates the subject distance at the in-focus position P in the image B (step S4). In parallel with this, the 3D information generation unit 18 generates 3D information (step S10).

[0060] Subsequently, after calculating the dimensional characteristics (step S5), the scale information calculation unit 16 calculates scale information at a predetermined measurement point Q in the image B, for example, at the center, based on the dimensional characteristics, the subject distance, and the 3D information (step S6), and the scale display generation unit 17 generates a scale display 5 representing the actual dimensions of the subject at the measurement point Q based on the scale information (step S7). As a result, the image B with the scale display 5 superimposed on the measurement point Q is displayed on the display device 30. The user may place the subject to be measured at measurement point Q in image B, for example at the center, and then start the measurement mode. This allows the user to easily measure the actual dimensions of the subject based on the scale display 5 of measurement point Q in image B.

[0061] When it is desired to measure the dimensions of the subject at another position, the user may specify the desired position in image B by operating user interface 4. In this case, scale information calculation unit 16 calculates scale information at the specified position (step S6), and scale display generation unit 17 generates scale display 5 for the specified position (step S7). As a result, image B with scale display 5 superimposed at the specified position is displayed on display device 30.

[0062] Thus, according to this embodiment, scale information is calculated based on 3D information in addition to the subject distance and optical characteristics. Since the 3D information has information on the overall relative dimensions of the subject in the image, it is possible to calculate information on the actual dimensions of the subject at a measurement point Q at a position different from the focus position P, for example, the actual distance from the focus position P to the measurement point Q, from a combination of the subject distance and the 3D information. Then, from the information on the actual dimensions and the dimensional characteristics, scale information is calculated for the measurement point Q at an arbitrary position in the image B, and a scale display 5 is generated. This makes it possible to generate scale display 5 at any position in image B, regardless of in-focus position P in image B. In other words, when it is desired to measure the dimensions of a subject at a position other than in-focus position P, there is no need to perform AF operation to focus on the subject.

[0063] Furthermore, according to this embodiment, scale information at multiple positions in an image can be calculated simultaneously, and multiple scale displays 5 can be generated simultaneously. Therefore, multiple measurement points Q may be set in image B. For example, a user may specify multiple positions simultaneously, and multiple scale displays 5 may be superimposed simultaneously on the specified multiple positions. According to this configuration, for example, the scale display 5 of FIG. 18A and the scale display 5 of FIG. 18B can be superimposed simultaneously on image B.

[0064] Although the embodiment of the present invention and its modified examples have been described in detail above with reference to the drawings, the specific configuration of the present invention is not limited to the above embodiment and modified examples, and various design modifications are possible within the scope of the gist of the present invention. Furthermore, the components shown in the above embodiment and modified examples can be appropriately combined. For example, the scale indication 5 is not limited to the concentric circles or linear scale described above, but may be in any form that represents the actual dimensions of the subject S. [Explanation of symbols]

[0065] 1,101 processors 2. Memory unit (recording medium) 2a Image processing program 5,51,52 Scale display 6 Display Pattern 7 Numbers 10 Image processing device 20 Endoscopy 21 Imaging optical system 24 Autofocus lens (focus lens) F focus S,Sa,Sb Subject Q Measurement point

Claims

1. An image processing device for processing an image acquired by an endoscope, the image pickup optical system of the endoscope having a focus lens for moving a focal point of the endoscope, A processor is provided. The processor, calculating a subject distance based on a focal position of the focus lens, the focal position being a position of the focus lens when the subject in the image is in focus; calculating dimensional characteristics that define apparent dimensions of the subject in the image based on optical system information of the imaging optical system when the focus lens is located at the in-focus position; calculating scale information defining a relationship between apparent and actual dimensions of an object in the image based on the object distance and the dimensional characteristics; The image processing device generates a display pattern representing the actual dimensions of the subject based on the scale information and to be superimposed on the image.

2. The image processing device according to claim 1 , wherein the processor changes a size of the display pattern depending on the subject distance.

3. The image processing device according to claim 1 , wherein the processor generates the display pattern of a fixed size, and changes the numerical value of the actual size assigned to the display pattern according to the subject distance.

4. The image processing device according to claim 1 , wherein the display pattern is a plurality of concentric circles having mutually different radii.

5. The image processing device according to claim 1 , wherein the display pattern is a linear scale.

6. An image processing device for processing an image acquired by an endoscope, the image pickup optical system of the endoscope having a focus lens for moving a focal point of the endoscope, A processor is provided. The processor, calculating a subject distance based on a focal position of the focus lens, the focal position being a position of the focus lens when the subject in the image is in focus; calculating dimensional characteristics that define apparent dimensions of the subject in the image based on optical system information of the imaging optical system when the focus lens is located at the in-focus position; generating three-dimensional information of the subject using the image; calculating scale information that defines a relationship between apparent and actual dimensions of the object in the image based on the object distance, the dimensional characteristics, and the three-dimensional information; The image processing device generates a display pattern representing the actual dimensions of the subject based on the scale information and to be superimposed on the image.

7. The processor, calculating the scale information that defines a relationship between apparent dimensions and actual dimensions of the subject at the measurement points in the image; The image processing apparatus according to claim 6 , further comprising: generating the display pattern, which represents an actual dimension of the subject at the measurement point and is superimposed on the measurement point, based on the scale information.

8. The image processing device according to claim 7 , wherein the processor sets a position in the image designated by a user as the measurement point.

9. The image processing device according to claim 6 , wherein the processor changes a size of the display pattern depending on the subject distance.

10. The image processing device according to claim 6 , wherein the processor generates the display pattern of a fixed size, and changes the numerical value of the actual size assigned to the display pattern according to the subject distance.

11. 7. The image processing device according to claim 6, wherein the display pattern is a plurality of concentric circles having mutually different radii.

12. The image processing device according to claim 6 , wherein the display pattern is a linear scale.

13. An image processing method for processing an image acquired by an endoscope, the image pickup optical system of the endoscope having a focus lens that moves a focal point of the endoscope, calculating a subject distance based on a focal position of the focus lens, the focal position being a position of the focus lens when the subject in the image is in focus; calculating dimensional characteristics that define apparent dimensions of the subject in the image based on optical system information of the imaging optical system when the focus lens is located at the in-focus position; calculating scale information defining a relationship between apparent and actual dimensions of an object in the image based on the object distance and the dimensional characteristics; generating a display pattern representing an actual size of the object based on the scale information, the display pattern being superimposed on the image.

14. An image processing method for processing an image acquired by an endoscope, the image pickup optical system of the endoscope having a focus lens that moves a focal point of the endoscope, calculating a subject distance based on a focal position of the focus lens, the focal position being a position of the focus lens when the subject in the image is in focus; calculating dimensional characteristics that define apparent dimensions of the subject in the image based on optical system information of the imaging optical system when the focus lens is located at the in-focus position; generating three-dimensional information of the subject using the image; calculating scale information that defines a relationship between apparent and actual dimensions of the object in the image based on the object distance, the dimensional characteristics, and the three-dimensional information; generating a display pattern representing an actual size of the object based on the scale information, the display pattern being superimposed on the image.

15. An image processing program for causing a computer to execute an image processing method for processing an image acquired by an endoscope, the image capturing optical system of the endoscope having a focus lens for moving a focal point of the endoscope, The image processing method comprises: calculating a subject distance based on a focal position of the focus lens, the focal position being a position of the focus lens when the subject in the image is in focus; calculating dimensional characteristics that define apparent dimensions of the subject in the image based on optical system information of the imaging optical system when the focus lens is located at the in-focus position; calculating scale information defining a relationship between apparent and actual dimensions of an object in the image based on the object distance and the dimensional characteristics; generating a display pattern representing an actual size of the subject based on the scale information and to be superimposed on the image.

16. An image processing program for causing a computer to execute an image processing method for processing an image acquired by an endoscope, the image capturing optical system of the endoscope having a focus lens for moving a focal point of the endoscope, The image processing method comprises: calculating a subject distance based on a focal position of the focus lens, the focal position being a position of the focus lens when the subject in the image is in focus; calculating dimensional characteristics that define apparent dimensions of the subject in the image based on optical system information of the imaging optical system when the focus lens is located at the in-focus position; generating three-dimensional information of the subject using the image; calculating scale information that defines a relationship between apparent and actual dimensions of the object in the image based on the object distance, the dimensional characteristics, and the three-dimensional information; generating a display pattern representing an actual size of the subject based on the scale information and to be superimposed on the image.

Images