Image processing apparatus, image processing method, and recording medium

Abstract

The invention provides an image processing apparatus, an image processing method, and a recording medium. The image processing device processes an image acquired by an endoscope, and includes a processor. The processor calculates a subject distance on the basis of a focus position of a focus lens of the endoscope when the subject in the image is focused, and calculates a size characteristic defining an apparent size of the subject in the image on the basis of optical system information of an imaging optical system of the endoscope when the focus lens is in the focus position. Scale information defining a relationship between an apparent size and an actual size of a subject in an image is calculated on the basis of a subject distance and a size characteristic, and a display pattern representing the actual size of the subject and superimposed on the image is generated on the basis of the scale information.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method, and a recording medium. Background Art

[0002] Conventionally, endoscope systems having a function of measuring the size of a subject are known (for example, see Patent Document 1). The endoscope system described in Patent Document 1 irradiates a subject with auxiliary measurement light from an endoscope to form a light spot on the subject, estimates the subject distance based on the position of the light spot on an imaging element, generates an indicator graphic indicating the actual size of the subject based on the subject distance, and displays the indicator graphic together with the image of the subject.

[0003] According to such an endoscope system, it is possible to estimate the accurate size of the object without relying on the subjective judgment and skills of the doctor. In addition, the doctor can select a treatment method according to the size of the object such as the lesion during endoscopic examination and treatment.

[0004] Prior art literature

[0005] Patent Literature

[0006] [Patent Document 1] Japanese Patent No. 6692440 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] The endoscope system of Patent Document 1 requires auxiliary light for measuring the size of an object. Therefore, a special endoscope equipped with an illumination optical system for measuring auxiliary light is required. Furthermore, such an endoscope is not suitable for reducing its diameter.

[0009] The present invention has been made in view of the above-mentioned circumstances, and an object thereof is to provide an image processing device, an image processing method, and a recording medium capable of measuring the size of a subject without requiring a special endoscope.

[0010] Means for solving problems

[0011] One embodiment of the present invention is an image processing device that processes an image obtained by an endoscope, wherein the imaging optical system of the endoscope has a focus lens that moves the focus of the endoscope, and the image processing device has a processor that calculates the distance to the subject based on the focus position of the focus lens, the focus position being the position of the focus lens when the focus is aligned with the subject in the image, the processor calculates a size characteristic based on optical system information of the imaging optical system when the focus lens is in the focus position, wherein the size characteristic is used to define the apparent size of the subject in the image, the processor uses the image to generate three-dimensional information of the subject, the processor calculates scale information based on the subject distance, the size characteristic, and the three-dimensional information, wherein the scale information is used to define the relationship between the apparent size and actual size of the subject in the image, and the processor generates a display pattern that represents the actual size of the subject and is superimposed on the image based on the scale information.

[0012] Another embodiment of the present invention is an image processing device that processes an image obtained by an endoscope, wherein the imaging optical system of the endoscope has a focus lens that moves the focus of the endoscope, and the image processing device has a processor that calculates the distance to the subject based on the focus position of the focus lens, the focus position being the position of the focus lens when the focus is aligned with the subject in the image, and the processor calculates a size characteristic based on optical system information of the imaging optical system when the focus lens is located at the focus position, wherein the size characteristic is used to define the apparent size of the subject in the image, and the processor uses the image to generate three-dimensional information of the subject, and the processor calculates scale information based on the subject distance, the size characteristic and the three-dimensional information, wherein the scale information is used to define the relationship between the apparent size and actual size of the subject in the image, and the processor generates a display pattern representing the actual size of the subject and superimposed on the image based on the scale information.

[0013] Another embodiment of the present invention is an image processing method for processing an image obtained by an endoscope, wherein the imaging optical system of the endoscope has a focus lens for moving the focus of the endoscope, and the image processing method includes the following steps: calculating the subject distance based on the focus position of the focus lens, the focus position being the position of the focus lens when the focus is aligned with the subject in the image; calculating size characteristics based on optical system information of the imaging optical system when the focus lens is located at the focus position, wherein the size characteristics are used to define the apparent size of the subject in the image; calculating scale information based on the subject distance and the size characteristics, wherein the scale information is used to define the relationship between the apparent size and actual size of the subject in the image; and generating a display pattern representing the actual size of the subject and superimposed on the image based on the scale information.

[0014] Another embodiment of the present invention is an image processing method for processing an image obtained by an endoscope, wherein the imaging optical system of the endoscope has a focus lens for moving the focus of the endoscope, and the image processing method includes the following steps: calculating the object distance based on the focus position of the focus lens, the focus position being the position of the focus lens when the focus is aligned with the object in the image; calculating the size characteristics based on optical system information of the imaging optical system when the focus lens is located at the focus position, wherein the size characteristics are used to define the apparent size of the object in the image; using the image to generate three-dimensional information of the object; calculating scale information based on the object distance, the size characteristics and the three-dimensional information, wherein the scale information is used to define the relationship between the apparent size and actual size of the object in the image; and generating a display pattern representing the actual size of the object and superimposed on the image based on the scale information.

[0015] Another embodiment of the present invention is a computer-readable non-temporary recording medium storing an image processing program that enables a computer to execute an image processing method, wherein the image processing method processes an image obtained by an endoscope, the imaging optical system of the endoscope having a focus lens that moves the focus of the endoscope, and the image processing method includes the following steps: calculating the subject distance based on the focus position of the focus lens, the focus position being the position of the focus lens when the focus is aligned with the subject in the image; calculating size characteristics based on optical system information of the imaging optical system when the focus lens is located at the focus position, wherein the size characteristics are used to define the apparent size of the subject in the image; calculating scale information based on the subject distance and the size characteristics, wherein the scale information is used to define the relationship between the apparent size and actual size of the subject in the image; and generating a display pattern representing the actual size of the subject and superimposed on the image based on the scale information.

[0016] Another embodiment of the present invention is a computer-readable non-transitory recording medium storing an image processing program that causes a computer to execute an image processing method, wherein the image processing method processes an image obtained by an endoscope, the endoscope having an imaging optical system having a focus lens that moves the focus of the endoscope, the image processing method comprising the following steps: calculating a subject distance based on a focus position of the focus lens, the focus position being the position of the focus lens when the focus is aligned with the subject in the image; calculating a size characteristic based on optical system information of the imaging optical system when the focus lens is in the focus position, wherein the size characteristic is used to define the apparent size of the subject in the image; generating three-dimensional information of the subject using the image; calculating scale information based on the subject distance, the size characteristic, and the three-dimensional information, wherein the scale information is used to define a relationship between the apparent size and actual size of the subject in the image; and generating a display pattern that represents the actual size of the subject and is superimposed on the image based on the scale information.

[0017] Effects of the Invention

[0018] According to the present invention, it is possible to measure the size of a subject without requiring a special endoscope. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a diagram showing the overall configuration of an image processing device and an endoscope system according to the first embodiment.

[0020] Figure 2 This is a functional block diagram of a processor of an image processing device.

[0021] Figure 3A A diagram showing an example of a scale display superimposed on an image.

[0022] Figure 3B This is a diagram showing another example of a scale display superimposed on an image.

[0023] Figure 3C This is a diagram showing another example of a scale display superimposed on an image.

[0024] Figure 4 This is a flowchart of an image processing method executed by an image processing device.

[0025] Figure 5A This diagram shows an example of a scale display in which the size of a display pattern is variable, and illustrates a case where the subject distance is long.

[0026] Figure 5B This diagram shows an example of a scale display in which the size of a display pattern is variable, and illustrates a case where the subject distance is short.

[0027] Figure 6A This is a diagram showing another example of a scale display in which the size of a display pattern is variable, and shows a case where the subject distance is long.

[0028] Figure 6B This is a diagram showing another example of a scale display in which the size of a display pattern is variable, and shows a case where the subject distance is short.

[0029] Figure 7A This diagram shows an example of a scale display in which the size of a display pattern is fixed and the numerical value is variable, and shows a case where the subject distance is long.

[0030] Figure 7B This diagram shows an example of a scale display in which the size of a display pattern is fixed and the numerical value is variable, and shows a case where the subject distance is short.

[0031] Figure 8A This is a diagram showing another example of a scale display in which the size of a display pattern is fixed and the numerical value is variable, and shows a case where the subject distance is long.

[0032] Figure 8B This is a diagram showing another example of a scale display in which the size of a display pattern is fixed and the numerical value is variable, and illustrates a case where the subject distance is short.

[0033] Figure 9A A diagram illustrating a plurality of ranging areas within an image.

[0034] Figure 9B The diagram shows a distorted image with scale displays superimposed on the center and periphery.

[0035] Figure 10AThis is a diagram showing an example of an image in which a scale display is superimposed on a subject in front.

[0036] Figure 10B This figure shows an example of an image in which a scale display is superimposed on a subject on the back side.

[0037] Figure 11A This is a diagram showing another example of an image in which a scale display is superimposed on a close subject.

[0038] Figure 11B This is a diagram showing another example of an image in which a scale display is superimposed on a subject on the back side.

[0039] Figure 12A This is a diagram showing another example of an image in which a scale display is superimposed on a close subject.

[0040] Figure 12B This is a diagram showing another example of an image in which a scale display is superimposed on a subject on the back side.

[0041] Figure 13A This is a diagram showing another example of an image in which a scale display is superimposed on a close subject.

[0042] Figure 13B This is a diagram showing another example of an image in which a scale display is superimposed on a subject on the back side.

[0043] Figure 14A This is a diagram showing another example of an image in which a scale display is superimposed on a close subject.

[0044] Figure 14B This is a diagram showing another example of an image in which a scale display is superimposed on a subject on the back side.

[0045] Figure 15 This is a functional block diagram of a processor of the image processing device according to the second embodiment.

[0046] Figure 16 This is a flowchart of an example of a method for generating 3D information.

[0047] Figure 17A This figure shows an example of an image in which a subject close to the camera is in focus and a scale display is superimposed at the center.

[0048] Figure 17B This figure shows an example of an image in which a subject in the back is focused and a scale display is superimposed at the center.

[0049] Figure 18A This is a diagram showing an example of an image with a scale display superimposed when a measurement point is set for a nearby subject.

[0050] Figure 18B This figure shows an example of an image with a scale display superimposed when a measurement point is set for a subject on the back side.

[0051] Figure 19 This is a flowchart of the image processing method according to the second embodiment. DETAILED DESCRIPTION

[0052] (First embodiment)

[0053] An image processing apparatus, an image processing method, and a recording medium according to a first embodiment of the present invention will be described with reference to the accompanying drawings.

[0054] like Figure 1 As shown, the image processing device 10 of this embodiment is applied to an endoscope system 100 .

[0055] The endoscope system 100 includes an image processing device 10, an endoscope 20, and a display device 30. The image processing device 10 is connected to the endoscope 20 and the display device 30.

[0056] The endoscope 20 includes an imaging optical system 21 and an imager 22. The imaging optical system 21 includes an objective lens 23 at the front end of the endoscope 20 and an autofocus (AF) lens 24 located between the objective lens 23 and the imager 22.

[0057] The imager 22 includes an image sensor such as a CMOS image sensor, and captures the subject S via the imaging optical system 21 to generate an image. The image output from the imager 22 is input to the display device 30 via the image processing device 10 and displayed on the display device 30. The display device 30 is any type of display device such as a liquid crystal display.

[0058] The AF lens 24 is movable along the optical axis A of the objective lens 23. The movement of the AF lens 24 causes the focal point F of the objective lens 23 to move along the optical axis A. The endoscope 20 can perform AF operations by moving the AF lens 24 along the optical axis A via an actuator (not shown) to achieve focus on the subject S. The endoscope 20 can always automatically perform AF operations or can perform AF operations based on external control signals. Furthermore, the endoscope 20 can also perform AF operations to achieve focus on a position within an image specified by the user.

[0059] The image processing device 10 includes a processor 1 such as a central processing unit, a storage unit 2 , a memory 3 , and a user interface 4 .

[0060] The storage unit 2 is a computer-readable non-transitory recording medium, such as a known magnetic disk, optical disk, or flash memory, etc. The storage unit 2 stores an image processing program 2 a for causing the processor 1 to execute an image processing method described below.

[0061] The memory 3 is composed of a volatile storage device such as a RAM (random access memory), and is used as a work area for the processor 1 .

[0062] The user interface 4 includes input devices such as a mouse, a keyboard, a touch panel, and a microphone. By operating the user interface 4 , the user can input various information and instructions to the image processing apparatus 10 .

[0063] Figure 2 1 is a functional block diagram of the processor 1. The processor 1 includes an image processing unit 11, an evaluation value calculation unit 12, a lens position reading unit 13, a subject distance estimation unit 14, a size characteristic calculation unit 15, a scale information calculation unit 16, and a scale display generation unit 17.

[0064] The image processing unit 11 obtains an image input from the endoscope 20 to the image processing device 10 , performs image processing such as missing pixel correction, white balance adjustment, noise reduction, and demosaicing on the image, and then outputs the image to the display device 30 .

[0065] The evaluation value calculation unit 12, lens position readout unit 13, subject distance estimation unit 14, size characteristic calculation unit 15, scale information calculation unit 16, and scale display generation unit 17 execute the processing described below during the execution of a measurement mode for measuring the size of the subject S. The processor 1 executes the measurement mode in response to, for example, a start instruction. The start instruction is input to the image processing device 10, for example, by a user operating the user interface 4 at a desired timing.

[0066] The evaluation value calculation unit 12 sequentially obtains images during AF operation input from the endoscope 20 to the image processing device 10 and uses the images to sequentially calculate AF evaluation values ​​indicating the degree of focus of the images. For example, the AF evaluation value is the contrast of the image, with the contrast reaching a peak when the subject S is in focus.

[0067] The evaluation value calculation unit 12 may calculate the AF evaluation value using phase difference information instead of the image. In this case, the imager 22 has pixels for detecting image plane phase differences, and the phase difference information is input from the endoscope 20 to the image processing device 10 .

[0068] The lens position reading unit 13 obtains drive information of the AF lens 24 from the endoscope 20 . The drive information includes the position of the AF lens 24 during AF operation, and reads the position of the AF lens 24 from the drive information.

[0069] The subject distance estimation unit 14 determines the focus position of the AF lens 24 based on the AF evaluation value and estimates the subject distance based on the focus position. The focus position is the position of the AF lens 24 when the focus F is aligned with the subject S. The subject distance is the actual distance from the objective lens 23 to the subject S along the optical axis A.

[0070] Specifically, the subject distance estimation unit 14 determines the AF evaluation value at the time of focus based on the AF evaluation value during AF operation. For example, if the AF evaluation value is image contrast, the subject distance estimation unit 14 determines the peak contrast. Next, the subject distance estimation unit 14 determines the position of the AF lens 24 when the following image is acquired as the focus position. This image is used to calculate the AF evaluation value at the time of focus. Next, the subject distance estimation unit 14 estimates the subject distance based on the focus position using a Look Up Table (LUT). The LUT represents the relationship between the focus position and the subject distance and is pre-stored in the storage unit 2.

[0071] The size characteristic calculation unit 15 acquires the optical system information of the imaging optical system 21 from the endoscope 20 and calculates the size characteristics of the subject S in the image based on the optical system information.

[0072] Optical system information is information related to the optical characteristics of the imaging optical system 21, including observation magnification, distortion, and the optical axis center position. Because optical system information changes depending on the position of the AF lens 24, the dimensional characteristics calculation unit 15 acquires optical system information for each position of the AF lens 24. Alternatively, the dimensional characteristics calculation unit 15 may automatically acquire optical system information before executing measurement mode, for example, when the endoscope 20 is connected to the image processing device 10.

[0073] Next, the size characteristic calculation unit 15 calculates the size characteristic based on the optical system information in the focused state where the AF lens 24 is arranged at the focused position. For example, in the case of a wide-angle objective lens 23, the apparent size of the subject S in the image varies depending on the position due to factors such as peripheral distortion and displacement of the optical axis center position generated in the image. For example, the apparent size decreases from the center toward the periphery (see Figure 9B The size characteristic is used to define the apparent size of the subject S at each position in the image.

[0074] The scale information calculation unit 16 calculates scale information based on the subject distance and size characteristics. Scale information defines the relationship between the apparent size and actual size of the subject S within the image. For example, it represents the relationship between the apparent distance from the optical axis center and the actual distance from the optical center in the image. The scale information calculation unit 16 calculates the actual size of the focused subject S based on the subject distance and calculates the scale information based on the actual size and size characteristics of the subject S.

[0075] The scale display generating unit 17 generates the scale display 5 as a virtual scale superimposed on the image based on the scale information. Figure 3A and Figure 3B An example of a scale display 5 is shown. The scale display 5 includes a display pattern 6 composed of a graphic representing the actual size of the subject S and a numerical value 7 of the actual size attached to the display pattern 6. The processor 1 outputs the image B together with the scale display 5 to the display device 30, causing the display device 30 to display the image B with the scale display 5 superimposed thereon.

[0076] Figure 3A The display pattern 6 is a concentric circle composed of a plurality of circles. The plurality of circles have different radii and are centered at the center of the image B. The radius of each circle represents the actual size of the subject S, and a numerical value 7 representing the actual size (5 mm, 10 mm, 15 mm) is assigned to each circle.

[0077] Figure 3B The display pattern 6 is a linear scale with an origin at the center of the image B. The distance from the origin represents the actual size of the subject S, and the numerical value 7 of the actual size (5mm, 10mm, 15mm) is added to each scale line. Figure 3B In , the configuration of the tick marks is symmetrical about the origin.

[0078] like Figure 3C As shown in FIG, the scale lines may be arranged asymmetrically about the origin.

[0079] Next, an image processing method executed by the image processing apparatus 10 will be described.

[0080] like Figure 4 As shown, the image processing method of this embodiment includes: step S3, obtaining the information required for the AF action of the endoscope 20; step S4, estimating the distance to the subject; step S5, calculating the size characteristics of the subject S in the image; step S6, calculating the scale information; and step S7, generating the scale display 5.

[0081] For example, in response to a start instruction, the processor 1 starts a measurement mode (step S1 ), and executes steps S3 to S7 .

[0082] Before step S3, the processor 1 may set the display mode (step S2) based on the user's operation of the user interface 4. The display mode setting includes the form of the scale display 5. For example, the user can select the form from concentric circles and linear scales.

[0083] Next, the processor 1 obtains information necessary to generate the scale display 5 from the endoscope 20 during the AF operation of the endoscope 20, specifically, images, drive information, and optical system information (step S3). In step S3, the processor 1 may also cause the endoscope 20 to perform the AF operation by sending a control signal to the endoscope 20. Alternatively, the processor 1 may obtain the above information while the endoscope 20 is automatically performing the AF operation.

[0084] Next, the processor 1 estimates the subject distance based on the image and the drive information (step S4). Specifically, the evaluation value calculation unit 12 calculates the AF evaluation value from the image, while the lens position readout unit 13 reads the position of the AF lens 24 from the drive information. Next, the subject distance estimation unit 14 determines the focus position of the AF lens 24 based on the AF evaluation value and estimates the subject distance based on the focus position.

[0085] Next, the size characteristic calculation unit 15 calculates the size characteristic of the subject S in the image based on the optical system information of the imaging optical system 21 in the focused state (step S5 ).

[0086] Next, the scale information calculation unit 16 calculates scale information based on the size characteristics and the subject distance (step S6 ).

[0087] Next, the scale display generator 17 generates the scale display 5 according to the set display mode based on the scale information (step S7 ).

[0088] The ruler display 5 is output from the image processing device 10 to the display device 30 together with the image B, and the image B superimposed with the ruler display 5 is displayed on the display device 30. A user such as a doctor can measure the actual size of the subject S such as an affected part based on the ruler display 5 on the image.

[0089] After step S7 , the processor 1 may also determine whether to switch the display mode (step S8 ) and whether to end the measurement mode (step S9 ).

[0090] For example, if the user wishes to change the display mode, they operate the user interface 4 to select the desired display mode. Based on the operation of the user interface 4, the processor 1 switches the display mode ("Yes" in step S8, "No" in step S9, and then step S2), and repeats steps S3 to S7. If the display mode is not to be switched ("No" in step S8), the processor 1 returns to step S3.

[0091] When the user wishes to end the measurement mode, he or she operates the user interface 4 to input an end instruction to the image processing device 10. In response to the end instruction, the processor 1 ends the measurement mode (YES in step S9).

[0092] To calculate scale information, information on the actual size of the subject S within the image, that is, the subject distance, is required. According to this embodiment, the subject distance is calculated based on the focal position of the AF lens 24. Therefore, the image processing device 10 can be combined with a common narrow-diameter endoscope 20 having a focus lens such as the AF lens 24, and can measure the actual size of the subject S without requiring a specialized endoscope, such as one equipped with an illumination optical system for auxiliary measurement light.

[0093] In this embodiment, the scale display 5 may also be changed according to the subject distance.

[0094] Specifically, the processor 1 may also change the size of the display pattern 6 according to the distance to the subject (see Figures 5A to 6B Alternatively, a fixed-size display pattern 6 may be generated, and the value 7 may be changed according to the subject distance (see Figures 7A to 8B ).

[0095] In this case, in step S2 , as the setting of the display mode, it can be selected whether the size of the display pattern 6 is variable or fixed.

[0096] exist Figure 5A and Figure 5B In the example, the radius of each concentric circle varies according to the estimated subject distance. Specifically, the shorter the subject distance, the larger the radius, thereby allowing the user to perceive that the distance to the subject S is closer.

[0097] exist Figure 6A and Figure 6B Specifically, the intervals between the scale lines are increased as the subject distance is shorter, thereby allowing the user to perceive that the distance to the subject S is closer.

[0098] exist 7A to 8B In the example, the shorter the subject distance, the smaller the numerical value 7 is, thereby allowing the user to perceive that the distance to the subject S is closer. The radius of each circle or the interval between scale lines is fixed regardless of the subject distance.

[0099] In the present embodiment, one scale display 5 is arranged at the center of the image B. However, one or more scale displays 5 may be arranged at other positions within the image B instead.

[0100] For example, Figure 9A and Figure 9BAs shown, a plurality of distance measurement areas C arranged in the horizontal and vertical directions are set in the image B, and scale displays 51 and 52 are respectively displayed in one or more distance measurement areas C that are in focus. Figure 9B Image B shows a plurality of circles of the same diameter arranged at equal intervals in the horizontal and vertical directions. The peripheral circles are reduced in size and deformed into ellipses due to distortion.

[0101] In such Figure 9B In such a case where the image B has distortion, the scale display generating unit 17 may generate the scale display 51 and the scale display 52 in consideration of the distortion. Figure 9B In the example shown, scale indicators 51 and 52 are single circles or ellipses, with the radius, or major radius, and minor radius, representing the actual size of subject S. Scale indicator 51 in the center, which is free of distortion, is a circle. Scale indicator 52 in the lower left portion, which is distorted by the distortion, becomes an ellipse tilted in the direction of the distortion. Both scale indicators 51 and 52 represent the same actual size.

[0102] In this case, the subject distance estimation unit 14 selects at least one ranging area C that is in focus, and estimates the subject distance of the selected ranging area C. The scale display generation unit 17 generates scale displays 51 and 52 for the selected ranging area C.

[0103] Here, the subject distance estimation unit 14 may select the ranging area C in focus based on the AF evaluation value of each ranging area C. In this case, the evaluation value calculation unit 12 calculates the AF evaluation value of each ranging area C.

[0104] Alternatively, the subject distance estimation unit 14 may select a distance measurement area including the position specified by the user as the focused distance measurement area C. In this case, the processor 1 causes the endoscope 20 to perform AF operation so that the selected distance measurement area C is focused.

[0105] 10A to 13B The image B shows the inside of a lumen extending from the lower left to the upper right, such as the large intestine, and a plurality of subjects Sa and Sb, such as tumors, exist at different positions in the depth direction.

[0106] Figures 10A to 11B The concentric circle scale shows the case where 5 is displayed. Figure 10A and Figure 10B In the display pattern 6, the size is fixed. Figure 11A and Figure 11B In , the size of the display pattern 6 can be changed. Figures 10A to 11B In the display pattern 6, the concentric circles are distorted according to the distortion of the image B, thereby becoming tilted concentric ellipses.

[0107] Figures 12A to 13BIn the case where the linear ruler shows 5, Figure 12A and Figure 12B In the display pattern 6, the size is fixed. Figure 13A and Figure 13B In the example, the size of the display pattern 6 can be changed.

[0108] exist Figure 10A 、 Figure 11A 、 Figure 12A as well as Figure 13A In the image processing, the focus is set on the subject Sa in the foreground automatically or based on the user's designation, and the scale display 5 is superimposed on the subject Sa. Figure 10B 、 Figure 11B 、 Figure 12B as well as Figure 13B In the image, the focus is set on the subject Sb on the back side automatically or based on the user's designation, and the scale display 5 is superimposed on the subject Sb.

[0109] exist Figures 12A to 13B In the figure, the linear scale is parallel to the horizontal direction of the image B. Alternatively, as shown in Figures 14 and Figure 14B As shown, the linear scale is tilted in the direction of distortion. Figure 14A as well as Figure 14B In this case, the actual distance from the origin is equal on both sides, so a linear scale can be generated that is symmetrical about the origin (that is, the interval between the scale lines and the value 7 are the same on both sides of the origin).

[0110] (Second embodiment)

[0111] Next, an image processing apparatus, an image processing method, and a recording medium according to a second embodiment of the present invention will be described.

[0112] This embodiment differs from the first embodiment in that, in addition to the subject distance and size characteristics, the three-dimensional information of the subject is also used to calculate the scale information. In this embodiment, the structures that differ from the first embodiment are described, and the structures that are the same as the first embodiment are marked with the same figure numbers and the description is omitted.

[0113] The image processing device 10 of this embodiment is applied to an endoscope system 100 including an endoscope 20 and a display device 30 , similarly to the first embodiment, and includes a processor 101 , a storage unit 2 , a memory 3 , and a user interface 4 .

[0114] Figure 15 A functional block diagram of the processor 101 in this embodiment is shown.

[0115] As a function, the processor 101 also has a three-dimensional (3D) information generation unit 18 in addition to the image processing unit 11, the evaluation value calculation unit 12, the lens position readout unit 13, the subject distance estimation unit 14, the size characteristic calculation unit 15, the scale information calculation unit 16 and the scale display generation unit 17.

[0116] The 3D information generation unit 18 generates 3D information of the subject S using multiple images that are in focus and have different viewpoints. The 3D information is a three-dimensional model of the subject S and is generated using known 3D reconstruction techniques. The positional relationship between the 3D information and the images used to generate it is known. The 3D information is defined by the relative size of the subject S and does not contain information about the subject's actual size.

[0117] Figure 16 An example of a method in which the 3D information generating unit 18 generates 3D information is shown.

[0118] First, a plurality of images for generating 3D information are acquired from images sequentially input to the image processing device 10 (step S101 ).

[0119] Next, feature points are extracted from each image (step S102). Feature points are preferably robust to changes in image magnification, rotation, and lighting, and can be extracted using methods such as SIFT (Scale-invariant feature transform), ORB (Oriented FAST and Rotated BRIEF), and AKAZE (Accelerated KAZE).

[0120] Next, matching between images is performed based on the feature amount of each feature point (step S103 ).

[0121] Next, the geometric constraints are used to remove the incorrectly matched feature points from the matched feature points (step S104). To remove the incorrectly matched feature points, a well-known method such as the RANSAC (Random Sample Consensus) algorithm is used.

[0122] Next, based on the changes in the matched feature points, the changes in the camera's posture are estimated (step S105). For example, if the camera's intrinsic parameters are known, five feature points are used to estimate the fundamental matrix. If the intrinsic parameters are unknown, eight or more feature points are used to estimate the fundamental matrix, thereby estimating the camera's posture.

[0123] Next, the three-dimensional position of each matched feature point is estimated by triangulation based on the estimated fundamental matrix or the fundamental matrix, thereby generating a three-dimensional point group (step S106).

[0124] Next, the camera posture and the three-dimensional point group are adjusted (bundle adjustment) by a nonlinear least squares method so that the error when the point group is re-projected onto the image is minimized (step S107 ).

[0125] The subject distance estimation unit 14 determines the focus position of the AF lens 24 based on the AF evaluation value, and estimates the subject distance based on the focus position.

[0126] Here, when the subject is distributed in the depth direction of the image, the focus is not limited to the center of the image, and other positions can be focused. Figure 17A As shown, you can focus on the subject Sa in front of you, or as Figure 17B As shown in FIG, the subject Sb on the back can be focused. Point P indicates the focus position where the subjects Sa and Sb are focused.

[0127] The subject distance estimation unit 14 detects the focus position P in the image B based on the AF evaluation values ​​of each of the plurality of distance measurement areas C in the image B, for example, and estimates the subject distance at the focus position P.

[0128] The scale information calculation unit 16 calculates the scale information at the measurement point Q based on the subject distance, size characteristics, and 3D information at the focus position P. The measurement point Q is a predetermined arbitrary position within the image B. For example, Figure 17A and Figure 17B As shown, the measurement point Q is initially set at the center of image B.

[0129] When the measurement point Q is the center of image B, the scale information calculation unit 16 estimates the actual distance Z_Rd between the center of image B and the focus position P based on the subject distance and the 3D information using the following equation (1). Here, Zo is the center position of the image in the 3D information, Zf is the focus position P in the 3D information, and Z_Rf is the subject distance at the focus position P estimated by the subject distance estimation unit 14.

[0130] Z_Rd=|Zf-Zo|×Z_Rf / Zf(1)

[0131] Next, the scale information calculation unit 16 generates scale information based on the distance Z_Rd and the size characteristics.

[0132] The scale display generator 17 generates a scale display 5 that indicates the actual size at the measurement point Q and overlaps the measurement point Q (for example, the center) in the image B based on the scale information.

[0133] The scale information calculation unit 16 may also set a position in the image B specified by the user as the measurement point Q. In this case, Figure 18A and Figure 18BAs shown, the user specifies an arbitrary position in the image B by operating the user interface 4. Figure 18A and Figure 18B , the subject Sa in the front and the subject Sb in the back are designated respectively.

[0134] In this case, the scale information calculation unit 16 uses the position of the measurement point Q in the 3D information instead of the center position Zo in equation (1), thereby estimating the actual distance between the measurement point Q and the focus position P. Next, based on the estimated actual distance and the dimensional characteristics, the scale information calculation unit 16 generates scale information that defines the relationship between the apparent distance to the measurement point Q and the actual distance to the measurement point Q.

[0135] The scale display generating unit 17 generates a scale display 5 indicating the actual size at the measurement point Q based on the scale information. The generated scale display 5 is superimposed on the measurement point Q, for example, the subject Sa or the subject Sb.

[0136] The scale display 5 is not limited to concentric circles, and may be other scale displays 5 described in the first embodiment (see Figure 3A 、 Figure 3B as well as 10A to 14B The size of the display pattern 6 may also vary according to the distance to the subject (see Figures 5A to 6B Alternatively, the size of the display pattern 6 may be fixed and the value 7 may vary according to the distance to the subject (see Figures 7A to 8B ). In addition, the scale display 5 may also be distorted according to the distortion of the image B (see Figures 10A to 11B 、 Figure 14A as well as Figure 14B ).

[0137] Next, an image processing method executed by the image processing apparatus 10 according to this embodiment will be described.

[0138] like Figure 19 As shown, the image processing method of this embodiment includes step S10 of generating 3D information in addition to steps S3 to S7.

[0139] After step S3 , the subject distance estimation unit 14 estimates the subject distance at the focus position P in the image B (step S4 ). In parallel with this, the 3D information generation unit 18 generates 3D information (step S10 ).

[0140] Next, after calculating the size characteristics (step S5), the scale information calculation unit 16 calculates scale information for a predetermined measurement point Q within image B, such as the center, based on the size characteristics, the subject distance, and the 3D information (step S6). The scale display generation unit 17 generates a scale display 5 indicating the actual size of the subject at measurement point Q based on the scale information (step S7). Thus, image B with the scale display 5 superimposed at measurement point Q is displayed on the display device 30.

[0141] Alternatively, the user may place the object to be measured at a measurement point Q, for example, the center, within image B and then start measurement mode.

[0142] If the user wishes to measure the size of a subject at another location, they can specify a desired location within image B by operating user interface 4. In this case, scale information calculation unit 16 calculates scale information at the specified location (step S6), and scale display generation unit 17 generates scale display 5 at the specified location (step S7). Image B, with scale display 5 superimposed at the specified location, is thus displayed on display device 30.

[0143] Thus, according to this embodiment, scale information is calculated based on 3D information in addition to the subject distance and optical characteristics. 3D information contains information about the overall relative size of the subject within the image. Therefore, based on the combination of the subject distance and 3D information, it is possible to calculate information about the actual size of the subject at a measurement point Q located at a position different from the focus position P, such as the actual distance from the focus position P to the measurement point Q. Then, based on this actual size information and the size characteristics, scale information is calculated for the measurement point Q at any position within image B, generating a scale display 5.

[0144] This allows the scale display 5 to be generated at any position in the image B regardless of the focus position P in the image B. That is, when it is desired to measure the size of a subject at a position different from the focus position P, it is not necessary to perform AF operation to focus on the subject.

[0145] In addition, according to this embodiment, it is possible to simultaneously calculate the scale information at multiple positions in the image and generate multiple scale displays 5 at the same time. Therefore, it is also possible to set multiple measurement points Q in the image B. For example, the user can specify multiple positions at the same time, so that multiple scale displays 5 are overlapped at the specified multiple positions. According to this structure, for example, Figure 18A The scale shows 5 and Figure 18B The scale shows 5 and is superimposed on image B.

[0146] While the embodiments of the present invention and their variations have been described in detail with reference to the accompanying drawings, the specific structure of the present invention is not limited to the above-described embodiments and variations, and various design changes can be made without departing from the scope of the present invention. In addition, the components shown in the above-described embodiments and variations can be appropriately combined.

[0147] For example, the scale display 5 is not limited to the above-mentioned concentric circles or linear scales, and may be any form that indicates the actual size of the subject S.

[0148] Description of Reference Numerals

[0149] 1. 101 processor

[0150] 2 Storage unit (recording medium)

[0151] 2a Image processing program

[0152] 5, 51, 52 scale display

[0153] 6 Display pattern

[0154] 7 Values

[0155] 10 Image processing device

[0156] 20 Endoscope

[0157] 21 Camera Optical System

[0158] 24 auto focus lens (focus lens)

[0159] F Focus

[0160] S, Sa, Sb subjects

[0161] Q measurement point

Claims

1. An image processing device for processing an image obtained by an endoscope, characterized in that: The imaging optical system of the endoscope includes a focus lens for moving the focal point of the endoscope. The image processing device comprises a processor. The processor calculates the distance to the subject according to the focus position of the focus lens, wherein the focus position is the position of the focus lens when the focus is aligned with the subject in the image. The processor calculates a size characteristic based on optical system information of the imaging optical system when the focus lens is located at the in-focus position, wherein the size characteristic defines an apparent size of the object in the image. The processor calculates scale information based on the object distance and the size characteristic, wherein the scale information defines a relationship between an apparent size and an actual size of the object in the image, The processor generates a display pattern that represents the actual size of the object and is superimposed on the image based on the scale information.

2. The image processing device according to claim 1, characterized in that The processor changes the size of the display pattern according to the object distance.

3. The image processing device according to claim 1, characterized in that The processor generates the display pattern of a fixed size, and changes a numerical value of the actual size added to the display pattern according to the subject distance.

4. The image processing device according to claim 1, characterized in that: The display pattern is concentric circles composed of a plurality of circles having radii different from each other.

5. The image processing device according to claim 1, characterized in that: The display pattern is a linear scale.

6. An image processing device for processing an image obtained by an endoscope, characterized in that: The imaging optical system of the endoscope includes a focus lens for moving the focal point of the endoscope. The image processing device comprises a processor. The processor calculates the distance to the subject according to the focus position of the focus lens, wherein the focus position is the position of the focus lens when the focus is aligned with the subject in the image. The processor calculates a size characteristic based on optical system information of the imaging optical system when the focus lens is located at the in-focus position, wherein the size characteristic defines an apparent size of the object in the image. The processor generates three-dimensional information of the object using the image. The processor calculates scale information based on the object distance, the size characteristic and the three-dimensional information, wherein the scale information defines a relationship between an apparent size and an actual size of the object in the image, The processor generates a display pattern that represents the actual size of the object and is superimposed on the image based on the scale information.

7. The image processing device according to claim 6, characterized in that: the processor calculating the scale information defining the relationship between the apparent size and the actual size of the object at the measurement point within the image, The processor generates the display pattern that indicates the actual size of the object at the measurement point and is superimposed on the measurement point based on the scale information.

8. The image processing device according to claim 7, characterized in that: The processor sets a position in the image specified by a user as the measurement point.

9. The image processing device according to claim 6, characterized in that: The processor changes the size of the display pattern according to the object distance.

10. The image processing device according to claim 6, characterized in that: The processor generates the display pattern of a fixed size, and changes a numerical value of the actual size added to the display pattern according to the subject distance.

11. The image processing device according to claim 6, characterized in that: The display pattern is concentric circles composed of a plurality of circles having radii different from each other.

12. The image processing device according to claim 6, characterized in that: The display pattern is a linear scale.

13. An image processing method for processing an image obtained by an endoscope, characterized in that: The imaging optical system of the endoscope includes a focus lens for moving the focal point of the endoscope. The image processing method comprises the following steps: Calculating the distance to the subject according to the focus position of the focus lens, wherein the focus position is the position of the focus lens when the focus is aligned with the subject in the image; calculating a size characteristic based on optical system information of the imaging optical system when the focus lens is located at the in-focus position, wherein the size characteristic defines an apparent size of the object in the image; Calculating scale information based on the object distance and the size characteristic, wherein the scale information defines a relationship between an apparent size and an actual size of the object in the image; as well as Based on the scale information, a display pattern is generated that indicates the actual size of the object and is superimposed on the image.

14. An image processing method for processing an image obtained by an endoscope, characterized in that: The imaging optical system of the endoscope includes a focus lens for moving the focal point of the endoscope. The image processing method comprises the following steps: Calculating the distance to the subject according to the focus position of the focus lens, wherein the focus position is the position of the focus lens when the focus is aligned with the subject in the image; calculating a size characteristic based on optical system information of the imaging optical system when the focus lens is located at the in-focus position, wherein the size characteristic defines an apparent size of the object in the image; generating three-dimensional information of the object using the image; Calculating scale information based on the object distance, the size characteristic and the three-dimensional information, wherein the scale information defines a relationship between an apparent size and an actual size of the object in the image; as well as Based on the scale information, a display pattern is generated that indicates the actual size of the object and is superimposed on the image.

15. A computer-readable non-transitory recording medium storing an image processing program for causing a computer to execute an image processing method, wherein the image processing method processes an image obtained by an endoscope, characterized in that: The imaging optical system of the endoscope includes a focus lens for moving the focal point of the endoscope. The image processing method comprises the following steps: Calculating the distance to the subject according to the focus position of the focus lens, wherein the focus position is the position of the focus lens when the focus is aligned with the subject in the image; calculating a size characteristic based on optical system information of the imaging optical system when the focus lens is located at the in-focus position, wherein the size characteristic defines an apparent size of the object in the image; Calculating scale information based on the object distance and the size characteristic, wherein the scale information defines a relationship between an apparent size and an actual size of the object in the image; as well as Based on the scale information, a display pattern is generated that indicates the actual size of the object and is superimposed on the image.

16. A computer-readable non-transitory recording medium storing an image processing program for causing a computer to execute an image processing method, wherein the image processing method processes an image obtained by an endoscope, wherein: The imaging optical system of the endoscope includes a focus lens for moving the focal point of the endoscope. The image processing method comprises the following steps: Calculating the distance to the subject according to the focus position of the focus lens, wherein the focus position is the position of the focus lens when the focus is aligned with the subject in the image; calculating a size characteristic based on optical system information of the imaging optical system when the focus lens is located at the in-focus position, wherein the size characteristic defines an apparent size of the object in the image; generating three-dimensional information of the object using the image; Calculating scale information based on the object distance, the size characteristic and the three-dimensional information, wherein the scale information defines a relationship between an apparent size and an actual size of the object in the image; as well as Based on the scale information, a display pattern is generated that indicates the actual size of the object and is superimposed on the image.

Images