Endoscope system and method of operation thereof

Abstract

To provide an endoscope system and an operation method thereof that are capable of adjusting the display of a virtual scale in a direction in which an object to be observed is easy to measure, and displaying the virtual scale in substantially real time.SOLUTION: A captured image including a specific region formed by auxiliary measurement light is acquired, a marker direction candidate position is detected on the basis of region-of-interest edge information extracted from the captured image and specified distance position information, a candidate distance from the position of the specific region to the marker direction candidate position is calculated, and a length measurement image is created and displayed in which a display marker is superimposed on the captured image using the position of the specific region as a base point, the display marker passing through a part of an extension line passing through the marker direction candidate position, which forms a candidate distance equal to or larger than a specified distance from the position of the specific region.SELECTED DRAWING: Figure 20

Description

[Technical Field]

[0001] The present invention relates to an endoscope system that displays a virtual scale for measuring the size of an object, and an operating method thereof. [Background technology]

[0002] In an endoscopic system having a light source device, an endoscope, and a processor device, it is common to acquire information such as the distance to a subject or the size of the subject. For example, Patent Document 1 describes "an auxiliary light irradiation unit that irradiates the subject with auxiliary measurement light, an imaging unit that acquires, via an imaging optical system and an imaging element, an image of the subject on which a spot is formed by the auxiliary measurement light, a display device that displays the acquired image of the subject, and a processor that causes the display device to display, together with the image of the subject, an index figure that indicates the actual size of a specific area in the subject as the imaging unit acquires the image of the subject, the index figure having a size set according to the position of the spot on the imaging element."

[0003] Furthermore, Patent Document 2 states that "the processor acquires a captured image of a subject including a specific area formed on the subject by measurement assist light, identifies the position of the specific area on the subject in the captured image, sets a reference scale indicating the actual size of the subject based on the position of the specific area, extracts a region of interest included in the subject in the captured image, determines a measurement portion in the region of interest for measuring the size, generates a measurement value marker indicating the measurement value obtained by measuring the measurement portion of the region of interest based on the reference scale, and creates a specific image in which the measurement value marker is superimposed on the captured image." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 051680 [Patent Document 2] International Publication No. 2021 / 131238 Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Documents 1 and 2, by aligning a laser spot with an observation object and comparing a target figure (virtual scale) with the observation object, such as a region of interest, it is possible to estimate the size of the observation object. However, the direction in which the virtual scale is displayed on the observation screen does not necessarily coincide with the direction in which a user, such as a doctor, wants to measure the observation object. For this reason, in order to measure the observation object using the virtual scale on the observation screen, the user must operate the endoscope to align the direction of the virtual scale with the desired observation direction, which can be time-consuming. Furthermore, depending on the position of the observation object within the living body, it may be difficult to operate the endoscope in the desired observation direction. Furthermore, in order to display the virtual scale in near real time during endoscopic examinations, it is desirable to speed up the processing required to display the virtual scale.

[0006] An object of the present invention is to provide an endoscope system and an operating method thereof that can adjust the display of a virtual scale in a direction that makes it easier to measure an object to be observed and display the virtual scale in almost real time. [Means for solving the problem]

[0007] An endoscope system according to the present invention includes an imaging element that images an object, a light source for use in measuring the object, and a processor. The processor acquires an image of the object including a specific area formed by the fill-measurement light, identifies the position of the specific area in the image, sets a reference scale indicating the actual size of the object based on the position of the specific area, generates a display marker to be superimposed on the image based on the reference scale, extracts area-of-interest edge information from the image, detects a marker orientation candidate position to be used in determining the display orientation of the display marker based on the area-of-interest edge information and specified distance position information indicating a distance from the position of the specific area, calculates a candidate distance that is the distance from the position of the specific area to the marker orientation candidate position, determines the marker orientation candidate position that is a candidate distance from the position of the specific area or greater as a marker orientation determination position, creates a length measurement image in which the display marker, which has the position of the specific area as its start point and which passes through a part of an extension line passing through the marker orientation determination position, is superimposed on the image so that the position of the specific area is its base point, and displays the length measurement image.

[0008] The designated distance position information is preferably information indicating positions equidistant from the position of the specific region, and is preferably information indicating concentric circles centered on the position of the specific region.

[0009] It is preferable that the specified distance and position information is corrected for distortion and that the specified distance and position information is associated with size information indicating the actual size.

[0010] The processor preferably determines, from among the designated distance position information, the designated distance position information that is closest to the position of the specific area as representative designated distance position information, and associates size information with the representative designated distance position information.

[0011] The display marker is a line segment whose base point is the position of the specific area and whose end point is the marker direction determination position, and it is preferable that the processor displays the actual size length of the display marker, which is the distance from the position of the specific area to the marker direction determination position, based on the size information.

[0012] When there are a plurality of marker orientation determination positions, the processor preferably displays a plurality of candidate markers as display markers on the length measurement image.

[0013] Preferably, the processor switches between displaying a plurality of candidate markers in the critical dimension image.

[0014] The display marker is preferably made up of multiple line segments with the position of the specific area as its base point, and one of the multiple line segments preferably has the position of the specific area as its starting point and includes an end point on an extension line passing through the marker direction determination position.

[0015] The region of interest edge information is preferably extracted by a structure enhancement process. The region of interest edge information is preferably extracted using a trained model. The trained model is preferably a convolutional neural network.

[0016] The processor preferably generates a notification instruction to irradiate the edge of the region of interest with auxiliary measurement light using the position of the specific region and the region of interest edge information.

[0017] a step of detecting a marker direction candidate position used to determine the display direction of the display marker based on the attention area edge information and specified distance position information indicating a distance from the position of the specific area; a step of calculating a candidate distance which is the distance from the position of the specific area to the marker direction candidate position; a step of determining, as a marker direction determination position, a marker direction candidate position which is a candidate distance from the position of the specific area that is equal to or greater than the specified distance; a step of creating a length measurement image in which a display marker which has a start point at the position of the specific area and passes through a part of an extension line passing through the marker direction determination position is superimposed on the captured image so that the start point is the position of the specific area; and a step of displaying the length measurement image. [Effects of the Invention]

[0018] According to the present invention, the display of the virtual scale can be adjusted to a direction that makes it easier to measure the object being observed, and the virtual scale can be displayed almost in real time. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic diagram of an endoscope system. [Figure 2] FIG. 2 is a plan view showing the tip portion of the endoscope. [Figure 3] FIG. 2 is a block diagram showing the functions of the endoscope device. [Figure 4] FIG. 2 is a block diagram showing the functions of a signal processing unit. [Figure 5] FIG. 4 is a block diagram showing the function of a measurement assist light emitting unit. [Figure 6] FIG. 10 is an image diagram illustrating an example of a captured image including a subject on which spots are formed. [Figure 7] 10A and 10B are explanatory diagrams illustrating the positions of spots formed on a subject by measurement fill light. [Figure 8] FIG. 2 is a block diagram showing functions of a length measurement image generating unit. [Figure 9] FIG. 4 is a block diagram showing the functions of a reference scale setting unit. [Figure 10] FIG. 10 is an image diagram showing an example of a length measurement image on which a reference scale is superimposed and displayed. [Figure 11] FIG. 2 is a block diagram showing functions of a display marker generation unit. [Figure 12] FIG. 10 is an image diagram showing an example of a marker shape setting image. [Figure 13] FIG. 10 is a block diagram showing the functions of a marker display adjustment unit. [Figure 14] FIG. 10 is an image diagram showing an example in which attention area edge information is displayed on an image. [Figure 15] FIG. 10 is an image diagram showing an example in which circular designated distance position information is displayed on an image. [Figure 16] FIG. 10 is an image diagram showing an example of linear designated distance position information displayed on an image. [Figure 17] FIG. 10 is an image diagram showing an example in which interval incremental position information is displayed on an image. [Figure 18] FIG. 10 is an image diagram showing an example in which interval decreasing position information is displayed on an image. [Figure 19] FIG. 10 is an image diagram showing an example in which marker direction candidate positions are displayed on an image. [Figure 20] FIG. 10 is an image diagram showing an example in which candidate distances are displayed on an image. [Figure 21] FIG. 10 is an image diagram showing an example of a display size setting image. [Figure 22] FIG. 10 is an image diagram showing an example of a length measurement image on which a display marker is displayed when the display marker is superimposed and displayed at a set size. [Figure 23] FIG. 10 is an image diagram showing an example of a length measurement image provided with a marker size display field. [Figure 24]FIG. 10 is an image diagram showing an example in which designated distance position information is displayed on an image when the fixed interval distance is large. [Figure 25] FIG. 10 is an image diagram showing an example of a distortion correction chart image. [Figure 26] FIG. 10 is an image diagram showing an example in which distortion-corrected designated distance position information is displayed on an image. [Figure 27] FIG. 10 is an explanatory diagram showing a method for correcting distortion of a reference scale. [Figure 28] 10 is an image diagram showing an example of a length measurement image when a display marker is superimposed and displayed at the maximum candidate distance. FIG. [Figure 29] FIG. 10 is an image diagram showing an example in which marker direction candidate positions are displayed on an image when there are multiple marker direction decision positions. [Figure 30] FIG. 10 is an image diagram showing an example in which marker orientation determination positions are displayed on an image when there are multiple marker orientation determination positions. [Figure 31] FIG. 10 is an image diagram showing an example of a length measurement image in which a plurality of candidate markers are displayed. [Figure 32] 10 is an explanatory diagram showing an example of switching and displaying length measurement images in which different candidate markers are displayed; FIG. [Figure 33] FIG. 10 is an image diagram showing an example of a length measurement image on which a cross-shaped display marker is displayed. [Figure 34] FIG. 10 is an image diagram showing an example of a length measurement image on which a three-line display marker is displayed. [Figure 35] FIG. 10 is an image diagram showing an example of a length measurement image on which a circular display marker is displayed. [Figure 36] FIG. 10 is a block diagram showing the functions of a length measurement image generating unit provided with a notification control unit. [Figure 37] 10 is an image diagram showing an example of a specific area image for notifying by displaying a message. FIG. [Figure 38] FIG. 10 is a flowchart showing the flow of operations of the endoscope system in a length measurement mode. DETAILED DESCRIPTION OF THE INVENTION

[0020] 1, an endoscope system 10 includes an endoscope 12, a light source device 13, a processor device 14, a display 15, and a user interface 16. The endoscope 12 is optically connected to the light source device 13 and electrically connected to the processor device 14. The various connections are not limited to wired connections and may be wireless. Furthermore, the connections may be made via a network.

[0021] The endoscope 12 has an insertion section 12a that is inserted into the body of an observation target, an operation section 12b provided at the base end of the insertion section 12a, and a bending section 12c and a tip section 12d provided at the tip side of the insertion section 12a. The bending section 12c is bent by operating an angle knob 12e of the operation section 12b. The tip section 12d is directed in a desired direction by bending the bending section 12c.

[0022] The distal end portion 12d irradiates the observation object with illumination light and receives light reflected from the observation object to capture an image of the observation object. A forceps channel (not shown) for inserting a treatment tool or the like may be provided from the insertion portion 12a to the distal end portion 12d. The treatment tool is inserted into the forceps channel through the forceps opening 12j.

[0023] The operation unit 12b also has an observation mode switching switch 12f used to switch the observation mode, a still image acquisition instruction switch 12g used to instruct the acquisition of a still image of the object of observation, and a zoom operation unit 12h used to operate a zoom lens (not shown).

[0024] The processor device 14 is connected to a display 15 and a user interface 16. The display 15 outputs and displays medical images (including captured images, specific area images, normal light images, special light images, and length measurement images) or information processed by the processor device 14. The user interface 16 has a keyboard, mouse, microphone, speaker, foot switch, touchpad, tablet, touch pen, etc., and accepts input operations such as function settings.

[0025] The endoscope 12 has a normal observation mode, a special observation mode, and a length measurement mode, which can be switched using the observation mode selector switch 12f. The normal observation mode is a mode in which the observation target is illuminated with normal light, which is a wideband illumination light. The special observation mode is a mode in which the observation target is illuminated with special light, which is a narrowband illumination light different from the normal light. The length measurement mode illuminates the observation target with illumination light or auxiliary measurement light, and displays a display marker as a virtual scale used to measure the size of the observation target, etc., on the captured image obtained by imaging the observation target. A length measurement image with a display marker superimposed thereon, or a captured image without a display marker superimposed thereon (including a normal light image, a special light image, and a specific area image) is displayed on the display 15. Note that multiple displays 15 may be connected to the processor device 14, and the captured image and the length measurement image may be displayed on different displays 15, respectively.

[0026] The normal light is illumination light used to provide brightness to the entire observation target and observe the entire observation target. The special light is illumination light used to emphasize and observe structures of the observation target, such as ductal structures and blood vessels. The auxiliary measurement light is light used to set a reference scale for generating a display marker, determine the position and direction for displaying the display marker, and determine the base point of the display marker. In addition, although the present embodiment describes a virtual scale displayed on a length measurement image, a real scale may be provided in an actual lumen so that the real scale can be confirmed through captured images and length measurement images. In this case, the real scale may be inserted through the forceps channel of the endoscope 12 and protrude from the distal end 12d.

[0027] When the user operates the still image acquisition instruction switch 12g, the screen displayed on the display 15 freezes and an alert sound is emitted to notify the user that a still image will be acquired. Then, still images of the subject image obtained around the time of operation of the still image acquisition instruction switch 12g are stored in the image storage unit 52 in the processor device 14. The image storage unit 52 is a non-volatile memory such as a hard disk or a USB (Universal Serial Bus) memory. If the processor device 14 can be connected to a network, the still images of the captured images may be stored in an image storage server (not shown) connected to the network instead of or in addition to the image storage unit 52.

[0028] Note that an operating device other than the still image capture instruction switch 12g may be used to issue a still image capture instruction. For example, a foot switch may be connected to the processor device 14, and a still image capture instruction may be issued when the user operates the foot switch. Mode switching may also be performed by a foot pedal. Furthermore, a gesture recognition unit (not shown) that recognizes a user's gesture may be connected to the processor device 14, and a still image capture instruction may be issued when the gesture recognition unit recognizes a specific gesture performed by the user. Mode switching may also be performed using the gesture recognition unit.

[0029] Furthermore, a gaze input unit (not shown) provided near the display 15 may be connected to the processor device 14, and when the gaze input unit recognizes that the user's gaze is within a predetermined area of ​​the display 15 for a certain period of time or more, it may issue an instruction to acquire a still image. Furthermore, a voice recognition unit (not shown) may be connected to the processor device 14, and when the voice recognition unit recognizes a specific voice uttered by the user input via a microphone, it may issue an instruction to acquire a still image or switch modes. Furthermore, an operation panel (not shown) such as a touch panel may be connected to the processor device 14, and when the user performs a specific operation on the operation panel, it may issue an instruction to acquire a still image or switch modes.

[0030] As shown in FIG. 2, the tip 12d of the endoscope 12 is approximately circular, and is provided with an objective lens 31 that is located closest to the subject among the optical members that make up the imaging optical system 44b of the endoscope 12, an illumination lens 32 for irradiating the subject with illumination light, a measurement assist light lens 33 for irradiating the subject with measurement assist light, which will be described later, an opening 34 for projecting a treatment tool toward the subject, and an air and water supply nozzle 35 for supplying air and water.

[0031] The optical axis Ax (see FIG. 7) of the imaging optical system 44b (see FIG. 3), into which reflected light from the subject enters, extends in a direction perpendicular to the paper surface. A first vertical direction D1 is perpendicular to the optical axis Ax, and a second horizontal direction D2 is perpendicular to the optical axis Ax and the first direction D1. The objective lens 31 and the measurement auxiliary light lens 33 are arranged along the first direction D1.

[0032] 3, the light source device 13 includes a light source unit 20 and a light source control unit 21. The light source unit 20 emits illumination light for illuminating a subject. The illumination light emitted from the light source unit 20 enters a light guide 22, passes through an illumination lens 32, and is irradiated onto the subject.

[0033] The light source unit 20 is configured, for example, with a semiconductor light source such as a multi-color LED (Light Emitting Diode), a combination of a laser diode and a phosphor, or a xenon lamp or halogen light source. For example, the light source unit 20 has LEDs such as a V-LED (Violet Light Emitting Diode) that emits violet light, a B-LED (Blue Light Emitting Diode) that emits blue light, a G-LED (Green Light Emitting Diode) that emits green light, and an R-LED (Red Light Emitting Diode) that emits red light. The light source unit 20 also includes an optical filter for adjusting the wavelength band of the light emitted by the LEDs, etc.

[0034] The light source control unit 21 controls the light source unit 20 based on instructions from the system control unit 60. The system control unit 60 not only issues instructions regarding light source control to the light source control unit 21, but also controls the auxiliary measurement light source 45a (see FIG. 5) of the auxiliary measurement light emitting unit 45. In the normal observation mode, the system control unit 60 light In the special observation mode, the system control unit 60 controls the light source unit 20 to emit normal light and to turn off the light source 45a for auxiliary measurement light. In the length measurement mode, the system control unit 60 controls the light sources of the light source unit 20 to emit normal light or special light, and the light source 45a for auxiliary measurement light to emit auxiliary measurement light, to turn on or off.

[0035] The illumination optical system 44a has an illumination lens 32, and light from the light guide 22 is irradiated onto the observation object via the illumination lens 32. The imaging optical system 44b has an objective lens 31, a zoom lens (not shown), and an imaging element 46. Reflected light from the observation object passes through the objective lens 31 and the zoom lens and enters the imaging element 46. As a result, a reflected image of the observation object is formed on the imaging element 46.

[0036] The zoom lens has an optical zoom function that zooms in or out on a subject by moving between the telephoto end and the wide-angle end. The optical zoom function can be switched on and off using the zoom operation unit 12h (see Figure 1) provided on the operation unit 12b of the endoscope. When the optical zoom function is on, further operation of the zoom operation unit 12h allows the subject to be enlarged or reduced at a specific magnification.

[0037] The imaging element 46 is a color imaging sensor that captures a reflected image of the subject and outputs an image signal. The imaging element 46 is preferably a CCD (Charge Coupled Device) imaging sensor or a CMOS (Complementary Metal-Oxide Semiconductor) imaging sensor. The imaging element 46 used in the present invention is a color imaging sensor for obtaining a red image, a green image, and a red image of three colors, R (red), G (green), and B (blue). The red image is an image output from red pixels in the imaging element 46 that are provided with red color filters. The green image is an image output from green pixels in the imaging element 46 that are provided with green color filters. The blue image is an image output from blue pixels in the imaging element 46 that are provided with blue color filters. The imaging element 46 is controlled by an imaging control unit 47.

[0038] The image signal output from the imaging element 46 is transmitted to a CDS / AGC circuit 48. The CDS / AGC circuit 48 performs correlated double sampling (CDS) and automatic gain control (AGC) on the analog image signal. The image signal that has passed through the CDS / AGC circuit 48 is converted into a digital image signal by an A / D (Analog / Digital) converter 49. The A / D converted digital image signal is input to the processor device 14 via a communication I / F (Interface) 50 (see FIG. 3).

[0039] The processor device 14 has programs related to various processes or controls stored in a program storage memory (not shown). The system control unit 60, which is configured by a processor, realizes the functions of the receiving unit 51, the display control unit 53, and the signal processing unit 70 by running the programs stored in the program storage memory.

[0040] The receiving unit 51 of the processor device 14 receives the image signal transmitted from the communication I / F 50 and transmits it to the signal processing unit 70. The signal processing unit 70 has a built-in memory for temporarily storing the image signal received from the receiving unit 51, and processes the image signal group which is a collection of the image signals stored in the memory. Note that the control signal related to the light source control unit 21 may be sent directly to the system control unit 60.

[0041] 4, the signal processing unit 70 includes a normal light image generating unit 80, a special light image generating unit 90, and a length measurement image generating unit 100. When set to normal observation mode, it is preferable that the normal light image generating unit 80 of the signal processing unit 70 perform color conversion processing, color enhancement processing, and structure enhancement processing on the captured image to obtain a normal light image. The normal light image is an image obtained based on normal light that is a well-balanced mixture of purple light V, blue light B, green light G, and red light R, and therefore has natural coloring.

[0042] On the other hand, when the special observation mode is set, it is preferable that the special light image generating unit 90 of the signal processing unit 70 performs structure enhancement processing on the captured image to emphasize structures such as blood vessels, and color difference enhancement processing to expand the color difference between normal parts and diseased parts, etc., of the object to be observed, to obtain a special light image.

[0043] When the measurement mode is set, the image sensor 46 captures an image of the subject illuminated with the measurement assist light, and a captured image (specific area image) including the specific area illuminated with the measurement assist light is input as an image signal to the processor device 14. Hereinafter, in the measurement mode, an image including the specific area and a captured image on which a display marker is superimposed will be specially referred to as a specific area image. In the claims, the "captured image" includes the specific area image. In the measurement mode, the specific area image is sent to the measurement image generation unit 100 of the signal processing unit 70, and a measurement image is created by superimposing a display marker on the specific area image. The generation of the measurement image will be described in detail later.

[0044] Furthermore, when the length measurement mode is set, the specific area image may also be sent to the normal light image generation unit 80, so that a normal light image including the specific area is obtained at the same time.The specific area image may also be sent to the special light image generation unit 90, so that a special light image including the specific area is obtained at the same time.

[0045] The display control unit 53 controls the display of the normal light image, special light image, or length measurement image created by the signal processing unit 70 on the display 15. The system control unit 60 controls the image sensor 46 via an imaging control unit 47 provided in the endoscope 12, and also controls the images stored in the image storage unit 52. The imaging control unit 47 controls the CDS / AGC 48 and A / D 49 in accordance with the control of the image sensor 46, and also sends information to the light source control unit 21.

[0046] As shown in FIG. 5, the measurement assist light emitting unit 45 emits measurement assist light obliquely with respect to the optical axis Ax (see FIG. 8) of the imaging optical system 44b. The measurement assist light emitting unit 45 includes a measurement assist light light source 45a, a measurement assist light generating element 45b, a prism 45c, and a measurement assist light lens 33. The measurement assist light light source 45a emits spot-shaped measurement assist light used for measuring the subject. The measurement assist light light source 45a emits light of a color that can be detected by the pixels of the imaging element 46 (specifically, visible light), and includes a light-emitting element such as a laser light source LD (Laser Diode) or an LED, and a condenser lens that condenses the light emitted from the light-emitting element.

[0047] The wavelength of the light emitted from the measurement assist light source 45a is preferably red light of 600 nm or more and 650 nm or less. Alternatively, green light of 495 nm or more and 570 nm or less may be used. The measurement assist light generating element 45b converts the light emitted from the measurement assist light light source 45a into measurement assist light for obtaining measurement information. Specifically, the measurement assist light generating element 45b uses a collimator lens, a diffractive optical element (DOE), or the like.

[0048] The prism 45c is an optical element for changing the traveling direction of the fill-in measurement light converted by the fill-in measurement light generating element 45b. The prism 45c changes the traveling direction of the fill-in measurement light so that it intersects with the field of view of the imaging optical system 44b, which includes the objective lens 31 and a group of lenses. The traveling direction of the fill-in measurement light will be described in detail later. The fill-in measurement light Lm emitted from the prism 45c passes through the fill-in measurement light lens 33 and is irradiated onto the subject.

[0049] Illuminating the object with the measurement assist light forms a spot (a specific area on the captured image) on the object. The receiving unit 51, which is an image acquisition unit, acquires a specific area image obtained by capturing an image of the object illuminated with illumination light and on which a spot is formed by the measurement assist light. The position of the specific area in the specific area image acquired by the receiving unit 51 is identified by the position identifying unit 120 (see FIG. 8 ). The reference scale setting unit 130 sets a reference scale according to the identified position of the specific area, and the display marker generating unit 140 generates a display marker representing the actual size (actual size) on the length measurement image based on the reference scale. The generated display marker is superimposed on the captured image and displayed on the display 15 as a length measurement image.

[0050] As shown in Fig. 6, when the measurement assist light recognized in the specific area image 121 is spot-shaped, the specific area is a circular spot SP. The position specifying unit 120 (see Fig. 8) specifies the position of the spot SP in the specific area image as the specific area. Note that the specific area image 121 shown in Fig. 6 is preferably displayed on the display 15 from the time when the spot SP is irradiated in the length measurement mode until the display marker is displayed. The captured image that is the specific area image 121 shown in Fig. 6 includes an area of ​​interest 122 on the subject. The area of ​​interest will be described later.

[0051] Instead of the measurement assist light lens 33, a measurement assist slit may be provided at the tip 12d of the endoscope 12. It is also preferable to apply an anti-reflection coating (AR (Anti-Reflection) coating) (anti-reflection section) to the measurement assist light lens 33. By providing an anti-reflection coating, it is possible to suppress a decrease in the proportion of measurement assist light irradiated onto the subject, which would be caused by the measurement assist light being reflected instead of passing through the measurement assist light lens 33, and to prevent the position specifying section 120 (see FIG. 8) from having difficulty recognizing the position of the spot SP formed on the subject by the measurement assist light.

[0052] The measurement assist light emitting unit 45 may be any unit capable of emitting measurement assist light toward the angle of view (field of view) of the imaging optical system 44b. For example, the measurement assist light source 45a may be provided in the light source device 13, and the light emitted from the measurement assist light light source 45a may be guided to the measurement assist light generating element 45b by an optical fiber or the like. Alternatively, the measurement assist light source 45a and the measurement assist light generating element 45b may be arranged in a direction perpendicular to the optical axis of the light source 45a. of The auxiliary measurement light may be emitted in a direction that crosses the field of view of the imaging optical system 44b by being disposed at an angle to the optical axis Ax of the imaging optical system 44b.

[0053] When the auxiliary measurement light is emitted in the length measurement mode, the auxiliary measurement light is emitted in a spot shape so that its optical axis Lm intersects with the optical axis Ax of the objective lens and falls within the imaging field angle of the imaging optical system (within the region between two solid lines Li1), as shown in FIG. 7 . Assuming that observation is possible within the observation distance range Rx, the position of the spot SP formed on the subject by the auxiliary measurement light within the imaging range (indicated by arrows Qx, Qy, and Qz) at each point (the point where each arrow Qx, Qy, and Qz intersects with the optical axis Lm) differs between the near end Px, the center Py, and the far end Pz of the range Rx. The position of the tip 12d of the endoscope 12 is defined as position P1. The observation distance is the distance between the tip 12d of the endoscope 12 and the subject. Therefore, the observation distance is the distance between position P1 and the near end Px, the center Py, or the far end Pz, respectively. Specifically, the observation distance is the distance from the starting point of the optical axis Ax of the imaging optical system 44b at the tip 12d of the endoscope 12 to the subject. The axis Dv indicates the observation distance. The imaging field of view of the imaging optical system 44b is represented by the area between two solid lines Li1, and measurement is performed in the central area of ​​this imaging field of view (the area between two dashed lines Li2) where aberration is small.

[0054] As described above, by emitting the fill-measurement light while the object is within the angle of view of the imaging optical system, the sensitivity of the spot position to changes in observation distance is high, allowing the size of the object to be measured with high accuracy. An image including the spot SP is obtained by capturing an image of the object illuminated with the fill-measurement light using the image sensor 46. In the captured image, the position of the spot SP varies depending on the relationship between the optical axis Ax of the imaging optical system 44b and the optical axis Lm of the fill-measurement light, as well as the observation distance. However, the number of pixels representing the same actual size (e.g., 5 mm) increases when the observation distance is short, and decreases when the observation distance is long. Therefore, by storing correspondence information (scale table 131, see FIG. 8 ) that associates the position of the spot SP with measurement information (the number of pixels on the image) corresponding to the actual size of the object, a reference scale can be set based on the position of the spot SP, and the display marker generation unit 140 can generate a display marker.

[0055] 8, the length measurement image generation unit 100 of the signal processing unit 70 includes a position identification unit 120, a reference scale setting unit 130, a display marker generation unit 140, a marker display adjustment unit 150, and a display marker superimposition unit 160. The position identification unit 120 identifies the position of a spot on the subject as the position of a specific region in the specific region image in order to set the reference scale, etc. The reference scale setting unit 130 sets a reference scale indicating the actual size of the subject based on the position of the specific region.

[0056] The display marker generation unit 140 generates a display marker to be superimposed on the specific region image based on the set reference scale and in accordance with the marker shape setting. The marker display adjustment unit 150 determines the display orientation when superimposing the display marker on the specific region image. The marker display adjustment unit 150 also reflects a preset display size in the display marker. The display marker superimposition unit 160 superimposes the display marker on the specific region image in accordance with the adjustment content of the marker display adjustment unit 150, and generates a length-measurement image. The generated length-measurement image is sent to the display control unit 53 and displayed on the display 15.

[0057] When the measurement mode is set, the light source unit 20 and the measurement assist light emitter 45 continuously emit illumination light and measurement assist light. In some cases, the measurement assist light may be flashed or dimmed. Note that the captured image is a three-color RGB image, but other color images (luminance signal Y, color difference signals Cr, Cb) may also be used.

[0058] The position specifying unit 120 specifies the position of the specific area. The position of the specific area is specified based on a specific area image in which the subject is illuminated with illumination light and measurement assist light in the length measurement mode. The specific area image of the subject in which a spot SP is formed by the measurement assist light is acquired via the imaging optical system 44b and the image sensor 46.

[0059] Preferably, the position identifying unit 120 recognizes the specific region from an image of the specific region that contains many components corresponding to the color of the measurement fill light, and identifies its position as position information. The measurement fill light is light that contains many components of a specific color, for example, red. If the specific color is red, it is preferable to identify the position of the specific region, which is the position of the spot SP formed on the subject, from the red image of the specific region image. One method for identifying the position of the specific region is, for example, to binarize the red image of the specific region image and identify the center of gravity of the black part (pixels whose signal value is higher than a position identification threshold) of the binarized image as the position of the specific region.

[0060] The specific color of the measurement assist light may be another color, such as green. In this case, the position of the specific area is identified from the green image of the specific area image. The shape of the spot SP may be circular, polygonal, including a star-shaped polygon, or star-shaped.

[0061] 9, the reference scale setting unit 130 includes a scale table 131. The reference scale setting unit 130 sets a reference scale indicating the actual size of the subject corresponding to the position of the identified specific area by referring to the scale table 131. The scale table 131 is correspondence information that associates the position of the specific area with measurement information corresponding to the actual size of the subject.

[0062] To specifically explain how to create the scale table 131, for example, the relationship between the position of a specific area and the size of a marker (measurement information) can be obtained by capturing an image of a chart on which a pattern of actual size is regularly formed. A spot-like measurement auxiliary light is emitted toward the chart, and an image of a graph paper-like chart with the same grid (5 mm) as the actual size or finer grid (for example, 1 mm) is captured while changing the observation distance to change the position of the spot, and the relationship between the position of the spot (pixel coordinates on the imaging surface of the image sensor 46) and the number of pixels corresponding to the actual size (how many pixels represent 5 mm, which is the actual size) is obtained.

[0063] The reference scale is a numerical value and unit indicating the actual size, and further, as that information, for example, a line segment having the number of pixels corresponding to 20 mm in the actual size. Normally, the reference scale is not displayed on the display 15, but the reference scale may be displayed on the display 15. In this case, for example, by providing the display marker generation unit 140 with a reference scale display function, the reference scale 133 (the actual size is 20 mm in FIG. 10) may be superimposed on the specific region image as a length measurement image 132 as shown in FIG. 10 and displayed on the display 15.

[0064] The display marker generation unit 140 receives information about the reference scale from the reference scale setting unit 130 or reads it from memory, and generates a display marker. The display marker is a virtual scale that is displayed on the length measurement image. As shown in Fig. 11, the display marker generation unit 140 includes a marker shape setting unit 141 in which marker shape settings are stored. The generated display marker is temporarily stored in memory.

[0065] The marker shape setting unit 141 stores marker shape settings selected by the user or automatically set. When allowing the user to select a marker shape, the system control unit 60 may display a marker shape setting image 142 as shown in FIG. 12 on the display 15. In the marker shape setting image 142, an arbitrary marker shape can be selected, for example, by using radio buttons 143. Marker shapes include, but are not limited to, a line segment, a cross, a triple line, and a circle. Note that circles include perfect circles and ellipses. In addition, unless otherwise specified, display markers are shown in solid lines in the drawings, but the display markers may also be displayed in dashed lines. Whether the display markers displayed on the length measurement image are displayed as solid lines or dashed lines may be set, and the color of the display markers may also be set.

[0066] As shown in FIG. 13, the marker display adjustment unit 150 includes a region of interest edge information extraction unit 170, a specified distance position information setting unit 180, a marker direction candidate position detection unit 190, a candidate distance calculation unit 200, a direction determination unit 210, and a display size setting unit 220.

[0067] In the marker display adjustment unit 150, the marker direction candidate position detection unit 190 detects at least one or more marker direction candidate positions using the attention area edge information extracted by the attention area edge information extraction unit 170 and the designated distance position information based on the settings read by the designated distance position information setting unit 180. Next, the candidate distance calculation unit 200 calculates a candidate distance, which is the distance from the position of the specific area to the marker direction candidate position. Next, the direction determination unit 210 determines to display the display marker in the direction of the marker direction candidate position that is the longest candidate distance from the position of the specific area.

[0068] The base point of the display marker is the position of the specific area. The display marker superimposing unit 160 (see FIG. 8) uses the position of the specific area identified by the position identifying unit 120 to superimpose and display the display marker. The display size setting unit 220 sets and stores the display size of the display marker. The display marker superimposing unit 160 reads out the display size and uses it to superimpose and display the display marker.

[0069] The attention area edge information extraction unit 170 of the marker display adjustment unit 150 extracts attention area edge information from the specific area image. The specific area image used is the one transmitted from the receiving unit 51 to the length measurement image generation unit 100. The attention area edge information is information extracted from the specific area image, which corresponds to the edge of the attention area of ​​the subject.

[0070] The region of interest is an area included in the subject that the user should pay attention to. The region of interest is, for example, an area where a lesion such as a polyp, tumor, or inflammation exists, or an area where a lesion is suspected to exist. By displaying a display marker on the length measurement image, the exact actual size of the region of interest can be measured. The accurate actual size of the region of interest is useful for improving the accuracy of diagnosis. It is also useful information for the user to determine what kind of treatment to perform and to select an appropriate treatment tool for the treatment.

[0071] For example, when a user finds a polyp in the stomach or large intestine, the actual size of the polyp shown in the captured image can be used as one of the factors to decide whether to perform polypectomy or apply another treatment method. Also, when performing procedures such as polypectomy or EMR (Endoscopic Mucosal Resection), knowing the actual size of the area of ​​interest during observation can help select a polypectomy snare with an appropriately sized snare loop.

[0072] A specific example of extracting attention region edge information from a specific region image will be described. In the example shown in FIG. 14, the attention region in the specific region image 171 in which the spot SP is formed is a polyp having a three-dimensional shape like overlapping spheres (see FIG. 6). The attention region edge information extraction unit 170 extracts the edge of the attention region included in the specific region image 171 as attention region edge information 173, as shown in FIG. 14. The attention region edge information 173 extracted in this manner is transmitted to the marker direction candidate position detection unit 190.

[0073] The specified distance position information setting unit 180 reads settings related to specified distance position information used to detect marker direction candidate positions, which will be described later, and transmits the settings to the marker direction candidate position detection unit 190. The specified distance position information refers to information about a position in the specific area image that is a specified distance away from the position of the specific area. In this embodiment, overlaying the specified distance position information on the specific area image is referred to as "setting the specified distance position information on the specific area image."

[0074] Specifically, as shown in FIG. 15, this is information on positions spaced at specified intervals from the position of a specific region (indicated by a spot SP) in the specific region image 171. FIG. 15 shows designated distance position information 181, which represents the locus of positions spaced at regular intervals using circles or arcs (dotted lines). The designated distance position information setting unit 180 reads out a setting for whether the designated distance position information 181 is represented by the number of pixels on the image in the specific region image 171, or by actual size accompanied by size information. Note that in FIG. 15, the designated distance position information 181 farthest from the spot SP is indicated by a leader line and a symbol as a representative. Furthermore, as will be described later, it is preferable that size information be associated only with a representative concentric circle of the designated distance position information 181.

[0075] When the designated distance position information is expressed in terms of the number of pixels, the designated distance position information does not include information about the actual size. In this case, the designated distance position information setting unit 180 reads out information about the number of pixels that make up the fixed interval.

[0076] For example, if the equal interval distance is 100 pixels, designated distance position information 181 indicated by the innermost circle from the position of the specific area (indicated by spot SP) in Fig. 15 indicates a locus of a position 100 pixels away from the position of the specific area. Similarly, designated distance position information 181 indicates loci of positions 200 pixels, 300 pixels, and 400 pixels away from the designated distance position information 181 indicated by the innermost circle, in increments of 100 pixels.

[0077] Although details will be described later, the designated distance position information 181 is not limited to the shape of a perfect circle as shown in Fig. 15. For example, as shown in Fig. 16, the designated distance position information 181 may be represented by multiple arrows drawn radially from the position of a specific area (shown as spot SP) with positions spaced apart by 100 pixels indicated by a scale. In this case, it is preferable that the number of designated distance position information 181 is not limited to eight and can be set to any number.

[0078] When the specified distance position information is expressed in actual size, the size information output by the reference scale setting unit 130 is associated with the specified distance position information. The size information refers to information on the reference scale (the correspondence between the number of pixels and the actual size). In this case, the specified distance position information setting unit 180 reads out information indicating how many millimeters the fixed interval distance is. The unit of the fixed interval distance is not limited to millimeters. For example, centimeters, micrometers, or nanometers may be used.

[0079] 15, if the evenly spaced fixed distance is 10 millimeters, designated distance position information 181 indicated by the innermost circle from the position of the specific area (indicated by spot SP) indicates a locus of positions 10 millimeters away from the position of the specific area. Similarly, designated distance position information is the locus of positions 20 millimeters, 30 millimeters, 40 millimeters, and so on, in 10-millimeter increments, from designated distance position information 181 indicated by the innermost circle.

[0080] The designated distance position information may be information on a distance at a constant interval that gradually increases or decreases, other than the equal intervals shown in Fig. 15. For example, the designated distance position information may be information on an interval that gradually increases at a constant interval (increasing interval position information). In this case, the designated distance position information gradually increases, for example, every 100 pixels.

[0081] Specifically, as shown in FIG. 17 , the designated distance position information 181 indicated by the innermost circle from the position of the specific region (indicated by spot SP) indicates a locus of a position 100 pixels away from the position of the specific region, and the designated distance position information 181 indicated by the circle one circle further out indicates a locus of a position 300 pixels away from the spot SP. The designated distance position information 181 indicated by the circle one circle further out indicates a locus of a position 600 pixels away from the spot SP. In other words, the designated distance position information 181 shown in FIG. 17 is designated distance position information 181 that is gradually increased at regular intervals, such as 100 pixels, 200 pixels, and 300 pixels, from the spot SP. By setting the designated distance position information to interval-increasing position information, the designated distance position information can be set over a wide range of the specific region image. This is particularly useful when the region of interest is present over a wide range of the specific region image, such as when performing high-magnification observation.

[0082] Furthermore, when the designated distance position information is information with intervals that gradually decrease at regular intervals (interval decreasing position information), the designated distance position information gradually increases, for example, every 100 pixels.

[0083] Specifically, as shown in FIG. 18 , the designated distance position information 181 indicated by the innermost circle from the position of the specific region (indicated by spot SP) indicates a locus of a position 150 pixels away from the position of the specific region, and the designated distance position information 181 indicated by the circle one circle away indicates a locus of a position 300 pixels away from the spot SP. Similarly, the designated distance position information 181 indicated by the outermost circle indicates a locus of a position 450 pixels and 500 pixels away from the spot SP. In other words, the designated distance position information 181 shown in FIG. 18 is designated distance position information 181 that gradually decreases at regular intervals from the spot SP, i.e., 150 pixels, 300 pixels, 450 pixels, and 500 pixels. By setting the designated distance position information to interval decreasing position information, the designated distance position information can be set within a narrow range of the specific region image. This is particularly useful when the region of interest is present within a narrow range of the specific region image, such as when performing low-magnification observation.

[0084] It is preferable that the setting of whether to express the specified distance position information in pixels or actual size, and the setting of the fixed interval distance, can be done automatically or arbitrarily. When the user sets the specified distance position information, a specified distance position information setting image (see Figures 15 to 18) can be displayed on display 15, allowing the user to set the fixed interval distance when the fixed interval is an equal distance, and the width of the gradual increase or decrease of the fixed interval.

[0085] The marker direction candidate position detection unit 190 uses the attention area edge information and the specified distance position information to detect marker direction candidate positions used to determine the display direction of the display marker. The marker direction candidate position detection unit 190 receives information on the position of the specified area from the position identification unit 120 or reads it from memory. In addition, the marker direction candidate position detection unit 190 receives the specified distance position information from the specified distance position information setting unit 180 or reads it from memory.

[0086] A specific method for detecting marker direction candidate positions will be described. The marker direction candidate position detection unit 190 detects intersections where attention area edge information 173 extracted from a specific area image 171 overlaps with designated distance position information 181 as marker direction candidate positions 191a, 191b, 191c, 191d, 191e, 191f, and 191g, as shown in the specific example of Fig. 19. In Fig. 19, the seven detected marker direction candidate positions 191a, 191b, 191c, 191d, 191e, 191f, and 191g are coordinate information in the specific area image 171.

[0087] 19 shows attention area edge information 173 and designated distance position information 181 in specific area image 171 to facilitate understanding, but these pieces of information are actually coordinate information processed inside processor device 14. Note that a designated distance position information image, such as that shown in FIG. 19, on which attention area edge information 173 and designated distance position information 181 can be confirmed may be generated and displayed on display 15.

[0088] The candidate distance calculation unit 200 calculates a candidate distance, which is the distance from the position of the specific area to a marker direction candidate position. Specifically, as shown in Fig. 20, the candidate distance calculation unit 200 calculates a candidate distance 201, which is the distance from the spot SP to each of the marker direction candidate positions 191a, 191b, 191c, 191d, 191e, 191f, and 191g. In Fig. 20, the candidate distance 201 from the spot SP to the marker direction candidate position 191a is representatively indicated by a leader line and a symbol.

[0089] Candidate distance 201 is preferably calculated from designated distance position information. For example, in Figures 19 and 20, if the designated distance position information indicates positions equidistant by 100 pixels, the candidate distance to marker direction candidate position 191a is 400 pixels, the candidate distance to marker direction candidate positions 191b and 191g is 300 pixels, the candidate distance to marker direction candidate positions 191c and 191f is 200 pixels, and the candidate distance to marker direction candidate positions 191d and 191e is 100 pixels.

[0090] The direction determination unit 210 refers to the multiple candidate distances calculated by the candidate distance calculation unit 200 and a preset designated distance, and determines the marker direction candidate position that is a candidate distance from the position of the specific area (the position of the spot SP) that is equal to or greater than the designated distance as the marker direction determination position. The designated distance is preferably set arbitrarily or automatically. The largest of the calculated candidate distances (maximum candidate distance) may be set as the designated distance.

[0091] In the specific example shown in Fig. 20, a leader line and a symbol are attached to candidate distance 201, which is the maximum candidate distance. When the maximum candidate distance is set to the specified distance, direction determination unit 210 determines marker direction candidate position 191a as the marker direction determination position. The marker direction determination position is coordinate information for determining the end direction of the display marker, which has the position of the specific area as its base point. In other words, the display direction of the display marker is determined by the marker direction determination position determined by direction determination unit 210.

[0092] Note that the specified distance position information may be numbered in order of proximity to the specific area position to form Nth specified distance position information (N is a natural number), and the specified distance may be the distance from the specific area position to the Mth specified distance position information. For example, in the specific example shown in FIG. 20, if the specified distance is the fourth specified distance position information, the marker direction determination position is marker direction candidate position 191a. If the specified distance is the third specified distance position information, the marker direction determination positions are marker direction candidate positions 191a, 191b, and 191g. A method for displaying a display marker when there are multiple marker direction determination positions will be described later.

[0093] The display size setting unit 220 reads out the setting of the display size of the display marker. For example, a specific example of the display size setting is shown in Fig. 21. In the display size setting image 221 shown in Fig. 21, it is possible to select "display at 20 millimeters" or "display at maximum candidate distance" for the display marker.

[0094] It is preferable that the display size of the display marker superimposed on the specific region image can be set arbitrarily. For example, the display size of the display marker can be changed by inputting an arbitrary numerical value into the display size input field 222 of the display size setting image 221. Furthermore, it may be possible to set the unit of the display size (for example, centimeters, micrometers, or nanometers).

[0095] The display marker superimposing unit 160 generates a length-measurement image by superimposing a display marker, which has the position of the specific region as its base point, on the specific region image so that the display marker passes at least on an extension line passing through the coordinates indicated by the marker orientation determination position, at the display size read out by the display size setting unit 220. The generated length-measurement image is transmitted to the display control unit 53 and displayed on the display 15.

[0096] A specific example of superimposing and displaying a display marker in actual size will be described. For example, the display size is set to "display at 20 millimeters." In this case, in the example of the length measurement image 230 shown in Fig. 22, a display marker 232, which is a line segment of actual size 20 millimeters, is displayed with its display direction adjusted so that it faces the direction of the marker direction determination position 231 (marker direction candidate position 191a in Fig. 20) with the position of the spot SP as the base point. The display marker 232 is superimposed and displayed so as to pass through at least a part of an extension line 233 (a part of the coordinates of the points constituting the extension line 233) that starts from the position of the specific area (the position of the spot SP) and passes through the marker direction determination position 231.

[0097] The actual size of the display marker 232 may be displayed near the display marker 232 as shown in Fig. 22, or may be displayed as a marker size display field 234 that does not overlap the subject as shown in Fig. 23. By displaying the actual size near the display marker, the actual size of the region of interest being observed can be confirmed near the spot SP. On the other hand, by displaying the actual size so that it does not overlap the subject, the user can observe a measurement image that eliminates as much information as possible, and can check the actual size only when necessary.

[0098] With the above configuration, the virtual scale can be displayed according to the size of the region of interest without having to operate the endoscope 12 according to the direction of the fixed virtual scale. By automatically determining the display direction, even in situations where observation is difficult due to physical obstructions from structures within the living body, such as folds in the large intestine or large lesions, the display direction of the virtual scale can be automatically adjusted according to the direction of the maximum size of the region of interest observed in the captured image without having to move the endoscope 12 according to the fixed display position of the virtual scale.

[0099] Furthermore, in the method of calculating candidate distances solely from the region-of-interest edge information, it is not possible to determine which portion of the region-of-interest edge information to use for calculating the candidate distances, resulting in a huge amount of calculation required for the candidate distances. In contrast, in the method of this embodiment, which combines edge information and specified distance position information, candidate marker orientation positions are detected as candidates for determining the orientation, and candidate distances are calculated for each candidate marker orientation position. Therefore, rather than calculating a large number of candidate distances solely from edge information, detecting candidate marker orientation positions by further combining specified distance position information can speed up the calculation of candidate distances, thereby enabling the display orientation of the display marker to be determined quickly. This configuration enables virtual scale display in near real time during endoscopic observation.

[0100] The calculation speed can be further increased by increasing the fixed interval distance of the designated distance position information, i.e., by reducing the number of specified distance position information. As shown in the example of Fig. 24, when the fixed interval distance of the designated distance position information 181 is increased compared to the specific example of Fig. 15, the number of marker direction candidate positions 191 is reduced. Therefore, the number of calculated candidate distances is also reduced, and the time until the display direction of the display marker is determined can be reduced.

[0101] On the other hand, if the fixed interval distance of the designated distance position information is reduced, that is, if the number of designated distance position information is increased, the calculation takes time, but the accuracy of adjusting the display direction of the display marker can be improved. In summary, if the number of marker direction candidate positions is small, the calculation speed increases, and if the number of marker direction candidate positions is large, the adjustment accuracy increases.

[0102] The designated distance position information is preferably information indicating positions at equal intervals from the position of the specific area. As shown in Fig. 15, by setting the designated distance position information at positions at equal intervals from the position of the specific area, the display direction of the display marker can be determined quickly and accurately regardless of the shooting magnification of the specific area image.

[0103] The specified distance position information is preferably concentric circles centered on the position of the specific area. That is, as shown in FIG. 15, it is preferably concentric circles that indicate a locus of positions at a fixed distance from the position of the specific area. If the specified distance position information is represented by lines, the number of marker direction candidate positions may be too large or too small depending on the number of lines. For this reason, by representing the specified distance position information as concentric circles, the display direction can be adjusted with stable accuracy.

[0104] It is preferable that the marker direction candidate position detection unit 190 detects the marker direction candidate positions using the designated distance position information in which distortion aberration has been corrected. In this case, the marker direction candidate position detection unit 190 may read out pre-stored designated distance position information in which distortion aberration has been corrected according to a fixed distance interval, or may perform distortion correction on the designated distance position information 181 using a distortion correction chart image 185 stored in the image storage unit 52.

[0105] The distortion correction chart image 185 can be obtained by using the endoscope 12 to capture an image of a chart 186 on which a pattern of actual size is regularly formed, as shown in Fig. 25. The chart 186 in the distortion correction chart image 185 is distorted compared to an actual image due to refraction of reflected light that enters the endoscope 12 when it passes through the objective lens 31. The designated distance position information setting unit 180 may set the designated distance position information 181 using the distortion correction chart image 185, as shown in Fig. 25, or may read out a distortion correction formula obtained from the correspondence between the chart 186 in the distortion correction chart image 185 and the coordinate deviation from a distortion-free virtual chart, and set the designated distance position information in which distortion aberration has been corrected.

[0106] When the distortion aberration of the designated distance position information is corrected and the designated distance position information 181 is set to concentric circles, the designated distance position information 181 set in the specific area image 171 becomes an ellipse as shown in FIG.

[0107] The reference scale setting unit 130 may also calculate a reference scale in which distortion has been corrected using the distortion correction chart image 185. In this case, a spot-shaped measurement assist light is emitted toward the chart 186, and an image of the graph paper-like chart 186 with gridlines the same as the actual size (5 mm) or finer gridlines (e.g., 1 mm) is captured while changing the observation distance and changing the spot position, and the relationship between the position of the spot SP (pixel coordinates on the imaging surface of the image sensor 46) and the number of correction pixels corresponding to the actual size (e.g., how many pixels represent 5 mm, the actual size) is obtained as the correction scale table 134. For example, as shown in FIG. 27 , the maximum value "20" of the reference scale 133 is associated with the maximum value P of the correction pixels, and the midpoint "10" of the reference scale 133 is associated with the midpoint 0.5P of the correction pixels.

[0108] With the above configuration, it is possible to detect a marker direction candidate position taking into consideration the distortion of the lens of the endoscope 12, and it is possible to improve the accuracy of determining the display direction of the display marker.

[0109] The designated distance position information is preferably associated with size information indicating the actual size. In this case, the designated distance position information setting unit 180 reads out the setting "indicate designated distance position information in actual size." Next, the candidate distance calculation unit 200 calculates candidate distance 201 (see FIG. 23) from the position of the specific area to the marker direction candidate position as the actual size.

[0110] In this case, the display marker superimposing unit 160 can display a display marker 232 (18 millimeters in FIG. 28) which is a line segment having the same actual size as the actual size of the region of interest on the length-measurement image 230, as shown in FIG. 28. Note that in FIG. 28, the display marker 232 is displayed at the maximum candidate distance. The maximum candidate distance superimposed as the display marker 232 is the actual size distance from the position of the specific region to the marker orientation determination position 231. With the above configuration, once the marker orientation determination position is determined, the actual size of the region of interest can be directly displayed as a display marker.

[0111] On the other hand, when the specified distance position information 181 is expressed in terms of the number of pixels without being associated with size information, i.e., when the display marker 232 is superimposed at an arbitrary display size such as "20 millimeters" rather than at the maximum candidate distance, the calculation process is faster because size information is not associated with the specified distance position information 181, and the time required to display the measurement image with the superimposed display marker can be shortened.

[0112] In addition, size information may be associated with the designated distance position information that exists at the coordinate position closest to the position of the specific area among the designated distance position information, and interpolation processing may be performed using distances at regular intervals when calculating the candidate distance.If the designated distance position information is concentric circles, the circle that is innermost from the position of the specific area is set as representative designated distance position information, and size information is associated with the representative designated distance position information.When calculating the candidate distance, interpolation processing may be performed using size information associated with the representative concentric circle, setting information for the regular interval distance, and information on how many pixels away the concentric circle is from the representative concentric circle to calculate the candidate distance.With the above configuration, calculation processing can be faster than when size information is associated with all designated distance position information.

[0113] Here, as a first modification, a case where there are multiple marker direction candidate positions that are at a candidate distance from the position of the specific region that is equal to or greater than the specified distance will be described. In this case, it is preferable that the direction determination unit 210 determines multiple marker direction determination positions, and the display marker superimposition unit 160 uses the position of the specific region as a base point and generates a length measurement image in which multiple display markers that include at least each of the marker direction determination positions are superimposed on the specific region image.

[0114] A specific example will be described. In the specific example of Fig. 29, when the specified distance is set as the maximum candidate distance, of six marker direction candidate positions 192a, 192b, 192c, 192d, 192e, and 192f in the specific area image 171, there are two marker direction candidate positions (marker direction candidate positions 192a and 192b) that are the maximum candidate distance from the position of the specific area (the position of the spot SP). In this case, the direction determining unit 210 determines the marker direction candidate position 192a (see Fig. 29) as the first marker direction determined position 231a and the marker direction candidate position 192b (see Fig. 29) as the second marker direction determined position 231b, as shown in Fig. 30.

[0115] Next, the display marker superimposing unit 160 generates a length-measurement image 230 by superimposing multiple display markers as candidate markers on the specific region image, as shown in FIG. 31. The first candidate marker 232a passes through the first marker direction determination position 231a. The second candidate marker 232b passes through the second marker direction determination position 231b. With the above configuration, multiple display markers having the same actual size can be displayed. Furthermore, options can be created for the user to select a display marker displayed in a direction that is easier for the user to observe.

[0116] The multiple candidate markers may be superimposed on the length measurement image 230 in the same display manner, as shown in Fig. 31, or may be superimposed in different display manners. For example, the first candidate marker 232a may be displayed with a solid line, and the second candidate marker 232b may be displayed with a dashed line. Furthermore, the first candidate marker 232a and the second candidate marker 232b may be displayed in different colors. The display manner of the multiple candidate markers is not limited to this.

[0117] Furthermore, it is preferable that the display of the multiple candidate markers be switched. Specifically, as shown in Fig. 32, a first length-measurement image 230a displaying a first candidate marker 232a and a second length-measurement image 230b displaying a second candidate marker 232b are switched between. In Fig. 32, the second candidate marker 232b not displayed in the first length-measurement image 230a and the first candidate marker 232a not displayed in the second length-measurement image 230b are indicated by dashed lines.

[0118] 32, the display mode of the candidate marker that is most desired to be emphasized and the other candidate markers may be made different for each switched length measurement image, such as displaying the first candidate marker 232a with a solid line and the second candidate marker 232b with a dashed line in the first length measurement image 230a, and displaying the first candidate marker 232a with a dashed line and the second candidate marker 232b with a solid line in the second length measurement image 230b. The way in which the display mode of the candidate markers is made different may be by using a solid line or a dashed line, or by using different colors or transparency, and is not limited to this.

[0119] As a switching method, a candidate marker changeover switch (not shown) may be provided on the endoscope 12 or on a foot switch or touch panel serving as the user interface 16, and operation of the candidate marker changeover switch may be used as a trigger to switch between the respective length measurement images 230a, 230b. However, the switching method is not limited to this. For example, voice recognition, gesture recognition, or gaze recognition (eye tracking) may also be used as a trigger for the switching operation.

[0120] Alternatively, one of the multiple candidate markers may be determined as the display marker, and only the determined display marker may be displayed on the length measurement image. In this case, a candidate marker selection button may be provided on the endoscope 12 or the user interface 16. Voice recognition, gesture recognition, or gaze recognition may be used as a trigger for the candidate marker selection operation. With the above configuration, the user can select a display marker displayed in a direction that is easier for the user to observe.

[0121] Here, as a second modified example, an example in which the display marker is displayed in a display mode other than a line segment (arrow) will be described. It is preferable that the display marker superimposing unit 160 displays the display marker 232 on the length-measurement image 230 in the marker shape read out by the display marker generating unit 140. For example, as shown in Fig. 33, when the marker shape of the display marker 232 displayed on the length-measurement image 230 is a cross, the base point is set to the position of the specific area (spot SP), and the end point of any one of the line segments constituting the cross shape, which is the display marker 232, is included on an extension line 233 that has the position of the specific area (spot SP) as its start point and passes through the marker direction determination position 231.

[0122] 34, when the marker shape of the display marker 232 displayed on the length measurement image 230 is a triple line, the base point is set to the position of the specific area (spot SP), and the end point of one of the line segments constituting the triple line is included on an extension line 233 that has the position of the specific area (spot SP) as its start point and passes through the marker direction determination position 231. Note that FIGS. 33 and 34 show an example in which the display size of the display marker 232 is set to "display in 10 millimeters."

[0123] 35, when the marker shape of the display marker 232 displayed on the length measurement image 230 is made circular, the center of the display marker 232 is used as the base point. In this case, the base point is the position of the specific area (spot SP), and the marker direction determination position 231 is included on the circumferential line of the display marker 232 or in an area inside the circumferential line. In the example shown in FIG. 35, the marker direction determination position 231 is included on the circumferential line of the display marker 232. Note that FIG. 35 shows an example in which the display size of the display marker 232 is set to "display at maximum candidate distance." Furthermore, the portion of the display marker 232 that is displayed on the length measurement image 230 is indicated by a solid line, and the portion that is off the screen and not displayed on the length measurement image 230 is indicated by a dashed line.

[0124] By displaying the display marker as a circle, the range of treatment can be visualized, which can assist in the selection of treatment tools. Also, by setting the display marker to be displayed larger than the maximum candidate distance, the range where local injection fluid should be injected when applying EMR can be visually confirmed.

[0125] Here, a method for extracting the region-of-interest edge information will be described. The region-of-interest edge information extraction unit 170 preferably extracts the region-of-interest edge information from the specific region image by performing a structure enhancement process on the specific region image. The structure enhancement process is image processing that performs a contrast adjustment process, an edge enhancement process, and / or a binarization process.

[0126] In contrast adjustment processing, first, a density histogram is calculated from the acquired specific region image, with pixel values ​​(brightness values) on the horizontal axis and frequency on the vertical axis. Next, a distribution function is calculated from the density histogram. Furthermore, tone correction is performed on the specific region image so that low density areas become lower and high density areas become higher. In this case, it is preferable to determine a calculation value for tone correction to be substituted into the distribution function. Furthermore, when performing tone correction, a tone correction table in which density and output values ​​are associated may be used.

[0127] Edge enhancement processing emphasizes edges by applying image processing to the pixels that make up a specific region image using a combination of filters such as a dilation filter, contraction filter, averaging filter, and / or median filter, which replace the density of the central pixel of a 3x3 pixel array. A dilation filter replaces the pixel with the brightest pixel among the pixels in the 3x3 pixel array with the central pixel. A contraction filter replaces the pixel with the least bright pixel among the pixels in the 3x3 pixel array with the central pixel. An averaging filter sets the average value of the pixel values ​​of the pixels in the 3x3 pixel array as the pixel value of the central pixel. A median filter sets the median value of the pixels in the 3x3 pixel array as the pixel value of the central pixel.

[0128] In the binarization process, the specific region image is binarized using a binarization threshold. For example, the pixel value of each pixel in the specific region image is referenced, and pixels equal to or greater than the binarization threshold are converted to black, and pixels less than the binarization threshold are converted to white, resulting in a binarized image. The specific region image that has undergone the contrast adjustment process and / or edge enhancement process may be further subjected to binarization. Alternatively, the binarization process may be performed without performing the contrast adjustment process and / or edge enhancement process.

[0129] The region-of-interest edge information extraction unit 170 extracts, as position information, region-of-interest edge information corresponding to the edges of the region of interest that appears in the specific region image from the processed specific region image that has undergone such structure enhancement processing. By extracting the region-of-interest edge information through structure enhancement processing, it is possible to apply existing image processing methods to obtain the region-of-interest edge information.

[0130] It is also preferable that the attention area edge information extraction unit 170 is a trained model, and extracts attention area edge information by inputting a specific area image. The model used to generate the trained model can be any of various models suitable for image recognition by machine learning.

[0131] It is preferable to use deep learning as the model to be trained. Furthermore, it is preferable to use a convolutional neural network for the purpose of extracting edge information of an area of ​​interest from an image. In addition to deep learning, machine learning includes decision trees, support vector machines, random forests, regression analysis, supervised learning, semi-unsupervised learning, unsupervised learning, reinforcement learning, deep reinforcement learning, generative adversarial networks, etc.

[0132] It is preferable to use captured images acquired in the past for model training. For example, attention area edge information generated from captured images acquired in the past is associated with the original image from which the attention area edge information was extracted, and used as training data or test data. Depending on the type of model, captured images without attention area edge information may be used as training images for training. Extracting attention area edge information using a trained model can improve the accuracy of extracting attention area edge information.

[0133] Here, as a third modified example, an example of issuing a notification to form a spot at the edge of the region of interest will be described. In this case, as shown in Fig. 36, the length-measurement image generating unit 100 is further provided with a notification control unit 240. It is preferable that the notification control unit 240 uses the position of the specific region and the region of interest edge information to generate a notification instruction to irradiate the edge of the region of interest with a spot, and control so as to issue a notification via the user interface 16.

[0134] When a subject includes a three-dimensional region of interest, such as a polyp with a raised portion of the mucosa, and a spot is formed at the apex of the raised region of interest (the apex that rises toward the tip 12d), the size of the display marker displayed on the critical dimension image will be smaller than when a spot is formed at the origin of the raised portion, which may result in the actual size of the entire region of interest that is the target of diagnosis or treatment being misjudged as being smaller. This problem occurs because the size of the generated virtual scale differs depending on the position where the spot is formed (see Figure 7). Therefore, by forming a spot on the mucosa near the origin of the raised portion of the region of interest or at the origin, a display marker with the accurate actual size of the entire region of interest can be displayed on the critical dimension image.

[0135] To superimpose a display marker showing the exact actual size of the entire region of interest, a measurement aid is required. lightIt is preferable that the spot formed on the subject by the endoscope 12 is formed at the edge of the region of interest, i.e., at the position indicated by the region of interest edge information on the specific region image. Therefore, by issuing a notification when "the spot is not formed at the edge of the region of interest," the user can be prompted to operate the endoscope 12 to form the spot at the edge of the region of interest.

[0136] The notification control unit 240 first receives or reads from memory the position information of the specific area identified by the position identification unit 120 and the attention area edge information. Next, the notification control unit 240 refers to the attention area edge information and the position information of the specific area, and if the coordinate information of both does not overlap, it sends a notification instruction to the system control unit 60.

[0137] It is also possible to calculate the distance between the coordinates included in the attention area edge information and the coordinates of the position of the specific area, and send a notification instruction if the distance is equal to or greater than a notification threshold. Notification instructions include notification instructions using sounds such as a buzzer sound or a voice message, notification instructions using a notification lamp provided as the user interface 16 and a notification by changing the color or blinking of the notification lamp, or notification instructions using visual information such as a message displayed on the specific area image or the measurement image. It is noted that notification instructions are not limited to these.

[0138] A specific example of notification by message display on a specific area image will be described using the specific example shown in Fig. 37. The length measurement image 230 includes a spot SP formed on the subject. If the notification control unit 240 determines that the spot SP is "not formed at the edge of the area of ​​interest," a message 261 is displayed in a portion of the length measurement image 230 that does not include the subject. In the specific example shown in Fig. 37, the message 261 displays "Please irradiate the edge with measurement assist light."

[0139] In this case, instructions regarding the operation of the endoscope 12 may be displayed, such as a message 261 saying "Move the endoscope A centimeters to the right." Also, candidate positions for where the spot SP should be formed may be displayed as candidate spot positions in a portion of the specific area image that includes the subject.

[0140] A series of operations of the endoscope system in the length measurement mode of this embodiment will be described with reference to the flowchart shown in Fig. 38. First, measurement assist light is emitted (step ST101), and an image of the subject including a spot formed on the subject by the measurement assist light is captured to obtain a captured image (specific area image) (step ST102). Next, the position specifying unit 120 specifies the position of the spot formed on the subject as the position of the specific area in the specific area image (step ST103). Next, the reference scale setting unit 130 sets a reference scale based on the position of the specific area (step ST104).

[0141] Next, the display marker generation unit 140 generates a display marker to be superimposed on the specific region image in accordance with the marker shape setting based on the set reference scale (step ST105). Next, the attention region edge information extraction unit 170 of the marker display adjustment unit 150 extracts attention region edge information (step ST106). This step may be performed simultaneously with the generation of the reference scale (step ST104).

[0142] Next, the marker direction candidate position detection unit 190 detects at least one or more marker direction candidate positions using the attention area edge information and the specified distance position information (step ST107). Next, the candidate distance calculation unit 200 calculates a candidate distance from the position of the specific area to the marker direction candidate position (step ST108). Next, the direction determination unit 210 determines a marker direction determination position and determines the display direction of the display marker (step ST109). Next, the display marker superimposition unit 160 superimposes the display marker whose display direction has been determined on the specific area image to generate a length measurement image (step ST110). Finally, the display control unit 53 displays the length measurement image on the display 15 (step ST111).

[0143] The critical dimension image generating unit 100 may be separated from the processor device 14 and provided in an extended processor device (not shown). In this case, the extended processor device may be connected to the processor device 14, the user interface 16, and the display 15.

[0144] In the above embodiment, the hardware structure of processing units that perform various processes, such as the receiving unit 51, the signal processing unit 70, the display control unit 53, and the system control unit 60, is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for performing various processes.

[0145] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which one processor is configured with a combination of one or more CPUs and software, as typified by client or server computers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a System on Chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0146] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit formed by combining circuit elements such as semiconductor elements, and the hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). [Explanation of symbols]

[0147] 10 Endoscopy System 12 Endoscopy 12a Insertion part 12b Operation section 12c curved section 12d Tip 12e Angle Knob 12f Observation mode switch 12g Still image acquisition instruction switch 12h Zoom operation section 12j forceps mouth 13 Light source device 14 Processor unit 15 Display 16 User Interface 20 Light source section 21 Light source control unit 22 Light Guide 31 Objective Lens 32 Lighting lens 33 Measurement auxiliary light lens 34 Aperture 35 Air and water supply nozzle 44a Illumination optical system 44b Imaging optical system 45 Measurement auxiliary light emitting section 45a Light source for auxiliary measurement light 45b Measurement auxiliary light generating element 45c Prism 46 image sensor 47 Imaging control unit 48 CDS / AGC circuit 49 A / D converter 50 Communication I / F 51 Receiving unit 52 Image storage section 53 Display control unit 60 System control section 70 Signal Processing Section 80 Normal light image generation unit 90 Special light image generation unit 100 Measurement image generation unit 120 Location identification part 121, 171 Specific area images 122 Areas of Interest 130 Reference scale setting section 131 Scale Table 132, 230 Measurement image 133 Reference Scale 134 Correction scale table 140 Display marker generation unit 141 Marker shape setting section 142 Marker shape setting image 143 Radio Buttons 150 Marker display adjustment unit 160 Display marker superimposition section 170 Region of interest edge information extraction unit 173 Area of ​​interest edge information 180 Specified distance position information setting section 181 Specified distance position information 185 Distortion correction chart image 186 charts 190 Marker direction candidate position detection unit 191, 191a, 191b, 191c, 191d, 191e, 191f, 191g, 192a, 192b, 192c, 192d, 192e, 192f Marker direction candidate positions 200 Candidate distance calculation unit 201 candidate distances 210 Direction determination unit 220 Display size setting section 221 Image for setting display size 222 Display size input field 231 Marker direction determination position 231a First marker direction determination position 231b Second marker direction determination position 232 Display Marker 232a First candidate marker 232b Second candidate marker 233 Extension line 234 Marker size display field 240 Notification control unit 261 Messages

Claims

1. an imaging element for capturing an image of a subject; a light source for auxiliary measurement light that emits auxiliary measurement light used for measuring the subject; a processor; a program storage memory for storing a program, The processor, by running the program, acquiring a captured image of the subject including the specific area formed by the measurement fill light; Identifying the position of the specific region in the captured image; setting a reference scale indicating the actual size of the subject based on the position of the specific area; generating a display marker to be superimposed on the captured image based on the reference scale; extracting area-of-interest edge information from the captured image; detects a marker direction candidate position used to determine a display direction of the display marker based on the attention area edge information and designated distance position information indicating a distance from the position of the specific area; calculating a candidate distance that is a distance from the position of the specific region to the candidate position of the marker direction; determining the marker direction candidate position that is at a candidate distance equal to or greater than a specified distance from the position of the specific region as a marker direction determined position; creating a length measurement image in which the display marker, which has a starting point at the position of the specific area and passes through a part of an extension line passing through the marker direction determination position, is superimposed on the captured image so that the position of the specific area is used as a base point; an endoscope system that displays the length-measuring image;

2. The endoscope system according to claim 1 , wherein the specified distance position information is information indicating positions at equal intervals from the position of the specific region.

3. The endoscope system according to claim 1 or 2, wherein the specified distance position information is a concentric circle centered on the position of the specific region.

4. The endoscope system according to claim 1 , wherein the specified distance position information is corrected for distortion.

5. The endoscope system according to claim 1 , wherein the specified distance position information is associated with size information indicating an actual size.

6. The processor: The endoscope system according to claim 5 , wherein the designated distance position information, among the designated distance position information, the designated distance position information that is closest to the position of the specific region is set as representative designated distance position information, and the size information is associated with the representative designated distance position information.

7. the display marker is a line segment having a base point at the position of the specific area and an end point at the marker direction determination position, The processor: The endoscope system according to claim 5 or 6, wherein the length of the actual size of the display marker, which is the distance from the position of the specific region to the marker direction determination position, is displayed based on the size information.

8. The processor:

8. The endoscope system according to claim 1, wherein when there are a plurality of the marker direction determination positions, a plurality of candidate markers are displayed as the display markers on the length measurement image.

9. The processor: The endoscope system according to claim 8 , wherein the plurality of candidate markers are displayed in a switched manner in the length measurement image.

10. the display marker is made up of a plurality of line segments starting from the position of the specific area; The endoscope system according to claim 1 , wherein one of the plurality of line segments has a start point at the position of the specific region and includes an end point on the extension line passing through the marker orientation determination position.

11. The endoscope system according to claim 1 , wherein the region-of-interest edge information is extracted by a structure enhancement process.

12. The endoscope system according to claim 1 , wherein the region-of-interest edge information is extracted using a trained model.

13. The endoscope system according to claim 12 , wherein the trained model is a convolutional neural network.

14. The processor: The endoscope system according to claim 1 , wherein a notification instruction to irradiate the edge of the region of interest with the auxiliary measurement light is generated using the position of the specific region and the region of interest edge information.

15. capturing an image of a subject; a step of emitting a measurement fill light used for measuring the object; acquiring a captured image of the subject including the specific area formed by the measurement fill light; identifying the position of the specific region in the captured image; setting a reference scale indicating the actual size of the subject based on the position of the specific area; generating a display marker to be superimposed on the captured image based on the reference scale; extracting area-of-interest edge information from the captured image; detecting a marker direction candidate position used to determine a display direction of the display marker based on the attention area edge information and designated distance position information indicating a distance from the position of the specific area; calculating a candidate distance from the position of the specific region to the marker direction candidate position; determining the marker direction candidate position, which is at a candidate distance equal to or greater than a designated distance from the position of the specific region, as a marker direction determined position; creating a length measurement image in which the display marker, which has a starting point at the position of the specific area and passes through a part of an extension line passing through the marker direction determination position, is superimposed on the captured image so that the position of the specific area is used as a base point; and displaying the length measurement image.

Images