Endoscope system

Abstract

An endoscope system comprises: an endoscope for photographing living tissue in a body cavity; and an image processing unit for performing image processing on the camera image photographed by the endoscope. The endoscope is equipped with an objective lens, which is arranged in front of the light-receiving surface of the camera element, and is configured to simultaneously form images of living tissue obtained through multiple windows as the camera image on the light-receiving surface. The image processing unit comprises a three-dimensional expansion processing unit, which calculates different directions of the feature part seen from the multiple windows based on the position information in the image of the feature part that is commonly contained in the multiple images obtained through the multiple windows in the camera image photographed by the endoscope and is recognizable relative to other parts, and expands the two-dimensional information of the image of the feature part into three-dimensional information.

Description

Technical Field

[0001] The present invention relates to an endoscope system for photographing living tissue in a body cavity. Background Art

[0002] An endoscope is a device equipped with an imaging element that is inserted into a body cavity, such as the human body, to image living tissue on the inner surface of the body cavity. For example, an endoscope is inserted into the large intestine, and the captured image is displayed on a monitor to determine whether there are unhealthy areas, such as lesions, in the living tissue. When diagnosing living tissue in the large intestine, it is necessary to pull the endoscope in one direction while tilting the folds in one direction, without affecting the protruding folds on the inner surface, to determine whether there are lesions at the bottom of the folds. However, even if the folds are tilted in one direction, lesions may still exist in areas hidden behind the folds. If the field of view of the image to be captured is narrow, the areas between adjacent folds may not be captured.

[0003] In order to be able to record images between the folds before they collapse, a wide-angle objective can be used in the objective system of the image-receiving element.

[0004] For example, a known endoscope system includes an endoscope having an insertion portion that is inserted into an observation object, a direct observation portion having a field of view in the tip direction of the insertion portion, a side viewing portion having a field of view in the lateral direction of the insertion portion, a protrusion that protrudes from the insertion portion and forms a blind spot in the field of view of the side viewing portion, and an image acquisition portion that uses the direct observation portion to acquire a direct observation image and uses the side viewing portion to acquire a side viewing image (refer to Patent Gazette No. JP2018-57799). Summary of the Invention

[0005] Problems to be solved by the invention

[0006] The above-mentioned endoscope system can simultaneously acquire direct observation images and side observation images, allowing a wide range of the observation object to be displayed on the display. However, the direct observation images and side observation images are two-dimensional. Therefore, when the camera image is displayed, it is difficult for the endoscope operator to obtain information about the surface roughness of the lesion from the displayed camera image. In addition to the size of the lesion, obtaining information about the surface roughness is important for diagnosing the progression of the lesion.

[0007] Therefore, an object of the present invention is to provide an endoscope system that can obtain three-dimensional information of characteristic parts such as lesions by using the captured images when imaging living tissue.

[0008] Technical solutions to problems

[0009] One aspect of the present invention is an endoscope system including an endoscope for capturing images of living tissue in a body cavity, and an image processing unit for performing image processing on images captured by the endoscope.

[0010] The endoscope comprises:

[0011] an imaging element configured to capture an image of living tissue; and

[0012] The objective lens is provided in front of the light receiving surface of the imaging element and is configured to form an image of the living tissue on the light receiving surface.

[0013] the image processing unit,

[0014] It is equipped with a three-dimensional expansion processing unit, which is configured to calculate the different directions of the characteristic parts visible from different camera positions based on the position information of the characteristic parts that are commonly contained in the camera images taken by the endoscope at at least two different camera positions and are identifiable relative to other parts, and expand the two-dimensional information of the characteristic part images into three-dimensional information.

[0015] The imaging positions are different positions where the endoscope moves relative to the living tissue;

[0016] Preferably, the three-dimensional extension processing unit calculates the three-dimensional information by using different directions of the characteristic portion visible from different imaging positions and distance information between the imaging positions.

[0017] Another aspect of the present invention is also an endoscope system, including an endoscope for photographing living tissue in a body cavity, and an image processing unit for processing images photographed by the endoscope.

[0018] The endoscope comprises:

[0019] an imaging element configured to capture an image of living tissue; and

[0020] an objective lens disposed in front of the light-receiving surface of the imaging element and configured to simultaneously form images of living tissue acquired through a plurality of windows on the light-receiving surface as the imaging images;

[0021] the image processing unit,

[0022] It is equipped with a three-dimensional expansion processing unit, which is configured to calculate the different directions of the characteristic parts visible from the multiple windows based on the respective position information of the characteristic part images that are commonly contained in the multiple images obtained through the multiple windows in the camera image taken by the endoscope and are identifiable relative to other parts, and expand the two-dimensional information of the characteristic parts into three-dimensional information.

[0023] The plurality of windows include a front window facing the front of the light-receiving surface of the imaging element, and side windows facing the side of the imaging element opposite to the front window;

[0024] The objective lens is configured to simultaneously form a direct view image of the living tissue obtained through the front window and a side view image of the living tissue obtained through the side window on the light receiving surface as the camera image;

[0025] The three-dimensional expansion processing unit is preferably configured to calculate the different directions of the characteristic part visible from the front window and the side window based on the position information of the characteristic part in the direct observation image commonly contained in the straight view image and the side view image in the camera image taken by the endoscope, and the position information of the characteristic part in the side view image, so as to expand the two-dimensional information into three-dimensional information.

[0026] The plurality of images include images of overlapping regions, the images including the same portion of the living tissue as one another;

[0027] Preferably, the respective position information of the characteristic portion images is information when the characteristic portion is located in the overlapping area.

[0028] Preferably, the imaging element continuously images the living tissue.

[0029] The plurality of images including the characteristic portion are images captured at different imaging positions before and after the endoscope moves within the body cavity.

[0030] Preferably, the image processing unit is provided with an identity determination section configured to determine whether the characteristic portions in the plurality of images are identical to each other using at least one of color information and contour shape information of the characteristic portions.

[0031] Preferably, the three-dimensional expansion processing unit has a prediction model and is configured to obtain the three-dimensional information by inputting the two-dimensional information of the camera image taken by the endoscope into the prediction model, and the prediction model uses the two-dimensional information and the three-dimensional information of the characteristic part image in multiple camera images including the characteristic part of the living tissue taken by the endoscope as learning data to machine learn the relationship between the two-dimensional information and the three-dimensional information.

[0032] Another aspect of the present invention is also an endoscope system, including an endoscope for photographing living tissue in a body cavity, and an image processing unit for processing images photographed by the endoscope.

[0033] The endoscope comprises:

[0034] an imaging element configured to capture an image of living tissue; and

[0035] The objective lens is provided in front of the light receiving surface of the imaging element and is configured to form an image of the living tissue on the light receiving surface.

[0036] the image processing unit,

[0037] A prediction unit having a prediction model is provided. The prediction model is configured to predict three-dimensional information from the two-dimensional information by inputting two-dimensional information of a characteristic portion that is identifiable from other portions in the camera image taken by the endoscope.

[0038] The prediction model is a machine learning model, that is, it uses multiple known camera images containing the feature parts as learning data to predict the relationship between the known two-dimensional information of the feature parts in the camera images and the known three-dimensional information of the feature parts.

[0039] Preferably, the endoscope system is equipped with a display configured to: display an image using the three-dimensional information obtained by the prediction unit;

[0040] The imaging element is configured to simultaneously capture images obtained through a plurality of windows;

[0041] The imaging element continuously captures images of living tissue.

[0042] Within the field of view of the imaging element, there is a blind spot where no living tissue can be captured through any of the plurality of windows.

[0043] The image processing unit includes an image display control unit, which controls when at least a portion of the feature part is about to be in the blind spot area and before at least a portion of the feature part is in the blind spot area, to display a three-dimensional image of the entire feature part on the display using the three-dimensional information predicted by the two-dimensional information of the feature part.

[0044] Furthermore, preferably, the endoscope system is equipped with a display configured to: display an image using the three-dimensional information obtained by the prediction unit;

[0045] The imaging element is configured to simultaneously capture images obtained through a plurality of windows;

[0046] The imaging element continuously captures images of living tissue.

[0047] Within the field of view of the imaging element, there is an overlapping area where images can be taken simultaneously through a plurality of the windows.

[0048] The image processing unit includes an image display control unit, which controls the display of a three-dimensional image of the entire feature part on the display using the three-dimensional information predicted by the two-dimensional information of the feature part before at least a part of the feature part is about to be in the overlapping area.

[0049] Effects of the Invention

[0050] According to the above-described endoscope system, when imaging living tissue, the captured image can be used to obtain three-dimensional information of a characteristic portion such as a lesion. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1A This is an exemplary diagram of the field of view as viewed from the front window and side windows provided at the distal end portion of an endoscope according to one embodiment.

[0052] Figure 1B express Figure 1A The figure shows an example of the lateral position of the distal end of an endoscope within an organ and the left-right difference in the field of view at that time.

[0053] Figure 2A The diagram illustrates characteristic parts of living tissue and the direction of sight toward the characteristic parts as viewed from the front window and the side windows.

[0054] Figure 2B express Figure 1A An example of a captured image of the characteristic portion shown.

[0055] Figure 3 This is an oblique view of the appearance of an endoscope according to one embodiment.

[0056] Figure 4 A block diagram of an endoscope system according to one embodiment.

[0057] Figure 5 This is a diagram illustrating an example of the configuration of the distal end portion of an endoscope according to one embodiment.

[0058] Figure 6 A diagram illustrating an example of the configuration of an image processing unit of an endoscope system according to an embodiment.

[0059] Figure 7A diagram illustrating an example of the configuration of an image processing unit of an endoscope system according to an embodiment.

[0060] Figure 8A This section describes, in time series, the position of the distal end portion of the endoscope relative to a characteristic portion and examples of captured images displayed on a monitor.

[0061] Figure 8B This section describes, in time series, the position of the distal end portion of the endoscope relative to a characteristic portion and examples of captured images displayed on a monitor.

[0062] Figure 8C This section describes, in time series, the position of the distal end portion of the endoscope relative to a characteristic portion and examples of captured images displayed on a monitor.

[0063] Figure 9A Illustration showing the position of the endoscope tip relative to the characteristic portion.

[0064] Figure 9B Illustrations of examples of images displayed on a display of an endoscope system according to one embodiment. DETAILED DESCRIPTION

[0065] The present invention relates to patent application No. 2020-79593 filed with the Japan Patent Office on April 28, 2020, the entire contents of which are incorporated herein by reference.

[0066] Hereinafter, one embodiment of the endoscope system will be described in detail.

[0067] (Endoscope System Overview)

[0068] An endoscope system according to one embodiment includes: an endoscope for imaging living tissue within a body cavity; and an image processing unit for performing image processing on the camera image captured by the endoscope. The endoscope includes: an imaging element configured to capture an image of living tissue; and an objective lens disposed in front of a light-receiving surface of the imaging element and configured to form an image of the living tissue on the light-receiving surface. The image processing unit includes a three-dimensional expansion processing unit. The three-dimensional expansion processing unit calculates information on different directions (hereinafter referred to as line-of-sight directions toward the characteristic portion) of the characteristic portion visible from different imaging positions or through different windows based on positional information of characteristic portions commonly included in at least two camera images captured by the endoscope at at least two different imaging positions, or commonly included in a single camera image obtained by integrating at least two images with different fields of view captured through at least two different windows. The characteristic portion is distinguished from other non-characteristic portions, for example, by the color component of the living tissue. For example, compared to a healthy area on the surface of a living tissue that appears yellow or green due to the mucous membrane, an inflamed area that appears red due to swelling or easy bleeding, or an ulcer area that appears white due to white fur or pus-like mucus can be used as a characteristic part.

[0069] Furthermore, the image processing unit is configured to expand the two-dimensional information of the image of the characteristic portion into three-dimensional information using the calculated information of the line-of-sight direction toward the characteristic portion.

[0070] Three-dimensional information includes, for example, two-dimensional positional information of the inner surface of an organ and depth-direction positional information of characteristic features in camera images. In addition to two-dimensional information, depth-direction positional information can also be obtained by using common characteristic features displayed in camera images captured from at least two different camera positions, or by using common characteristic features displayed in a single camera image obtained by merging images viewed through at least two windows (displayed in the overlapping area of ​​two fields of view).

[0071] The three-dimensional information thus obtained can be used to display the image of the characteristic portion in a three-dimensional shape on a display through rendering processing, etc. Furthermore, the captured images of two characteristic portions that appear in a captured image due to the characteristic portions being located in an overlapping region can be combined into a single image of the characteristic portion and displayed on the display.

[0072] Figure 1A This is an example of the field of view as seen from the front window W1 and side window W2 provided at the distal end T of an endoscope according to one embodiment. The side window W2 is provided along the outer circumference of the cylindrical distal end T. The front window W1 is provided in front of the light-receiving surface of the imaging element I provided at the distal end T, and can observe the direction in front of the light-receiving surface (in the direction of the image sensor I) through the objective lens. Figure 1AThe side window W2 is provided in front of the light receiving surface of the imaging element 1, and can observe the direction perpendicular to the front ( Figure 1A The side windows W2 in this embodiment are windows facing a side perpendicular to the front. However, unlike the front window W1, they only need to face to the side. The orientation of the side windows W2 (the direction of the central axis of the field of view as seen from the side windows W2) can be inclined, for example, 30 to 90 degrees relative to the front.

[0073] The field of view visible through the front window W1 and the unillustrated objective lens provided at the tip portion T includes an area A1 and an area B, and the field of view visible through the side window W2 and the unillustrated objective lens provided at the tip portion T includes an area A2 and an area B. Area B overlaps with the field of view visible from the front window W1 and the field of view visible from the side window W2, so area B is hereinafter referred to as overlapping area B.

[0074] On the other hand, the visual field visible from the front window W1 and the side window W2 does not include the illustrated area C. That is, the subject in area C is not included in any visual field and therefore does not appear in the image captured by the imaging element 1. Hereinafter, area C is referred to as a blind spot C.

[0075] Figure 1B An example of the lateral position of the endoscope's tip T within an organ, and the difference in the left and right field of view at that time. Figure 1B As shown in FIG, the visual range of the living tissue surface in the organ depends on the lateral position of the tip T in the organ. Figure 1B The overall visual field of the right front window W1 and the side window W2 includes the area A1, the overlapping area B and the area A2. Figure 1B The total field of view seen by the left front window W1 and the side window W2 includes the area A1 and the area A2, but does not include the blind area C. Therefore, the blind area C exists in the field of view of the imaging element 1.

[0076] exist Figure 1B In the example shown, in the right-hand field of view, due to the presence of overlapping region B, a characteristic portion having a different color from other portions exists within overlapping region B. For example, if a lesion is present, the lesion is located both within the field of view visible from the front window W1 and within the field of view visible from the side window W2. Therefore, the imaging element 1 can capture, through the objective lens, the image of the characteristic portion seen from the front window W1 and the image of the characteristic portion seen from the side window W2 as a single image.

[0077] Figure 2A The figures illustrate the characteristic portion S, and are views showing the direction of sight toward the characteristic portion S as viewed from the front window W1 and the side windows W2. Figure 2Bexpress Figure 2A An example of a captured image of the characteristic portion S is shown.

[0078] Figure 2A and Figure 2B The central axis Ax1, which passes through the center of the front window W1 and extends in the direction normal to the window surface, and the central axis Ax2, which passes through the center of the window width of the side window W2 and extends in the direction normal to the window surface, are shown. Central axis Ax1 is located on the extension line of the objective lens optical axis, described later, and is represented as a dot in the camera image. Meanwhile, the side window W2 is arranged to circumscribe the circumference of the cylindrical tip T, and therefore appears as a circular shape in the camera image.

[0079] like Figure 2A As shown, since the characteristic portion S is located in the overlapping area B, Figure 2B As shown in FIG, two images of the characteristic portion S appear in the captured image. When two images of the characteristic portion S appear in the captured image, the 3D expansion processing unit immediately uses the lines of sight V1 and V2 from the two images of the characteristic portion S to obtain position information in the depth direction (forward) using the principle of triangulation.

[0080] It should be noted that the depth direction viewed from the front window W1 and side window W2 is Figure 2A In the example shown, the direction is not upward, but rather a direction inclined from the top to the side. For example, the lines of sight V1 and V2 can be used to determine the distance and direction from the center of the front window W1 to the characteristic portion S. Therefore, the distance and direction of the characteristic portion S can be used to refer to the boundary between the characteristic portion S and the non-characteristic portion to determine the height position information of the characteristic portion S from the surface of the organ. In addition, the forward direction from the center of the front window W1 to the characteristic portion S can also be determined ( Figure 2A Position information of the display (center is upward).

[0081] Therefore, when the characteristic part S enters the overlapping area B and two images of the characteristic part S appear, the three-dimensional expansion processing unit immediately obtains the above-mentioned three-dimensional information, including the position information of the characteristic part S protruding from the surface of the living tissue in the height direction, and also immediately obtains the three-dimensional position information of the characteristic part S in the forward direction.

[0082] exist Figure 2A In the example shown, the distal end of the endoscope is moved downward to take an image of the surface of living tissue in an organ. Figure 2A The characteristic portion S shown is located before the overlapping area B. The characteristic portion S is located in the field of view A2 of the side window W2 and only one image appears in the camera image. After this, two images of the characteristic portion S appear in the camera image, as shown in FIG. Figure 2B As shown. Figure 2BIn the example shown, a circular field of view is used, but the captured image may also be displayed on a display screen with a rectangular shape that cuts off a portion of the circle. Since the characteristic portion S is located in area A1, the image of the characteristic portion S appears as a single image in the captured image.

[0083] Therefore, if a characteristic portion S is located within overlapping region B and images of the characteristic portion S appear in both camera images, overlapping region B can be eliminated, and an image of the surface within the organ can be generated, as if region A1, overlapping region B, and region A2 were viewed through a single window (e.g., front window W1). Furthermore, since three-dimensional information can be immediately obtained when two images of the characteristic portion S appear, when the image of the characteristic portion S is displayed on a monitor, this three-dimensional information can be reflected in the image, thereby forming a three-dimensional image. For example, since information about the height of the characteristic portion S can be obtained, rendering processing can also be performed to reproduce the surface unevenness of the characteristic portion S relative to the surrounding area.

[0084] The above-mentioned process of expanding the image into three-dimensional information is performed using information on the direction of sight from the front window W1 and the side window W2 toward the visible characteristic portion S. However, this is not necessarily limited to the front window W1 and the side window W2. Multiple windows located at different positions can also be used, as long as the principle of triangulation can be applied. For example, multiple windows can be provided on the tip surface of the tip portion T. Furthermore, as long as the principle of triangulation can be applied, the process of expanding the image into three-dimensional information can also be performed using two or more camera images captured at different positions while the tip portion T is moved within an organ, i.e., images in which the characteristic portion S appears. In this case, in order to perform the process of expanding the image into three-dimensional information, it is preferable to obtain at least distance information between the camera positions, which is the camera position information. Therefore, three-dimensional information can be obtained using the direction of sight from the same characteristic portion S and the obtained distance information.

[0085] In order to obtain three-dimensional information more accurately, it is also possible to use camera images observed from three or more windows, or camera images from three or more camera positions. In this case, due to the excessive amount of information, it is preferable to use the principle of triangulation to obtain three-dimensional information in a way that minimizes the error.

[0086] (Specific methods of endoscope system)

[0087] Figure 3 This is an oblique view of the appearance of an endoscope according to one embodiment. Figure 4 A block diagram of an endoscope system according to one embodiment. Figure 4 This is a diagram showing an example of the configuration of the distal end portion of an endoscope according to one embodiment.

[0088] Figure 3The endoscope (hereinafter referred to as electronic endoscope) 100 shown is connected to Figure 4 The electronic endoscope processor 200 shown in the figure forms an endoscope system 1. The endoscope system 1 is a system designed specifically for medical use, and is mainly equipped with an electronic endoscope 100, an electronic endoscope processor 200 and a display 300. Figure 4 The electronic endoscope 100 and the display 300 are connected to the processor 200 respectively.

[0089] like Figure 3 As shown, the electronic endoscope 100 mainly includes a connector 110, an operating unit 120, and a distal end 132. It also includes a flexible cable 130 extending from the operating unit 120 to the distal end 132 and having flexibility, a curved tube 134 connected to the front of the flexible cable 130 via a connecting unit and freely bendable, and a universal tube 128 extending rearward from the operating unit 120. The connector 110 is fixed to the rear end of the universal tube 128 and is configured to be connected to the processor 200.

[0090] A plurality of bending wires are inserted into the operating portion 120, the flexible cable 130, and the bending tube 134. The tip of each bending wire is connected to the rear end of the bending tube 134, and the rear end of each bending wire is connected to the bending operation knob 122 of the operating portion 120. The bending tube 134 is bent in any direction and at any angle according to the operation of the bending operation knob 122.

[0091] Furthermore, the operation section 120 has a plurality of operation buttons 124. The operation buttons 124 allow the endoscope operator (the operator or an assistant) to instruct the following functions by pressing the operation buttons 124: discharge of water or gas from the air and water supply ports provided on the tip surface of the tip portion 132, suction of liquid or gas in living tissue through the suction port, and discharge of cleaning liquid from the cleaning liquid discharge nozzle to clean the objective lens. The operation to be performed by pressing the operation button 124 can be predetermined, and a function can be assigned to the operation button 124 to perform that operation.

[0092] The tip portion 132 of the bent pipe 134 is made of a hard resin material (such as ABS, modified PPO, PSU, etc.) and is basically not elastically deformable.

[0093] An LED light source 102 and an imaging element 108 are located within the distal end 132, immediately after the objective lens 106. Specifically, the distal end 132, disposed at the front end of the curved tube 134, includes the LED light source 102, the objective lens 106, and the imaging element 108. The objective lens 106 is disposed in front of the imaging element 108 and images living tissue at a viewing angle greater than or equal to 180 degrees, preferably, greater than 180 degrees, on the light-receiving surface of the imaging element 108. As described later, the distal end 132 is provided with a front window facing forward of the light-receiving surface of the imaging element 108 and side windows facing to the sides perpendicular to the front. The imaging element 108 is configured to capture the image formed on the light-receiving surface by the objective lens 106 through the front window and the side windows.

[0094] Flexible cable 130, curved tube 134, and distal end 132 form insertion portion 135, which is inserted into a body cavity. An image signal cable extending from imaging element 108 disposed at distal end 132 extends from distal end 132 through curved tube 134, flexible cable 130, operating unit 120, and universal tube 128 into connector 110. Connector 110 is connected to processor 200. Processor 200 processes image signals transmitted from the imaging element and controls the display 300 to display images of a subject captured by imaging element 108.

[0095] like Figure 4 The processor 200 of the endoscope system 1 shown includes a system controller 202 and a timing controller 206. The system controller 202 executes various programs stored in a memory 204 and provides integrated control of the entire endoscope system 1. Furthermore, the system controller 202 changes various settings of the endoscope system 1 based on instructions input by the endoscope operator (operator or assistant) via an operation panel 208. The timing controller 206 outputs clock pulses to various circuits within the endoscope system 1 for adjusting the operating timing of each component.

[0096] In addition to the imaging element 108, an LED light source 102 is provided in the distal end portion 132 of the electronic endoscope 100. The LED light source 102 emits illumination light that illuminates living tissue, and the imaging element 108 captures the light.

[0097] The LED light source 102 is driven by a drive signal generated by a light source control circuit 116 provided in the connector 110, thereby emitting light. A laser element may be used instead of the LED light source 102, or a high-intensity lamp such as a xenon lamp, a metal halide lamp, a mercury lamp, or a halogen lamp may be used.

[0098] exist Figure 4In the example shown, the LED light source 102 is provided at the front end portion 132, but it may also be provided as a light source device at the connector 110 or the processor 200. In this case, illumination light from the light source device to the front end portion 132 is guided to the front end portion 132 by a light guide bundled with multiple optical cables.

[0099] The light emitted from the LED light source 102 is illuminated as the illumination light, and is irradiated onto the living tissue as the subject through the light distribution lens 104. The reflected light from the living tissue is reflected through the front window 140 and the side window 150 (see FIG. Figure 5 ) and the objective lens 106 form an optical image on the light receiving surface of the imaging element 108.

[0100] The imaging element 108 is, for example, a single-chip color CCD (Charge-Coupled Device) image sensor with various filters arranged on its light-receiving surface, such as an IR (Infrared) cutoff filter 108a and a Bayer array color filter 108b. It generates R (Red), G (Green), and B (Blue) primary color signals based on the optical image formed on the light-receiving surface. Instead of a single-chip color CCD image sensor, a single-chip color CMOS (Complementary Metal Oxide Semiconductor) image sensor may also be used. In this way, the electronic endoscope 100 uses the imaging element 108 to capture living tissue within an organ and generate moving images.

[0101] A driver signal processing circuit 112 is provided within the connector 110 of the electronic endoscope 100. The driver signal processing circuit 112 performs predetermined signal processing such as color interpolation and matrix operations on the primary color signals input from the imaging element 108 to generate image signals (luminance signal Y and color difference signals Cb and Cr). The generated image signals are then output to the image processing unit 220 of the electronic endoscope processor 200. Furthermore, the driver signal processing circuit 112 accesses the memory 114 and reads information unique to the electronic endoscope 100. The information unique to the electronic endoscope 100 stored in the memory 114 includes, for example, the number of pixels, sensitivity, operable frame rate, and model number of the imaging element 108. The driver signal processing circuit 112 outputs the unique information read from the memory 114 to the system controller 202.

[0102] The system controller 202 performs various operations and generates control signals based on information stored in the memory 204 and information specific to the electronic endoscope 100. The system controller 202 uses the generated control signals to control the operation and timing of various circuits within the electronic endoscope processor 200 so as to execute processing appropriate for the electronic endoscope 100 connected to the electronic endoscope processor 200.

[0103] The timing controller 206 supplies clock pulses to the driver signal processing circuit 112, the image processing unit 220, and the light source section 230 in accordance with the timing control executed by the system controller 202. The driver signal processing circuit 112 drives and controls the imaging element 108 in accordance with the clock pulses supplied from the timing controller 206 at a timing synchronized with the image frame rate processed by the electronic endoscope processor 200.

[0104] The image processing unit 220 includes an image processing section 220A (see Figure 4 ), which generates a video signal for displaying an image or the like based on the image signal input from the driver signal processing circuit 112 under the control of the system controller 202, and outputs the video signal to the display 300.

[0105] The image processing unit 220 further includes a three-dimensional expansion processing unit 220B (see Figure 4 ), the three-dimensional expansion processing unit 220B is configured to expand the two-dimensional information of the image of the characteristic portion in the image captured by the imaging element 108 into three-dimensional information. The three-dimensional expansion processing unit 220B will be described later. The image processing unit 220A can also perform quantitative processing on the image of the living tissue obtained by the electronic endoscope 100, quantify the characteristic quantity of each pixel in the image that can distinguish between lesions and healthy parts, evaluate the degree of development of the lesion in the image, and further replace the numerical value of each pixel obtained by the quantitative processing with a color to generate a color map image. In this case, the image processing unit 220A generates a video signal for displaying the information and color map image generated by the quantitative processing, and outputs it to the display 300. As a result, the endoscope operator can perform a high-precision inspection through the image displayed on the display screen of the display 300. The image processing unit 220A outputs the image, the information of the quantitative processing result, and the color map image to the printer 400 as needed.

[0106] The processor 200 is connected to the server 600 via the NIC (network interface card) 210 and the network 500. The processor 200 can download information related to endoscopic examinations (e.g., patient electronic medical records and operator information) from the server 600. The downloaded information is displayed on, for example, the display screen of the display 300 or the operation panel 208. The processor 200 can also upload the examination results of the electronic endoscope 100 to the server 600, so that the examination results are stored on the server 600.

[0107] Figure 5 Schematic diagram illustrating the internal structure of the tip portion 132 .

[0108] The front end portion 132 includes an objective lens 106, an imaging element 108, a front window 140, and a side window 150. The objective lens 106 and the imaging element 108 are located in a cylindrical member 133 made of a hard resin material at the front end portion 132. The front end portion 132 corresponds to Figure 1A 、 1B and the tip portion T in the example shown in 2A.

[0109] The front window 140 faces forward of the light receiving surface 108c of the imaging element 108. The side windows 150 face laterally perpendicular to the front. As described above, the side windows 150 only need to face laterally relative to the front window 140.

[0110] Objective lens 106 comprises a set of lenses 106a to 106e, including a meniscus lens, a convex lens, and a concave lens. Its half-angle of view exceeds 90 degrees, preferably greater than or equal to 110 degrees. Therefore, objective lens 106 simultaneously forms, on light-receiving surface 108c, a direct view image of the living tissue, obtained through front window 140, and a side view image of the living tissue, obtained through side window 150. The subject-facing surface of lens 106a also serves as front window 140. Side window 150 is provided with a glass cover.

[0111] like Figure 1A As shown, the visual field visible through the front window 140 and the side window 150 of the tip portion 132 includes area A1, an overlapping area B, and area A2. Furthermore, there is a blind spot C where the subject cannot be seen from the front window 140 and the side window 150. The image of the living tissue is simultaneously captured as a straight-view image obtained through the front window 140 and a side-view image obtained through the side window 150. Therefore, the characteristic portion S can be displayed as two images in the captured images, as shown in FIG. Figure 2B shown.

[0112] The image processing unit 220 is provided with a three-dimensional expansion processing section 220B that calculates three-dimensional information of the characteristic portions S from two-dimensional information (x coordinate position and y coordinate position on the image) of the two characteristic portions S appearing in the captured image.

[0113] The three-dimensional expansion processing unit 220B calculates the line-of-sight direction of the characteristic portion S as seen through the front window 140 and the side window 150 based on the common positional information contained in the two images (the front view image and the side view image) of the characteristic portion S captured by the electronic endoscope 100. The calculated line-of-sight direction is then used to expand the two-dimensional information of the image of the characteristic portion S into three-dimensional information. Whether the two images captured through the front window 140 and the side window 150 are common can be determined based on the positions of the two characteristic portions S in the captured images and the color information or contour shape information of the characteristic portions S. Furthermore, when the distal end portion 132 of the electronic endoscope 100 is moved in one direction while capturing images of the surface of living tissue, the characteristic portion S moves from area A2 to the overlapping area B. Therefore, using multiple captured images generated in a time series, it is possible to easily determine whether two common characteristic portions S appear in the captured images.

[0114] Furthermore, the two-dimensional information of the image of the characteristic portion S can be expanded into three-dimensional information not only when capturing images while moving the distal end portion 132 of the electronic endoscope 100 in one direction but also when capturing images while bending the curved tube 134 .

[0115] The three-dimensional expansion processing unit 220B is based on ( Figure 2B The coordinate positions in the X and Y directions (in an XY coordinate system in which the X and Y directions are coordinate axes), that is, the two-dimensional information of each position of interest of the characteristic portion S in the direct view image obtained through the front window 140, are calculated relative to the azimuth direction of the central axis Ax1 of the front window 140 to determine the azimuth direction of each position of interest of the characteristic portion S. The central axis Ax1 is set to coincide with the optical axis of the objective lens 106, so that the azimuth direction of the characteristic portion S in the camera image is consistent with the azimuth direction viewed from the front window 140 at the distal end portion 132 within the organ. Figure 2B In the example shown, the right direction relative to the central axis Ax1 is the azimuth direction of the position of interest in the characteristic portion S. Similarly, the three-dimensional expansion processing unit 220B moves from the center point Ax2 of the central axis Ax2* of the side window 150 to the center point Ax2* of the side window 150. * The azimuth direction of each focus position of the characteristic portion S in the captured image visible through the side window 150 is obtained. The central axes Ax1 and Ax2 are axes extending from the window centers of the front window 140 and the side window 150 in the normal direction of the window surface.

[0116] By using the color information of the pixel values ​​in the characteristic portion S or the information of the outline shape, the position of interest when obtaining the azimuth direction relative to the center axis Ax1 and the position of interest when obtaining the azimuth direction from the center point Ax2* are determined to be the same position. In addition, the side view image and the front view image are images obtained using the same objective lens 106, so the azimuth direction of the position of the characteristic portion S in the side view image as seen from the center axis Ax1 is consistent with the azimuth direction of the same position of the same characteristic portion S in the front view image as seen from the center axis Ax1. Therefore, the center point Ax2* is the same as the center point Ax2*. * The central axis Ax1 ( Figure 2B The straight line connecting the position of the characteristic part S and the central axis Ax2 ( Figure 2B The intersection point where the two axes intersect (as shown by the dotted circle).

[0117] On the other hand, the central axis Ax2 is an axis extending from the window center of the side window 150 in the normal direction. Therefore, the position coordinates of the focus position based on the center point Ax2* in the side view image are obtained, and the lens characteristics of the objective lens 106 are calculated based on the position coordinates. Figure 2A The elevation direction shown (relative to Figure 2A The angle θ of the lateral upper direction (forward direction) can be obtained from this. Figure 2A and Figure 2B The directions of the sight lines V1 and V2 of the characteristic part S are shown.

[0118] Furthermore, since the positions of the window centers of the front window 140 and the side windows 150 in the tip portion 132 are known, the three-dimensional expansion processing unit 220B calculates the position where the line of sight V1 toward the characteristic portion S and the line of sight V2 toward the characteristic portion S intersect based on the positions and the directions of the lines of sight V1 and V2 toward the characteristic portion S obtained, using the principle of triangulation.

[0119] Therefore, the three-dimensional expansion processing unit 220B can calculate the azimuth direction, elevation direction, and distance information of each position of the characteristic portion S from the window center of the front window 140 or the window center of the side window 150. The three-dimensional expansion processing unit 220B can calculate the position information of the characteristic portion S relative to the tip portion 132 along the direction in which the organ extends, based on the azimuth direction, elevation direction, and distance information, and can also calculate the horizontal direction ( Figure 2A Thus, the three-dimensional information of the characteristic portion S is obtained.

[0120] In obtaining Figure 2BIn the case of the captured image shown in , as described above, the image processing unit 220 obtains three-dimensional information of the characteristic portion S. However, in the captured image, processing can be performed to eliminate the overlapping region B, and an image of the surface of the organ can be created, as if the region A1, the overlapping region B, and the region A2 were viewed through the front window W1, for display on the display 300. In this case, by performing three-dimensional processing using the acquired three-dimensional information of the characteristic portion S, the image of the characteristic portion S can be displayed on the display 300 as a three-dimensional image. The distance information in the lateral direction from the center of the front window 140 to the position of interest of the characteristic portion S can be used to obtain information in the height direction of the characteristic portion S. Therefore, as a three-dimensional processing, for example, rendering processing can be performed to reproduce the surface unevenness of the characteristic portion S. In this way, the characteristic portion S can be displayed on the display 300 as a three-dimensional image, which is very useful for determining whether the tumor is a malignant tumor requiring resection and for diagnosis.

[0121] If it is known from a previous diagnosis that there is a lesion at the bottom of the folds in an organ such as the large intestine with folds on the inner surface, the distal end portion 132 can be positioned so that the lesion is in the overlapping area B, and the above-mentioned endoscope system can be used to obtain three-dimensional information of the lesion through the above-mentioned method, and information on irregularities on the surface of the lesion can also be obtained.

[0122] In the above embodiment, the two-dimensional information of the characteristic portion S is expanded into three-dimensional information using the living tissue image viewed through the front window 140 and the side window 150. However, the windows provided in the distal end portion 132 are not limited to the front window 140 and the side window 150. Multiple front windows provided at different positions in the distal end portion 132 or multiple side windows provided at different positions in the distal end portion 132 may also be used. In this case, the two windows are provided at different positions in the distal end portion 132, so the imaging positions are different. Therefore, the three-dimensional expansion processing unit 220B can expand the two-dimensional information of the image of the characteristic portion S into three-dimensional information based on the positional information of the characteristic portion S commonly included in the camera images captured at at least two camera positions, and using the line of sight directions of the characteristic portion S visible at the different camera positions.

[0123] The imaging positions described above may be different positions of the electronic endoscope 100 as it moves relative to the living tissue, for example, different positions in the direction in which the organ extends. In this case, the three-dimensional expansion processing unit 220B preferably calculates three-dimensional information using the different directions in which the characteristic portion S is visible due to the different imaging positions and the distance information between the imaging positions. The distance information between the imaging positions can be obtained from information about the imaging position of the distal end portion 132 in the position measurement system of the electronic endoscope 100. For example, the position measurement system refers to a system that uses a magnetic sensor to obtain the position of the imaging element 108 at the distal end portion 132 of the electronic endoscope 100 inserted into the organ, as well as the subsequent positions of the flexible cable 130, or the insertion length of the electronic endoscope 100 inserted through the open end of the organ.

[0124] As in the above-described embodiment, preferably, objective lens 106 is configured to simultaneously form an image on the light-receiving surface of a front view image of the living tissue obtained through front window 140 and a side view image of the living tissue obtained through side window 150 as camera images. The three-dimensional expansion processing unit 220B is configured to calculate the line-of-sight direction of characteristic portion S as seen through front window 140 and side window 150 based on the positional information in the front view image of the characteristic portion S, which is included in both the front view image and the side view image of the captured camera images, and to expand the two-dimensional information into three-dimensional information using the calculated line-of-sight direction. Because the direction in which front window 140 and side window 150 are orthogonal to each other is the direction of the central axis of the field of view, the three-dimensional information can be calculated more accurately.

[0125] As described above, the images captured through different windows (front window 140 and side window 150) include overlapping regions B containing the same portion of living tissue, and the positional information of the characteristic portion S in each image is information obtained when the characteristic portion S is located in the overlapping region B. Therefore, three-dimensional information can be accurately calculated from the positional information of each position in the image of the characteristic portion S in a single captured image.

[0126] As described above, if two or more different imaging positions are used, the directions of sight lines to the characteristic portion S are different. Therefore, when the imaging element 108 continuously images living tissue, multiple images containing the same characteristic portion S may be images captured at different imaging positions before and after the distal end portion 132 of the electronic endoscope 100 moves within the body cavity. Here, different imaging positions refer to imaging positions that differ in the direction in which the organ extends.

[0127] According to one embodiment, Figure 6 As shown, in addition to the image processing unit 220A and the three-dimensional expansion processing unit 220B, the image processing unit 220 is preferably further equipped with an identity determination unit 220C. Figure 6This is a diagram illustrating an example of the configuration of an image processing unit of an endoscope system according to one embodiment. The identity determination unit 220C is configured to determine whether the feature portions S included in multiple images (e.g., a straight-view image and a side-view image) are identical using at least one of the color information of the feature portion S and the contour shape information of the feature portion S. When shooting a video, the image of the feature portion S moves in one direction in the camera image, so it is easy to determine whether it is identical. However, if there are multiple feature portions S, it may be difficult to identify whether they are identical. Therefore, preferably, when calculating the three-dimensional information of the feature portion S, the identity determination unit 220C determines whether they are identical.

[0128] According to one embodiment, the three-dimensional expansion processing unit 220B is preferably equipped with a prediction model that uses the two-dimensional and three-dimensional information in multiple camera images previously captured by the electronic endoscope 100, including the characteristic portion S of living tissue, as learning data, and machine-learns the relationship between the two-dimensional and three-dimensional information. Preferably, the prediction model is configured to obtain three-dimensional information by inputting the two-dimensional information of the image of the characteristic portion S in the image currently captured by the electronic endoscope 100 into the prediction model. Based on the color information contained in the two-dimensional information and the size of the characteristic portion S, three-dimensional information including the surface unevenness information of the characteristic portion S can be predicted. As machine learning for the prediction model, for example, deep learning based on neural networks (Deep Learning) is used. In addition, a random forest using a tree structure can also be used as part of the prediction model. As the prediction model, well-known models such as convolutional neural networks and stacked autoencoders can be used.

[0129] By equipping such a prediction model with machine learning, the three-dimensional information of the feature part S can be obtained without calculating the directions of the sight lines V1 and V2 of the feature part S by the three-dimensional expansion processing unit 220B. Figure 7 As shown, the image processing unit 220 does not include a three-dimensional expansion processing section 220B, but includes a prediction section 220D. Figure 7This is an example diagram of the structure of an image processing unit of an endoscope system according to one embodiment. Preferably, the prediction unit 220D is equipped with a prediction model, which is configured to predict three-dimensional information by inputting two-dimensional information of the image of the characteristic portion S contained in the image taken by the electronic endoscope 100. The prediction model is a machine learning model that uses the known two-dimensional information of the image of the characteristic portion S in a plurality of known camera images including the characteristic portion S and the known three-dimensional information of the corresponding characteristic portion S as learning data, and performs machine learning on the relationship between the two-dimensional information of the image of the characteristic portion S in the camera image and the three-dimensional information of the characteristic portion S. As a machine learning model, for example, deep learning based on a neural network is used. In addition, a random forest using a tree structure may also be used as part of the prediction model. As a prediction model, well-known models such as convolutional neural networks and stacked autoencoders can be used.

[0130] According to one embodiment, the image processing unit 220 includes an image display control unit 220E. When the imaging element 108 continuously captures video of living tissue and displays the image on the display 300 using three-dimensional information obtained from the captured images by the prediction unit 220D, subjects in the blind spot C are not displayed. When at least a portion of a characteristic portion S is located within the blind spot C, the image display control unit 220E preferably controls the display 300 to display a three-dimensional image of the entire characteristic portion S using three-dimensional information predicted from two-dimensional information of the image of the characteristic portion S contained in the captured images before at least a portion of the characteristic portion S is located within the blind spot C. The three-dimensional image is, for example, a rendered image.

[0131] Furthermore, preferably, when at least a portion of the characteristic portion S is located in the overlapping area B, the image display control unit 220E performs control so that before at least a portion of the characteristic portion S is located in the overlapping area B, an image of the entire characteristic portion S is displayed using three-dimensional information predicted based on two-dimensional information of the image of the characteristic portion S included in the camera image, and a three-dimensional image is displayed.

[0132] Figures 8A-8C 1 and 2 are exemplary diagrams describing the position of the tip portion T relative to the characteristic portion S and captured images in time series. Figures 8A-8C The upper portion shows the position of the tip portion T relative to the characteristic portion S, and Figures 8A-8C The lower part of FIG shows an example of a display screen 300A of the display. Figures 8A to 8C The example shown is using Figure 1B The examples shown are for illustration.

[0133] When Figures 8A to 8CAs shown in FIG, when the electronic endoscope 100 is pulled downward to be inserted into the deepest part of the organ to be examined and the surface of the living tissue is continuously photographed, as shown in FIG. Figure 8A As shown, the characteristic portion S is initially located in the area A2 of the visual field of the side window W2, and therefore, the characteristic portion S appears on the display screen 300A of the display. Thereafter, when the electronic endoscope 100 is moved, as shown in FIG. Figure 8B As shown in FIG. 3 , the characteristic portion S enters the blind area C. Therefore, the characteristic portion S disappears from the display screen 300A. When the electronic endoscope 100 is moved again, as shown in FIG. Figure 8C As shown, the characteristic portion S enters the area A1. Therefore, the characteristic portion S is displayed again on the display screen 300A.

[0134] like Figure 8B As shown, when characteristic portion S is located in blind spot C, image display control unit 220E uses the three-dimensional information predicted by prediction unit 220D based on the two-dimensional information of the image of characteristic portion S located in area A2 preceding blind spot C, and controls the display 300 to display the entire image of characteristic portion S as a three-dimensional image. Therefore, image display control unit 220E extracts the edge region of characteristic portion S using color information and monitors the movement of this extracted edge region over time. When at least a portion of characteristic portion S enters blind spot C, image display control unit 220E controls the display 300 to display the entire image of characteristic portion S as a three-dimensional image using the three-dimensional information predicted by prediction unit 220D.

[0135] Furthermore, the image display control unit 220E monitors the movement of the characteristic portion S of the edge area extracted by the color information, and when at least a part of the characteristic portion S enters the overlapping area B, the image display control unit 220E generates an image so that the image of the characteristic portion S becomes one image, and controls the image display of the characteristic portion S to be a three-dimensional image generated using the three-dimensional information predicted by the prediction unit 220D.

[0136] Figure 9A An example of the position of the tip portion T relative to the characteristic portion S is shown. Figure 9B FIG. 2 shows an example of an image displayed on a display of an embodiment of an endoscope system. Figure 9A As shown, there are characteristic parts S1-S4 on the surface of the living tissue, characteristic part S1 is located in area A1, characteristic part S2 is located in overlapping area B, characteristic part S3 is located in blind area C, and characteristic part S4 is located in area A2.

[0137] In this case, the characteristic portion S2 located in the overlapping area B does not display two images on the screen but displays one image, while the characteristic portion S3 located in the blind area C displays an image without disappearing. The images of the characteristic portions S1 to S4 are all three-dimensional images.

[0138] Figure 9B The image display mode shown, that is, the image of the three-dimensional characteristic parts S1-S4, is preferably selected by pressing the operation button 124 on the operation unit 120 (see Figure 3 ). Therefore, preferably, the image display in the display 300 is switched by operating the button 124. Figure 9B The display shown switches at any time between the display of a conventional camera image using two-dimensional information, in which two images of the characteristic portion S2 in the overlapping area B appear, while the characteristic portion S3 in the blind area C is not displayed.

[0139] In the endoscope system 1, the three-dimensional expansion processing unit 220B, the prediction unit 220D, the identity determination unit 220C, and the image display control unit 220E are provided in the image processing unit 220 of the processor 200, but are not necessarily provided in the processor 200. For example, they may be installed in a data processing unit at another location via the network 500, and images and information may be transmitted and received via communication via the network 500.

[0140] The endoscope system provided by the present invention has been described in detail above. However, the present invention is not limited to the above-mentioned embodiments, and various improvements and changes can be made without departing from the scope of the present invention.

Claims

1. An endoscope system, comprising: endoscopes, which take images of living tissue inside body cavities; and an image processing unit, which performs image processing on the camera image taken by the endoscope, The endoscope includes: an imaging element configured to capture images of living tissue; and an objective lens disposed in front of the light receiving surface of the imaging element and configured to form an image of the living tissue on the light receiving surface, The image processing unit is equipped with a three-dimensional expansion processing unit, which is configured to calculate different directions of the characteristic part visible from different camera positions based on the position information of the characteristic part commonly included in the camera images taken by the endoscope at at least two different camera positions and identifiable relative to other parts, and expand the two-dimensional information of the characteristic part image into three-dimensional information. The three-dimensional expansion processing unit has a prediction model and is configured to predict the three-dimensional information and obtain the three-dimensional information by inputting two-dimensional information of a characteristic part contained in the camera image taken by the endoscope and identifiable relative to other parts into the prediction model. The prediction model uses multiple camera images of the characteristic part including living tissue taken by the endoscope as learning data, and performs machine learning on the relationship between the two-dimensional information of the image of the characteristic part in the camera image and the three-dimensional information of the image of the characteristic part.

2. The endoscope system according to claim 1, wherein: The imaging positions are different positions where the endoscope moves relative to the living tissue. The three-dimensional extension processing unit calculates the three-dimensional information using the different directions of the characteristic portion visible from the different imaging positions and distance information between the imaging positions.

3. An endoscope system, comprising: endoscopes, which take images of living tissue inside body cavities; and an image processing unit, which performs image processing on the camera image taken by the endoscope, The endoscope includes: an imaging element configured to capture images of living tissue; and an objective lens, which is arranged in front of the light receiving surface of the imaging element and is configured to form an image of the living tissue images obtained through the plurality of windows simultaneously on the light receiving surface as the imaging image; The image processing unit is equipped with a three-dimensional expansion processing unit, which is configured to calculate different directions of the characteristic parts visible from the multiple windows based on the respective position information of the characteristic part images that are respectively contained in the multiple images obtained through the multiple windows in the camera image taken by the endoscope and are identifiable relative to other parts, and expand the two-dimensional information of the characteristic parts into three-dimensional information. The three-dimensional expansion processing unit has a prediction model and is configured to predict the three-dimensional information and obtain the three-dimensional information by inputting two-dimensional information of a characteristic part contained in the camera image taken by the endoscope and identifiable relative to other parts into the prediction model. The prediction model uses multiple camera images of the characteristic part including living tissue taken by the endoscope as learning data, and performs machine learning on the relationship between the two-dimensional information of the image of the characteristic part in the camera image and the three-dimensional information of the image of the characteristic part.

4. The endoscope system according to claim 3, wherein: The plurality of windows include a front window facing the front of the light receiving surface of the imaging element and a side window facing the side of the imaging element relative to the front window. The objective lens is configured to simultaneously form a direct view image of the living tissue obtained through the front window and a side view image of the living tissue obtained through the side window on a light receiving surface as the camera image. The three-dimensional expansion processing unit is configured to calculate different directions of the feature part visible through the front window and the side window based on the position information of the feature part in the straight-view image and the side-view image respectively contained in the camera image taken by the endoscope, and the position information of the feature part in the side-view image, so as to expand the two-dimensional information into three-dimensional information.

5. The endoscope system according to claim 3 or 4, wherein: The plurality of images include images of overlapping regions, the images including the same portion of the living tissue as each other, The respective position information of the characteristic portion images is information when the characteristic portion is located in the overlapping area.

6. The endoscope system according to claim 3 or 4, wherein: The imaging element continuously images the living tissue. The plurality of images including the characteristic portion are images captured at different imaging positions before and after the endoscope moves in the body cavity.

7. The endoscope system according to any one of claims 1 to 4, wherein: The image processing unit is provided with an identity determination section configured to determine whether the characteristic portions in the plurality of images are identical to each other using at least one of color information and contour shape information of the characteristic portions.

8. An endoscope system comprising: endoscopes, which take images of living tissue inside body cavities; and an image processing unit, which performs image processing on the camera image taken by the endoscope, The endoscope includes: an imaging element configured to capture images of living tissue; and an objective lens disposed in front of the light receiving surface of the imaging element and configured to form an image of the living tissue on the light receiving surface, The image processing unit is provided with a prediction unit having a prediction model, wherein the prediction model is configured to predict three-dimensional information from the two-dimensional information by inputting two-dimensional information of a characteristic part that is identifiable relative to other parts contained in the camera image taken by the endoscope, The prediction model is a machine learning model, that is, using multiple known camera images containing the feature parts as learning data, the relationship between the known two-dimensional information and the known three-dimensional information of the feature parts in the camera images is predicted.

9. The endoscope system according to claim 8, wherein: The endoscope system is provided with a display configured to display an image using the three-dimensional information obtained by the prediction unit. The imaging element is configured to simultaneously capture images obtained through a plurality of windows. The imaging element continuously images the living tissue. There is a blind spot area within the field of view of the imaging element where no living tissue can be captured through any of the plurality of windows. The image processing unit includes an image display control unit, which controls when at least a portion of the feature part is about to be in the blind spot area and before at least a portion of the feature part is in the blind spot area, to display a three-dimensional image of the entire feature part on the display using the three-dimensional information predicted by the two-dimensional information of the feature part.

10. The endoscope system according to claim 8, wherein: The endoscope system is provided with a display configured to display an image using the three-dimensional information obtained by the prediction unit. The imaging element is configured to simultaneously capture images obtained through a plurality of windows. The imaging element continuously images the living tissue. There is an overlapping area within the field of view of the imaging element where images can be taken simultaneously through a plurality of windows. The image processing unit includes an image display control unit, which controls, when at least a portion of the feature portion is about to be in the overlapping area and before at least a portion of the feature portion is in the overlapping area, to display a three-dimensional image of the entire feature portion on the display using the three-dimensional information predicted by the two-dimensional information of the feature portion.

Images