Endoscopic system and its operating method
The endoscope system uses landmark detection and estimation processes to accurately locate targets despite reduced visibility, enhancing target identification and tracking.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- FUJIFILM CORP
- Filing Date
- 2022-03-07
- Publication Date
- 2026-07-30
AI Technical Summary
Existing endoscope systems struggle to accurately identify detection targets, such as bleeding sites, when visibility is reduced due to factors like pooled blood, making it difficult to determine their position.
The endoscope system employs a processor that performs multiple detection processes to identify landmarks and calculate estimated location information, displaying this information on the screen to aid in target identification even when visibility is reduced.
The system effectively specifies the position of detection targets even when visibility is decreased, ensuring accurate identification and tracking.
Smart Images
Figure 0007897836000001 
Figure 0007897836000002 
Figure 0007897836000003
Abstract
Description
Technical Field
[0004] ,
[0006] , , , , , , , , , , , , , , , ,
[0005]
[0001] The present invention relates to an endoscope system for detecting a detection target such as a bleeding site and an operation method thereof.
Background Art
[0002] In the medical field, an endoscope system having a light source device, an endoscope, and a processor device is widely used. In endoscope diagnosis, it is sometimes necessary to detect a detection target such as a bleeding site during an endoscopic procedure. The detection of the detection target has been performed not only by visual inspection but also by estimation based on comparison with past images. Patent Documents 1 to 3 describe detecting a bleeding site or region from an image.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Patent Document 2
Patent Document 3
Summary of the Invention
Problems to be Solved by the Invention
[0004] However, when factors that reduce the visibility of a detection target such as pooled blood due to bleeding occur over time, it has been difficult to identify the position of the detection target by visual inspection or from an image. [[ID=D4]] [[ID=D5]]
[0005] [[ID=D6]] [[ID=D7]] An object of the present invention is to provide an endoscope system and an operation method thereof that can identify the position of a detection target even when the visibility of the detection target is reduced.
Means for Solving the Problems
[0006] The endoscope system of the present invention includes a processor, which acquires an endoscope image, performs a first detection process on the endoscope image to detect a target and obtain the actual position information of the target, performs a second detection process on the endoscope image to detect a landmark and obtain the location information of the landmark, performs a landmark setting process to associate the location information of the landmark with the actual position information of the target, and performs a target undetected display process to display the location information of the landmark on the display if no target is detected and a landmark is detected. In the display process when no target is detected, the display of the landmark's location information on the display will continue until the landmark is no longer detected. .
[0007] Preferably, when the processor does not detect a target and a landmark is detected, it calculates the estimated location information of the target by performing a location information estimation process based on the location information of the landmark, and the display process when no target is detected displays the estimated location information of the target and the location information of the landmark on the display.
[0008] When the processor does not detect a target and a landmark is detected, it calculates the estimated location information of the target by performing a location information estimation process based on the location information of the landmark. The display has a main screen for displaying the endoscopic image and a sub-screen located at a different position from the main screen. In the display process when no target is detected, it is preferable to display the estimated location information of the target and the location information of the landmark on the sub-screen.
[0009] Preferably, the processor calculates the estimated location information of the target to be detected by location information estimation processing based on the location information of landmarks, and the display has a main screen that displays the endoscopic image and a sub-screen located at a different position from the main screen. In the display process when no target is detected, it is preferable that the main screen displays the estimated location information of the target to be detected and the location information of landmarks, and the sub-screen displays a still image of the endoscopic image acquired at the time the target to be detected, along with the location information of the target to be detected and the location information of landmarks.
[0010] In the landmark setting process, it is preferable to associate the positional information of landmarks detected around the target with the actual positional information of the target. The endoscopic image includes a first endoscopic image based on a first illumination light and a second endoscopic image based on a second illumination light with a different spectrum from the first illumination light. It is preferable that the processor detects the target and landmarks from the second endoscopic image and displays the actual positional information of the target and the positional information of the landmarks on the first endoscopic image.
[0011] The landmarks preferably include blood vessels that are highlighted by drug fluorescence. The targets for detection are preferably at least one of the blood vessels, bleeding sites, lesions, and specific organs that are highlighted by drug fluorescence.
[0012] The processor is Based on landmark location information When a new frame of endoscopic image is acquired while the location information estimation process is ongoing, it is preferable to detect a new landmark at a different location from already detected landmarks, perform a new landmark setting process to associate the estimated location information of the detected target with the new landmark, and if, after the new landmark setting process, another new frame of endoscopic image is acquired and the landmark necessary for the location information estimation process is not recognized, perform the location information estimation process using the new landmark setting process to calculate the estimated location information of the detected target and display the estimated location information of the detected target on the display. The new landmark is the location information of at least one of the mucosal pattern, organ shape, or marking by user operation, and it is preferable to display the location information of the new landmark on the display in a manner different from that of the aforementioned landmark. If the target is detected, It is preferable to perform a display process upon detection, which displays the actual location information of the detected target and the location information of landmarks on the display.
[0013] The endoscope system of the present invention includes a processor, which acquires an endoscope image, performs a second detection process on the endoscope image to acquire location information of landmarks, updates the endoscope image, and each time it acquires estimated location information of a detection target obtained from a location information estimation process based on the location information of landmarks, it performs a landmark setting process to associate the estimated location information of the detection target with the location information of landmarks and set a relative relationship, and displays the estimated location information of the detection target on a display.
[0014] When the position information estimation process is ongoing and a new frame of endoscopic image is acquired and a new landmark is detected, it is preferable to perform a new landmark setting process as a landmark setting process, which associates the estimated position information of the detected target with the new landmark to set a new relative relationship, and if no landmark necessary for the position information estimation process is recognized after the new landmark setting process, it is preferable to perform a position information estimation process based on the new relative relationship to calculate the estimated position information of the new detected target and display the estimated position information of the new detected target on the display. It is preferable that the new landmark is the position information of at least one of the following: mucosal pattern, organ shape, or marking by user operation.
[0015] The operation method of the endoscope system of the present invention includes the steps of: the processor acquiring an endoscope image; detecting a target to be detected and acquiring the actual position information of the target to be detected by performing a first detection process on the endoscope image; if a target to be detected, detecting a landmark and acquiring the location information of the landmark by performing a second detection process on the endoscope image, and performing a landmark setting process to associate the location information of the landmark with the actual position information of the target to be detected; and, if no target to be detected and a landmark is detected, performing a target undetected display process to display the location information of the landmark on the display. Furthermore, in the display process when no target is detected, the display of the landmark's location information on the display continues until the landmark is no longer detected. .
[0016] When the processor fails to detect the detection target and detects a landmark, it has a step of calculating the estimated position information of the detection target by position information estimation processing based on the position information of the landmark. It is preferable that the display processing when the target is not detected displays the estimated position information of the detection target and the position information of the landmark on the display.
Advantages of the Invention
[0017] According to the present invention, even if the visibility of the detection target decreases, the position of the detection target can be specified.
Brief Description of the Drawings
[0018] [Figure 1] It is a schematic diagram of an endoscope system. [Figure 2] It is a graph showing the spectra of purple light V, blue light B, green light G, and red light R. [Figure 3] (A) It is an explanatory diagram showing a single emission mode, and (B) it is an explanatory diagram showing a multi-emission mode. [Figure 4] It is a block diagram showing the functions of an extended processor device. [Figure 5] It is an image diagram showing a bleeding site detected by the first detection process and a circle for displaying the detection position. [Figure 6] It is an image diagram showing a circle for displaying the detection position, a landmark detected by the second detection process, and a circle for displaying the landmark position. [Figure 7] It is an image diagram showing the display mode of a circle for displaying the landmark position of a landmark with low reliability regarding detection. [Figure 8] It is an explanatory diagram showing the display processing when the target is detected and the display processing when the target is not detected. [Figure 9] It is an image diagram showing a circle for displaying the estimated position. [Figure 10] It is an image diagram showing a circle for displaying the estimated position on a sub-screen of the display. [Figure 11]This image shows a circle for displaying the detected location and a circle for displaying the landmark location on a sub-screen of the display. [Figure 12] This is an explanatory diagram showing how to rotate the image on the sub-screen by 180 degrees. [Figure 13] This is an explanatory diagram showing how to detect the target object and landmarks from the second endoscopic image, and how to display the actual location information of the detected object and the location information of the landmarks from the first endoscopic image. [Figure 14] This is an explanatory diagram showing the sequence of events in tracking mode. [Figure 15] This is an explanatory diagram for updating landmarks used in location information estimation processing. [Modes for carrying out the invention]
[0019] As shown in Figure 1, the endoscope system 10 includes an endoscope 12, a light source device 13, a processor device 14, a display 15, a user interface 16, an expansion processor device 17, and a display 18. The endoscope 12 is optically connected to the light source device 13 and electrically connected to the processor device 14. The light source device 13 supplies illumination light to the endoscope 12.
[0020] The endoscope 12 is used to illuminate the object to be observed with illumination light and to image the object to acquire an endoscopic image. The endoscope 12 has an insertion section 12a that is inserted into the body of the object to be observed, an operating section 12b provided at the base end of the insertion section 12a, and a bending section 12c and a tip section 12d provided at the tip end of the insertion section 12a. The bending section 12c bends when the operating section 12b is operated. The tip section 12d irradiates illumination light toward the object to be observed and receives reflected light from the object to image the object. The tip section 12d is directed in a desired direction by the bending movement of the bending section 12c. The operating section 12b is provided with a mode switching switch 12f used for switching modes, a still image acquisition instruction switch 12g used for instructing the acquisition of a still image of the object to be observed, and a zoom operation section 12h used for operating the zoom lens 21b.
[0021] The processor unit 14 is electrically connected to the display 15 and the user interface 16. The processor unit 14 receives endoscopic images from the endoscope 12. The display 15 outputs and displays images or information of the observed object processed by the processor unit 14. The user interface 16 has a keyboard, mouse, touchpad, microphone, etc., and has the function of accepting input operations such as function settings. The expansion processor unit 17 is electrically connected to the processor unit 14. The expansion processor unit 17 receives images or various information from the processor unit 14. The display 18 outputs and displays images or information processed by the expansion processor unit 17.
[0022] The endoscope system 10 is equipped with a mono-illumination mode, a multi-illumination mode, and a tracking mode, which can be switched using a mode selector switch 12f. The mono-illumination mode is a mode in which illumination light of the same spectrum is continuously used to illuminate the object of observation. The multi-illumination mode is a mode in which multiple illumination lights of different spectra are switched according to a specific pattern to illuminate the object of observation. The illumination light includes normal light (broadband light such as white light) used to provide brightness to the entire object of observation for observation, or special light used to highlight a specific area of the object of observation. In addition, in mono-illumination mode, the mode selector switch 12f may be used to switch between illumination lights of different spectra. For example, the system may be used to switch between a first illumination light and a second illumination light with different spectra.
[0023] The tracking mode is a mode that detects the actual location information of the detected object, such as a bleeding area, and displays the location information of the detected object and the location information of landmarks associated with the detected object's location information on display 18 (or display 15), so that the user can understand the location of the detected object even if there is a change in the image of the subject.
[0024] As shown in Figure 2, the illumination light is preferably emitted by combining violet light V, blue light B, green light G, and red light R. The violet light V is preferably centered at 405±10 nm and has a wavelength range of 380-420 nm. The blue light B is preferably centered at 450±10 nm and has a wavelength range of 420-500 nm. The green light G is preferably in a wavelength range of 480-600 nm. The red light R is preferably centered at 620-630 nm and has a wavelength range of 600-650 nm.
[0025] The light source device 13 independently controls the light intensity of four colors: violet light V, blue light B, green light G, and red light R. As shown in Figure 3(A), in mono-emitting mode, illumination light L of the same spectrum is emitted continuously for each frame. On the other hand, in multi-emitting mode, the light intensity of the four colors—violet light V, blue light B, green light G, and red light R—is controlled according to a specific pattern. For example, as shown in Figure 3(B), one specific pattern is to alternate between a first emission pattern in which a first illumination light L1 having a first spectrum is emitted for two consecutive frames, and a second emission pattern in which a second illumination light L2 having a second spectrum different from the first spectrum is emitted for one frame. A frame refers to the time from when the imaging sensor (not shown) located at the tip 12d of the endoscope starts receiving reflected light from the object being observed until the output of the charge signal accumulated based on the received light is completed.
[0026] As shown in Figure 4, the expansion processor device 17 includes an image acquisition unit 20, a detection target detection unit 21, a landmark processing unit 22, a display control unit 23, an estimated position information calculation unit 24, and a detection memory 26. The expansion processor device 17 is provided with a program memory for storing programs related to various processes. By executing the above programs using the processor provided in the expansion processor device 17, the functions of the image acquisition unit 20, the detection target detection unit 21, the landmark processing unit 22, the estimated position information calculation unit 24, and the display control unit 23 are realized. The display control unit 23 may control the display of the display 15 in addition to controlling the display of the display 18.
[0027] The image acquisition unit 20 acquires endoscopic images transmitted from the processor unit 14. The processor unit 14 transmits endoscopic images to the extended processor unit 17 one frame at a time. The image acquisition unit acquires the endoscopic images one frame at a time transmitted from the processor unit 14.
[0028] The detection target detection unit 21 detects a target and acquires its actual position information by performing a first detection process on the endoscopic image. As shown in Figure 5, if the first detection process detects a bleeding site BS, which is one of the detection targets, in the endoscopic image on the display 18, the display control unit 23 displays a detection position display circle 30 around the bleeding site BS on the display 18 as the actual position information of the bleeding site BS. The detection target is preferably at least one of the following: blood vessels highlighted by drug fluorescence, bleeding sites BS, lesions such as cancer, and specific organs.
[0029] Furthermore, it is preferable that the detection target detection unit 21 is a trained model that has been machine-trained using training image data that includes the detection target. Machine learning includes supervised learning, semi-unsupervised learning, unsupervised learning, reinforcement learning, deep reinforcement learning, neural network learning, and deep learning. Also, if a detection is performed by the first detection process, the information of the detected detection target is stored in the detection memory 26.
[0030] The landmark processing unit 22 detects landmarks and acquires landmark position information by performing a second detection process on the endoscopic image when the detection target detection unit 21 detects a target. Landmarks include blood vessels, such as those highlighted by photodynamic diagnosis (PDD), as well as various structures such as glandular structures. As shown in Figure 6, in an endoscopic image in which a bleeding site BS, one of the detection targets, is detected, if multiple landmarks LM are detected by the second detection process, the display control unit 23 displays multiple landmark position display circles 32 on the display 18 as location information for the landmarks LM. In this case, it is preferable to make it possible to distinguish between each other's landmark position display circles 32. For example, each landmark position display circle 32 is assigned a distinguishing number NB (distinguishing number). It is preferable to detect landmarks LM not only near the detection target such as the bleeding area, but also from a position away from the detection target in order to eliminate factors that reduce the visibility of the detection target, such as pooled blood flowing out of the bleeding site.
[0031] Furthermore, if a landmark is detected by the second detection process, a landmark setting process is performed to associate the location information of the landmark with the actual location information of the detected target. As shown in Figure 6, in the landmark setting process, the detection location display circle 30 and the landmark location display circle 32 are connected by a link line 34 as a method for associating the location information of the landmark LM with the actual location information of the detected target. In this case, it is preferable to associate the location information of the landmark LM detected around the detected target with the actual location information of the detected target. That is, in the case of Figure 6, the landmark location display circles 32 with distinguishing numbers "1", "2", "3", "4", and "5" around the detection location display circle 30 must be connected to the detection location display circle 30 by a link line 34. It is also preferable to connect different landmark location display circles 32 with link lines 34. The information regarding the location information of the landmark LM and the actual location information of the detected target associated by the landmark setting process is stored in the detection memory 26.
[0032] Furthermore, of the landmark processing unit 22, the second detection The processing unit for performing the processing is preferably a trained model for landmark detection that has been machine-trained on training image data including landmarks. Furthermore, if the landmark processing unit 22 can calculate the confidence level for landmark detection, it is preferable to change the display manner of the landmark's location information (such as color or line style) according to the confidence level. For example, as shown in Figure 7, if the confidence level of the landmark LM with distinguishing number "1" is lower than that of the other landmark LMs, it is preferable to make the display manner of the landmark location display circle 32 for the landmark LM with distinguishing number "1" (dotted line in Figure 7) different from that of the landmark location display circle 32 for the other landmark LMs (solid line in Figure 7).
[0033] The display control unit 23 performs either a target detection display process, which displays the actual location information of the detected target and the location information of the landmark on the display 18 when a target is detected, or a target non-detection display process, which displays the location information of the landmark on the display 18 when no target is detected but a landmark is detected. As shown in Figure 8, in the endoscope image on the display 18, the endoscope image on the display 18 displays a detection position display circle 30 as the actual location information of the detected target, and a landmark position display circle 32 as the location information of the landmark LM. In the target detection display process, it is preferable to also display the link line 34 on the display 18. Note that if the landmark setting process has been performed, the display control unit 23 may perform only the target non-detection display process and not the target detection display process.
[0034] Furthermore, if the detection target disappears from the endoscopic image due to a large amount of blood (blood pool BP) flowing out from the bleeding site, the detection target detection unit 21 will no longer detect the target. Even if the detection target disappears in this way, if a landmark remains in the endoscopic image, the landmark processing unit 22 will maintain detection of the landmark. In this case, the target undetected display processing is performed, and although the detection position display circle 30 is not displayed, the landmark position display circle 32 is displayed on the display 18 as landmark position information. In the target undetected display processing, it is preferable to also display the link line 34 on the display 18.
[0035] The estimated location information calculation unit 24 calculates the estimated location information of the detected object by performing a location information estimation process based on the location information of the landmark when the detected object is not detected. As shown in Figure 9, if the display of the detected object disappears, the display control unit 23 maintains the display of the landmark location display circle 32 on the display 18 by performing a display process for when the object is not detected. In this case, by performing a location information estimation process, the estimated location display circle 36 is displayed as the estimated location information of the detected object in the area where the detected object is estimated to be located. It is preferable that the location information estimation process is performed based on the positional relationship between the landmark location display circles 32, for example, the shape of the link formed from the link line 34. It is preferable that the estimated location display circle 36 has a different display pattern from the detection location display circle 30 in order to make it easier to distinguish from the detection location display circle 30. In Figure 9, the periphery of the estimated location display circle 36 is doubled (the periphery of the detection location display circle 30 is single (see Figure 5)).
[0036] The estimated position information may also be displayed in a display mode other than that shown in Figure 9. For example, as shown in Figure 10, if the display 18 has a main screen 18a for displaying endoscopic images and a sub-screen 18b located at a different position from the main screen 18a, the estimated position information, represented by the estimated position display circle 36, is displayed on the sub-screen 18b. In this case, in addition to the estimated position display circle 36, the sub-screen 18b also displays a landmark position display circle 32. It is also preferable to display the landmark position display circle 32 on the main screen 18a.
[0037] Furthermore, as shown in Figure 11, it is preferable that the main screen 18a displays the estimated position circle 36 and the landmark position circle 32 superimposed on the real-time endoscopic image, and the sub-screen 18b displays the detection position circle 30, which is the actual position information of the detected object, and the landmark position circle 32, which is the position information of the landmark LM, on a still image of the endoscopic image acquired at the time the object was detected. By displaying the position information (detected object and landmark) at the time the object was detected on the sub-screen 18b in this way, the user can confirm the accuracy of the estimated position information displayed on the real-time endoscopic image. In addition, even if the estimated position information can no longer be displayed on the main screen 18a, the user can visually estimate the object by displaying the position information (detected object and landmark) at the time the object was detected on the sub-screen 18b. Note that in Figure 11, the link line 34 may also be displayed (the same applies to Figure 12).
[0038] Furthermore, when displaying a still image of the endoscope on the sub-screen 18b, depending on the position of the tip 12d of the endoscope, the vertical and horizontal position of the real-time endoscope image on the main screen 18a may differ from the vertical and horizontal position of the still image of the endoscope on the sub-screen 18b. For example, as shown in Figure 12(A), the real-time endoscope image on the main screen 18a and the still image of the endoscope on the sub-screen 18b may be inverted vertically. In this case, a rotation state detection unit (not shown) provided in the expansion processor device 17 detects the rotation state from the real-time endoscope image on the main screen 18a. Then, as shown in Figure 12(B), an image rotation unit (not shown) provided in the expansion processor device 17 rotates the still image of the endoscope on the sub-screen 18b so that its vertical direction matches that of the real-time endoscope image on the main screen 18a, based on the detected rotation state.
[0039] In tracking mode, as shown in Figure 13, it is preferable to detect the target and landmarks from the second endoscopic image based on the second illumination light, perform landmark setting processing, then switch from the second illumination light to the first illumination light, and display the actual position information of the target and the position information of the landmarks on the first endoscopic image based on the first illumination light. This prevents the loss of the position or area of the target when switching from the second illumination light to the first illumination light. The second illumination light is preferably a special light suitable for detecting the target and landmarks, for example, a purple light that can highlight structures. On the other hand, the first illumination light is preferably a light suitable for displaying the actual position information of the target and the position information of the landmarks, for example, a white light.
[0040] Next, the sequence of steps in tracking mode will be explained according to the flowchart in Figure 14. The tracking mode is turned ON by operating the mode switching switch 12f. This causes the first detection process to be performed on the endoscopic image. In the first detection process, if a target containing a bleeding site BS is detected, the actual location information of the detected target is acquired. If the detected target is a new target, a landmark setting process is performed. In the landmark setting process, a landmark is detected and its location information is acquired by performing a second detection process on the endoscopic image. In addition, the location information of the landmark is associated with the actual location information of the detected target. The location information of the landmark and the actual location information of the detected target, which are associated with each other, are displayed on the display 18. On the other hand, if the detected target is not a new target, the location information of the landmark already associated in the landmark setting process and the actual location information of the detected target are displayed on the display 18.
[0041] The determination of whether or not a target is new is made by checking whether or not information about the target exists in the detection memory 26. If a new target is detected, the information about the target already stored in the detection memory 26 is erased, and the information about the new target is newly stored in the detection memory 26. In this case, it is preferable to also erase the information about the landmark associated with the target to be erased.
[0042] In response to this, if the first detection process fails to detect the target, it is determined whether the landmark setting process has already been performed (determination of whether the landmark setting process is in progress). If the landmark setting process has already been performed, the estimated location information of the target is calculated based on the location information of the landmark. The calculated estimated location information of the target and the location information of the landmark are then displayed on the display 18. This series of processes is repeated as long as the tracking mode is ON. When the mode switching switch 12f is operated and the tracking mode is turned OFF, the detection of the target ends.
[0043] Furthermore, if there is insufficient location information for landmarks, or if it is not possible to calculate the estimated location information of the detection target based on the location information of landmarks, it is preferable to erase the information regarding the detection target and the landmarks stored in the detection memory 26 and perform the detection of the detection target or the detection of landmarks again. Also, the display method of the landmark location information (landmark location display circle 32) when a detection target is detected, and the detection target detection It is preferable to display the location information of the landmark in a different manner than when it is not possible to do so. For example, it is preferable to display it in a different manner, such as by using a different color.
[0044] Since the endoscope 12 is operated manually, even if the estimated position of the target to be detected is continuously captured within the endoscopic image, the range of the endoscopic image changes, and the landmark LM surrounding the estimated position of the target to be detected may no longer fit within the endoscopic image. In other cases, the organ may deform, changing the relationship between the landmark and the target to be detected. As shown in Figure 15, when the endoscope 12 takes the next frame, it may detect a new landmark LM2 at a different position from the landmark LM used for position information estimation processing, update the landmark LM used for position information estimation processing, and continue displaying the estimated position indicator 36 on the bleeding area BP.
[0045] When updating the landmark LM used for location estimation processing, the original landmark LM is designated as landmark LM, and the updated landmark LM2 as the new landmark LM. Figure 15(A) shows the location estimation processing using landmark LM for displaying the estimated location. circle Display 36. When a new frame is acquired, as shown in Figure 15(B), a new landmark LM2 is detected that surrounds the estimated position of the detected target in accordance with the movement direction of the preceding and succeeding frame captures, and a new landmark position indicator 38 is displayed. In addition, a new landmark setting process is performed to associate the new landmark LM2 with the estimated position information of the detected target, and a new relative relationship is calculated. The new relative relationship is displayed by a new link line 39. It is preferable that the new link line 39 be less conspicuous than the link line 34 and not confused with it, such as by using a dotted line. Note that a distinguishing number NB (distinguishing number) for each landmark position indicator 32 can also be assigned to the new landmark position indicator 38, but it is not necessary to assign it if it reduces visibility. Also, the new landmark position indicator 38 is for landmark position display circle The shape may be the same as or different from that of 32.
[0046] After the new landmark setting process, if the endoscope 12 acquires an endoscopic image of a new frame and the landmark LM necessary for the position information estimation process is not recognized, a new position information estimation process based on the new relative relationship is performed as shown in Figure 15(C), and the estimated position information of the detected target is calculated and used for the estimated position display. circle 36 is displayed on display 18. Since the location information estimation process by landmark LM is finished, link line 34 is hidden, and the new link line 39 is displayed as a solid line like link line 34. For landmark LMs that continue to be detected, immediately after updating the location information estimation process, the landmark location display is used. circle You may display "32," but it is preferable to hide it after a certain period of time has elapsed.
[0047] As the range captured by the endoscope 12 shifts even after the position information estimation process using the new landmark LM2, the updating of the landmark LM used for position information estimation continues. The new landmark LM2 is treated as the landmark LM, and the new link line 39 as the link line 34. The new landmark LM2 updates its relative relationship through the new landmark setting process, and the landmark LM performs the position information estimation process.
[0048] In the above embodiment, the hardware structure of the processing unit that performs various processes such as the image acquisition unit 20, the detection target detection unit 21, the landmark processing unit 22, the display control unit 23, the estimated position information calculation unit 24, or the detection memory 26 is a various type of processor as shown below. These various processors include a CPU (Central Processing Unit), a general-purpose processor that executes software (programs) and functions as various processing units; a GPU (Graphical Processing Unit); a Programmable Logic Device (PLD), a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array); and a dedicated electrical circuit, a processor with a circuit configuration specifically designed to perform various processes.
[0049] A single processing unit may be composed of one of these various processors, or it may be composed of a combination of two or more processors of the same or different types (for example, multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software are combined to form a single processor, and this processor functions as multiple processing units, as is typical of computers such as clients and servers. Secondly, a configuration where a processor is used that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of a System on a Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors.
[0050] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit formed by combining circuit elements such as semiconductor devices. The hardware structure of the memory unit is a storage device such as an HDD (hard disk drive) or SSD (solid state drive). [Explanation of Symbols]
[0051] 10 Endoscopy Systems 12 Endoscopes 12a Insertion section 12b Operation section 12c curved section 12d Tip 12f Mode Selector Switch 12g still image acquisition instruction switch 12h Zoom control section 13 Light source device 14 Processor Unit 15 displays 16 User Interface 17. Expansion Processor Unit 18 displays 18a Main screen 18b Sub-screen 20 Image acquisition unit 21 Detection target detection unit 22 Landmark Processing 23 Display Control Unit 24 Estimated position information calculation unit 26 Detection memory 30 Circles for indicating detection position 32. Circles for indicating landmark locations 34 Linkline 36 Circle for displaying estimated position 38. New landmark location indicator 39 New Linkline BP pool of blood L illumination light L1 1st illumination light L2 2nd illumination light BS bleeding site LM Landmark LM2 New Landmark
Claims
1. Equipped with a processor, The aforementioned processor Obtain endoscopic images, By performing a first detection process on the endoscopic image, the target to be detected is detected and the actual position information of the target to be detected is obtained. When the target is detected, a second detection process is performed on the endoscope image to detect a landmark and obtain the location information of the landmark, and a landmark setting process is performed to associate the location information of the landmark with the actual location information of the detected target. If the target is not detected and the landmark is detected, a target undetected display process is performed to display the location information of the landmark on the display. In the aforementioned target undetected display process, the display of the landmark's location information on the display continues until the landmark is no longer detected. If the detection target is not detected and the landmark is detected, the processor calculates the estimated location information of the detection target by performing a location information estimation process based on the location information of the landmark. The aforementioned target undetected display process is an endoscope system that displays the estimated location information of the detected target and the location information of the landmark on the display.
2. If the detection target is not detected and the landmark is detected, the processor calculates the estimated location information of the detection target by performing a location information estimation process based on the location information of the landmark. The display has a main screen for displaying the endoscopic image, and a sub-screen located at a different position from the main screen. The endoscope system according to claim 1, wherein in the process of displaying when no target is detected, the estimated location information of the detected target and the location information of the landmark are displayed on the sub-screen.
3. The processor calculates the estimated location information of the detected target by performing a location information estimation process based on the location information of the landmark. The display has a main screen for displaying the endoscopic image, and a sub-screen located at a different position from the main screen. The endoscope system according to claim 1, wherein in the process of displaying when no target is detected, the main screen displays the estimated location information of the detected target and the location information of the landmark, and the sub-screen displays a still image of the endoscope acquired at the time the target was detected, along with the location information of the detected target and the location information of the landmark.
4. The endoscope system according to any one of claims 1 to 3, wherein the landmark setting process associates the location information of the landmarks detected around the target with the actual location information of the target.
5. The endoscopic images include a first endoscopic image based on a first illumination light and a second endoscopic image based on a second illumination light having a different spectrum from the first illumination light. The endoscopic system according to any one of claims 1 to 4, wherein the processor detects the target to be detected and the landmark from the second endoscopic image, and displays the actual position information of the target to be detected and the position information of the landmark on the first endoscopic image.
6. The endoscopic system according to any one of claims 1 to 5, wherein the landmark includes a blood vessel that is highlighted by drug fluorescence.
7. The endoscopic system according to any one of claims 1 to 6, wherein the detection target is at least one of blood vessels, bleeding sites, lesions, and specific organs that are highlighted by drug fluorescence.
8. The aforementioned processor, If a new frame of the endoscopic image is acquired while the location information estimation process based on the location information of the aforementioned landmark is continuing, a new landmark at a different location from the previously detected landmark will be detected. A new landmark setting process is performed to associate the estimated location information of the detected target with the new landmark. If, after the aforementioned new landmark setting process, a new frame of the endoscopic image is acquired and the landmark necessary for the location information estimation process is not recognized, the location information estimation process is performed using the aforementioned new landmark setting process to calculate the estimated location information of the detected target. The endoscope system according to any one of claims 1 to 7, wherein the estimated position information of the detected target is displayed on the display.
9. The aforementioned novel landmark is location information of at least one of the following: mucosal pattern, organ shape, or user-operated marking. The endoscope system according to claim 8, which displays the location information of the new landmark on the display in a manner different from that of the landmark.
10. The endoscope system according to any one of claims 1 to 9, wherein when the detection target is detected, a target detection display process is performed to display the actual location information of the detection target and the location information of the landmark on the display.
11. Equipped with a processor, Each time the endoscopic image is updated and the estimated position information of the detected target obtained from the position information estimation process is acquired, a landmark setting process is performed to associate the estimated position information of the detected target with the position information of the landmark and set a relative relationship. The endoscope system according to any one of claims 1 to 10, which displays the estimated position information of the detected target on the display.
12. If, while the position information estimation process is continuing, a new frame of the endoscopic image is acquired and a new landmark is detected, the landmark setting process is performed to associate the estimated position information of the detected target with the new landmark and set a new relative relationship. If, after the aforementioned new landmark setting process, no landmark necessary for the location information estimation process is recognized, the location information estimation process based on the new relative relationship is performed to calculate the estimated location information of the new detection target. The endoscope system according to claim 11, which displays estimated position information of the new detection target on the display.
13. The endoscopic system according to claim 12, wherein the novel landmark is location information of at least one of the mucosal pattern, organ shape, or user-operated marking.
14. The processor, Steps to acquire endoscopic images, The steps include: performing a first detection process on the endoscopic image to detect a target and obtain the actual position information of the target; When the target to be detected is detected, a second detection process is performed on the endoscope image to detect a landmark and obtain the location information of the landmark, and a landmark setting process is performed to associate the location information of the landmark with the actual location information of the target to be detected. The process includes the step of performing a display process for when the target is not detected, and when the landmark is detected, displaying the location information of the landmark on the display. In the aforementioned target undetected display process, the display of the landmark's location information on the display continues until the landmark is no longer detected. The processor has a step of calculating the estimated location information of the detected target by a location information estimation process based on the location information of the landmark, when the detected target is not detected and the landmark is detected. The aforementioned target undetected display process is a method for operating an endoscope system that displays the estimated location information of the detected target and the location information of the landmark on the display.