Processor device and method of operation thereof

Abstract

The present application provides a processor device capable of appropriately displaying lesion information according to observation conditions and a working method thereof. An observation condition acquisition section (70) acquires an observation condition including at least one of a moving speed of an endoscope (12), an observation distance of the endoscope (12) from an observation object, or a brightness of the observation object. A lesion information acquisition section (72) acquires lesion information including at least one of a reliability of a lesion obtained from an endoscope image or a diagnosis purpose at a time point when the observation condition is acquired. A display form decision section (74) decides a display form of the lesion information in a display (18) according to at least either one of the observation condition or the lesion information. Control is performed to display the lesion information in the display (18) in accordance with the display form.

Description

Technical Field

[0001] This invention relates to a processor device and its operating method, wherein the processor device controls the display of lesion information, such as the confidence level of the lesion, on a display screen. Background Technology

[0002] In the medical field, medical images are widely used for diagnosis. For example, endoscopic systems, which utilize medical images, include a light source, an endoscope, and a processor. In an endoscopic system, an endoscopic image is acquired by illuminating the object being observed and photographing the illuminated object. This endoscopic image is then displayed on a monitor and used for diagnosis.

[0003] Furthermore, in recent endoscopic systems, areas of interest, such as lesions, are detected from endoscopic images and highlighted to assist operators in diagnosis. For example, Patent Document 1 describes a system that, upon detecting an area of ​​interest, determines whether a warning image should be displayed based on factors such as the size of the area, and displays or does not display the warning image based on the determination result. Therefore, frequent detection of areas of interest would be inconvenient for operators; thus, a warning image is only displayed when the lesion is large or when notification to the operator is required.

[0004] Previous technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2011-255006 Summary of the Invention

[0007] The technical problem to be solved by the invention

[0008] As described in Patent Document 1, when determining the detection or extent of a lesion based on endoscopic images, non-lesions may sometimes be mistakenly detected as lesions or lesions may be missed. This depends on the observation conditions when acquiring the endoscopic image, such as the distance from the object being observed, the shooting angle, or the brightness of the object. If there are too many false detections, they can cause flickering when the detection results are overlaid on the observation image, potentially hindering the operator's diagnosis. On the other hand, if lesion detection or other aids are not used to avoid false detections, lesions may be missed. Therefore, it is necessary to display lesion information appropriately according to the observation conditions to suppress flickering caused by false detections without missing lesions.

[0009] The purpose of this invention is to provide a processor device and its working method that can appropriately display lesion information according to observation conditions.

[0010] means for solving technical problems

[0011] In the processor device of the present invention, the image processing processor performs the following processing: acquiring observation conditions including at least one of the following: the movement speed of the endoscope, the observation distance between the endoscope and the object being observed, or the brightness of the object being observed; at the time point of acquiring the observation conditions, acquiring lesion information including at least one of the following: the confidence level of the lesion obtained from the endoscopic image or the diagnostic purpose; determining the display format of the lesion information on the display based on at least one of the observation conditions or the lesion information; and controlling the display of the lesion information on the display according to the display format.

[0012] The image processing processor preferably determines that the display format when the moving speed is a first moving speed is different from the display format when the moving speed is a second moving speed, which is slower than the first moving speed. The image processing processor preferably determines a non-display display format that does not display lesion information in at least one of the cases where the moving speed is the first moving speed or the brightness is lower than a brightness threshold.

[0013] The image processing processor preferably determines the display format for displaying lesion information when the movement speed is a second movement speed and the brightness is above a brightness threshold. The image processing processor preferably determines different display formats based on confidence level when the observation distance is a first observation distance, and determines different display formats based on diagnostic purposes when the observation distance is a second observation distance shorter than the first observation distance.

[0014] The image processing processor preferably determines the display format for displaying lesion information on the display frame by frame when the observation distance is the first observation distance and the confidence level is above the confidence threshold. When the observation distance is the first observation distance and the confidence level is below the confidence threshold, it determines a first display format. In this first display format, it determines multiple specific frames before and after the frame with a confidence level below the confidence threshold and processes and displays the lesion information based on a first operation based on the lesion information of the multiple specific frames. Preferably, in the first display format, the lesion information is displayed on the display when the number of frames with high confidence among the multiple specific frames is a specific number or more.

[0015] The image processing processor preferably determines a second display format as a display format when the observation distance is a second observation distance and the diagnostic purpose is lesion range diagnosis. In this second display format, lesion information related to lesion range diagnosis is displayed based on a second operation processing of lesion information based on multiple range diagnosis frames. When the observation distance is a second observation distance and the diagnostic purpose is differential diagnosis, the processor determines a third display format as a display format. In this third display format, lesion information related to differential diagnosis is displayed based on a third operation processing of lesion information based on multiple differential diagnosis frames.

[0016] In the second display format, it is preferable to determine the lesion range based on lesion information from multiple range diagnostic frames, and display the lesion information using the lesion range. In the third display format, it is preferable to determine the differential diagnosis content based on lesion information from multiple differential diagnosis frames, and display the lesion information using the differential diagnosis content. Preferably, the display image used to display the lesion information is obtained based on the emission of the first illumination light, and the lesion information acquisition image used to acquire the lesion information is obtained based on the emission of a second illumination light with a different emission spectrum from the first illumination light.

[0017] In the method of operating the processor device of the present invention, the image processing processor performs the following processing: acquiring observation conditions including at least one of the following: the moving speed of the endoscope, the observation distance between the endoscope and the object being observed, or the brightness of the object being observed; at the time point of acquiring the observation conditions, acquiring lesion information including at least one of the following: the confidence level of the lesion obtained from the endoscope image or the diagnostic purpose; determining the display format of the lesion information on the display based on at least one of the observation conditions or the lesion information; and controlling the display of the lesion information on the display according to the display format.

[0018] Invention Effects

[0019] According to the present invention, lesion information can be displayed appropriately according to observation conditions. Attached Figure Description

[0020] Figure 1 This is an external view of the endoscope system.

[0021] Figure 2 It is a block diagram representing the function of an endoscope system.

[0022] Figure 3 It is a graph showing the spectral transmittance of each color filter in a camera sensor.

[0023] Figure 4 This is a block diagram representing the functions of the lesion information processing unit.

[0024] Figure 5 It represents an image in a non-display display format.

[0025] Figure 6 It represents an image or diagram used for display.

[0026] Figure 7 This is an explanatory diagram showing how lesion information is acquired and displayed frame by frame.

[0027] Figure 8 This is an explanatory diagram showing the display format used for the first display.

[0028] Figure 9This is an explanatory diagram showing the display format used for the second display.

[0029] Figure 10 This is an explanatory diagram illustrating the resetting of the lesion range through the second operation.

[0030] Figure 11 This is an explanatory diagram showing the display format used for the third display.

[0031] Figure 12 It is an image that displays lesion information using identification content (DIJ).

[0032] Figure 13 It is a flowchart representing a series of processes that indicate the display mode of lesion information.

[0033] Figure 14 This is an explanatory diagram showing the first A emission pattern or the second A pattern in the analysis and processing mode.

[0034] Figure 15 This is an explanatory diagram showing the first emission pattern in the analysis and processing mode.

[0035] Figure 16 This is an explanatory diagram of pattern 2B representing the analysis and processing mode.

[0036] Figure 17 This is an explanatory diagram of pattern 2C when representing the analysis and processing mode.

[0037] Figure 18 This is an explanatory diagram of the 2D pattern representing the analysis and processing mode. Detailed Implementation

[0038] Figure 1 The endoscope system 10 includes an endoscope 12, a light source device 14, a processor device 16, a display 18, and an operator interface 19. The endoscope 12 is optically connected to the light source device 14 and electrically connected to the processor device 16. The endoscope 12 has an insertion part 12a that is inserted into the body of the object being observed, an operation part 12b provided at the base of the insertion part 12a, a bending part 12c provided at the front end of the insertion part 12a, and a front end part 12d. By operating the bending knob 12e of the operation part 12b, the bending part 12c is bent. The front end part 12d is oriented in the desired direction by the bending action of the bending part 12c.

[0039] Furthermore, in addition to the bend button 12e, the operation unit 12b also includes a mode switch SW (mode switching switch) 12f for mode switching operations, a still image acquisition command unit 12g for commanding the acquisition of still images of the observed object, and a zoom lens 43 (reference) for operation. Figure 2 The zoom operation unit is 12h.

[0040] Furthermore, the endoscope system 10 has three modes: a conventional observation mode, a special observation mode, and a lesion information display mode. In the conventional observation mode, a conventional observation image with a natural tone is displayed on the monitor 18 by illuminating the observed object with conventional light such as white light. In the special observation mode, a special observation image emphasizing specific structures is displayed on the monitor 18 by illuminating the observed object with special light of a different wavelength band than conventional light. In the lesion information display mode, the display format of the lesion information on the monitor 18 is determined according to at least one of the observation conditions or lesion information, and the lesion information is displayed on the monitor 18 according to the determined display format. In addition, in the lesion information display mode, in addition to continuously emitting either conventional light or special light, it can automatically switch between a first illumination light and a second illumination light with different emission spectra to emit light onto a specific emission pattern.

[0041] The operator operates the still image acquisition command unit 12g, which sends signals related to the still image acquisition command to the endoscope 12, the light source device 14, and the processor device 16. If the still image acquisition command is executed, a still image of the observed object is stored in the still image storage memory 69 of the processor device 16 (see reference). Figure 2 ).

[0042] The processor device 16 is electrically connected to the display 18 and the operator interface 19. The display 18 outputs images of the observed object or information accompanying the images of the observed object. The operator interface 19 has a keyboard, mouse, touchpad, etc., and has the function of accepting input operations such as function settings. In addition, an external recording unit (not shown) for recording images, image information, etc. can be connected to the processor device 16.

[0043] Figure 2 In this device, the light source apparatus 14 includes a light source unit 20 and a light source processor 21 for controlling the light source unit 20. The light source unit 20 emits illumination light for illuminating the object being observed. The light source processor 21 controls the amount of illumination light emitted from the light source unit 20. The illumination light from the light source unit 20 passes through a light path coupling section 23, which is composed of a mirror or lens, and is incident on a light guide 25. The light guide 25 is built into the endoscope 12 and a universal plug (a plug connecting the endoscope 12 to the light source apparatus 14 and the processor apparatus 16). The light guide 25 propagates the light from the light path coupling section 23 to the front end portion 12d of the endoscope 12.

[0044] An illumination optical system 30a and an imaging optical system 30b are provided in the anterior end portion 12d of the endoscope 12. The illumination optical system 30a has an illumination lens 32, through which illumination light propagating via the light guide 25 illuminates the object being observed. The imaging optical system 30b has an objective lens 42 and an image sensor 44. Light from the object being illuminated by the illumination light enters the image sensor 44 via the objective lens 42 and the zoom lens 43. Thus, an image of the object being observed is formed in the image sensor 44. The zoom lens 43 is a lens used to magnify the object being observed, and it moves between a telephoto end and a wide-angle end by operating the zoom operation unit 12h.

[0045] The camera sensor 44 is a primary color sensor, equipped with three types of pixels: B pixels (blue pixels) with a blue filter, G pixels (green pixels) with a green filter, and R pixels (red pixels) with a red filter. For example... Figure 3 As shown, the blue filter BF primarily transmits light in the blue wavelength range, specifically, light in the 380–560 nm wavelength band. The transmittance of the blue filter BF peaks near the 460–470 nm wavelength range. The green filter GF primarily transmits light in the green wavelength range, specifically, light in the 460–620 nm wavelength band. The red filter RF primarily transmits light in the red wavelength range, specifically, light in the 580–760 nm wavelength band.

[0046] Furthermore, the camera sensor 44 is preferably a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera processor 45 controls the camera sensor 44. Specifically, the camera processor 45 reads the signals from the camera sensor 44 and outputs image signals from the camera sensor 44.

[0047] like Figure 2 As shown, the CDS / AGC (Correlated Double Sampling / Automatic Gain Control) circuit 46 performs correlation double sampling (CDS) or automatic gain control (AGC) on the analog image signal obtained from the camera sensor 44. The image signal after passing through the CDS / AGC circuit 46 is converted into a digital image signal by the A / D (Analog / Digital) converter 48. The digital image signal after A / D conversion is input to the processor device 16.

[0048] The processor device 16 includes an image acquisition unit 50, a DSP (Digital Signal Processor) 52, a noise reduction unit 54, an image processing switching unit 56, an image processing unit 58, and a display control unit 60. The image processing unit 58 includes a conventional observation image generation unit 62, a special observation image generation unit 64, and a lesion information processing unit 66.

[0049] In the processor device 16, programs for performing various processing tasks, such as processing related to lesion information, are stored in a program memory (not shown). The central control unit 68, which is composed of an image processing processor, executes the programs in the program memory to realize the functions of the image acquisition unit 50, the DSP (Digital Signal Processor) 52, the noise reduction unit 54, the image processing switching unit 56, the image processing unit 58, and the display control unit 60. Simultaneously, the functions of the conventional observation image generation unit 62, the special observation image generation unit 64, and the lesion information processing unit 66 included in the image processing unit 58 are realized. Furthermore, the lesion information processing unit 66 realizes the functions of the observation condition acquisition unit 70, the lesion information acquisition unit 72, and the display format determination unit 74 (see reference). Figure 4 ).

[0050] The image acquisition unit 50 acquires the endoscopic image input from the endoscope 12. The endoscopic image is preferably a color image composed of blue signals (B image signal), green signals (G image signal), and red signals (R image signal) output from the B, G, and R pixels of the image sensor 44. The acquired color image is sent to the DSP 52. The DSP 52 performs various signal processing on the received color image, including defect correction processing, offset processing, gain correction processing, matrix processing, gamma conversion processing, demosaic processing, and YC conversion processing. In the defect correction processing, the signals of defective pixels of the image sensor 44 are corrected. In the offset processing, dark current components are removed from the image signal after defect correction processing, and a correct zero level is set. The gain correction processing adjusts the signal level of the color image by multiplying the image signals of each color after offset processing by a specific gain coefficient. Furthermore, when a monochrome sensor is used as the image sensor 44, the endoscopic image is preferably a monochrome image of multiple colors output from the monochrome sensor, capturing each emission of a specific color of light.

[0051] After gain correction, matrix processing is applied to each color image signal to improve color reproducibility. Then, gamma conversion is performed to adjust the brightness and chroma of the color image. Demosaic processing (isotropic processing, also known as synchronization processing) is then applied to the matrix-processed color image, using interpolation to generate signals for the colors missing from each pixel. Through demosaic processing, all pixels possess signals for each of the RGB colors. The DSP52 performs YC conversion on the demosaiced color image and outputs the brightness signal Y, color difference signal Cb, and color difference signal Cr to the noise reduction unit 54.

[0052] The noise reduction unit 54 performs noise reduction processing on the color image that has undergone de-mosaic processing using the DSP 52, for example, using a moving average method or a median filtering method. The noise-reduced color image is then input to the image processing switching unit 56.

[0053] The image processing switching unit 56 switches the destination of the image signal from the noise reduction unit 54 to any one of the following: the conventional observation image generation unit 62, the special observation image generation unit 64, and the lesion information processing unit 66, depending on the set mode. Specifically, when the conventional observation mode is set, the image signal from the noise reduction unit 54 is input to the conventional observation image generation unit 62. When the special observation mode is set, the image signal from the noise reduction unit 54 is input to the special observation image generation unit 64. When the lesion information display mode is set, the image signal from the noise reduction unit 54 is input to the lesion information processing unit 66.

[0054] The conventional observation image generation unit 62 performs conventional observation image processing on the input endoscopic image. This conventional observation image processing includes 3×3 matrix processing, grayscale conversion processing, three-dimensional LUT (Look Up Table) processing, color conversion processing, color emphasis processing, spatial frequency emphasis, and other structural emphasis processing. The endoscopic image that has undergone conventional observation image processing is input to the display control unit 60 as a conventional observation image.

[0055] The special observation image generation unit 64 performs special observation image processing on the input endoscopic image. This special observation image processing includes 3×3 matrix processing, grayscale conversion processing, three-dimensional LUT (Look Up Table) processing, color conversion processing, color emphasis processing, spatial frequency emphasis, and other structural emphasis processing. The endoscopic image that has undergone special observation image processing is input to the display control unit 60 as a special observation image.

[0056] The lesion information processing unit 66 determines the acquisition of observation conditions, the extraction of lesion information, and the display format of the lesion information based on the input endoscopic image. The endoscopic image, lesion information, and display format of the lesion information are sent to the display control unit 60. Details regarding the lesion information processing unit 66 will be described later.

[0057] The display control unit 60 controls the display of images output from the image processing unit 58 onto the display 18. Specifically, in either a normal viewing mode or a special viewing mode, the display control unit 60 converts the normal viewing image or the special viewing image into a video signal that can be displayed in full color on the display 18. The converted video signal is then input to the display 18. As a result, the normal viewing image or the special viewing image is displayed on the display 18.

[0058] Furthermore, in the lesion information display mode, the display control unit 60 converts the endoscopic image into a video signal that can be displayed in full color on the display 18 and that displays lesion information in a format corresponding to the lesion information. The converted video signal is input to the display 18. As a result, the display 18 displays an endoscopic image with lesion information superimposed on it.

[0059] like Figure 4 As shown, the lesion information processing unit 66 includes an observation condition acquisition unit 70, a lesion information acquisition unit 72, and a display format determination unit 74. The observation condition acquisition unit 70 acquires at least one of the following observation conditions: the moving speed of the endoscope 12, the observation distance between the endoscope 12 and the observed object, or the brightness of the observed object. Observation conditions refer to other conditions, including photographic conditions at the time when the operator takes a picture of the observed object.

[0060] Specifically, the observation conditions include the movement speed of the tip 12d of the endoscope 12. The movement speed is obtained based on differential comparisons (simplified block matching, defining sub-blocks or search ranges) of several endoscopic images before or after the frame acquired at the captured time point, and movement information of the tip 12d obtained from a position information sensor (not shown) located on the tip 12d of the endoscope 12. The movement speed is used to determine whether the time point of the operator's endoscopic observation is the time point for detecting a lesion or simply the time point for moving to the target site. Furthermore, a frame refers to a unit of time in the imaging sensor 44 that includes at least the period from a specific time point to the end of signal reading.

[0061] The observation distance is preferably expressed as the distance between the tip 12d of the endoscope 12 and the object being observed. The observation distance can be represented by the zoom level used when the object is magnified or reduced by operating the zoom operation unit 12h. For example, the zoom level is determined based on the magnification of the object being observed (no magnification, 25x, 50x, 75x, 125x, etc.). Furthermore, the observation distance can be obtained by irradiating the object with a ranging laser from the tip 12d of the endoscope 12 and based on the irradiation position of the ranging laser within the object. The observation distance can also be obtained by measuring the area of ​​the halo region (a region with extremely high brightness) generated by the illumination light emitted from the tip 12d of the endoscope 12. In this case, a large halo region results in a short observation distance, while a small halo region results in a long observation distance. The observation distance is used to determine whether the time point at which the operator performs endoscopic observation is for diagnosing the presence of a lesion, determining the extent of a lesion, or differentiating the stage of a lesion.

[0062] The brightness of the object being observed is preferably calculated based on the endoscopic image. For example, the brightness of the object can be the average of the pixel values ​​of the entire endoscopic image, or it can be a value obtained by measuring the area of ​​dark regions with pixel values ​​below a specific value within the effective pixel region of the endoscopic image. The brightness of the object being observed is used to determine whether the time point at which the operator performs endoscopic observation is suitable for lesion detection, etc.

[0063] At the time point when the observation conditions are acquired, the lesion information acquisition unit 72 acquires lesion information, including at least the credibility of the lesion obtained from the endoscopic image or information for diagnostic purposes. The credibility of the lesion is preferably calculated by performing AI (Artificial Intelligence) processing on the endoscopic image. For example, the credibility of the lesion is preferably expressed as a value such as "60" or "80". As for the AI ​​processing, a CNN (Convolutional Neural Network) is preferably used. The diagnostic purpose is preferably input by the operator through the operator interface 19. The diagnostic purpose includes lesion presence diagnosis, lesion extent diagnosis to determine the extent of the lesion, or differential diagnosis of the lesion stage, etc. Furthermore, the lesion information may be information obtained by extracting vascular information from the endoscopic image and based on features such as vascular density and density distribution, vascular thickness variation and distribution, vascular diameter distribution and presence or absence of bleeding, and the regularity and complexity of vascular and surface structures based on AI.

[0064] The display format determination unit 74 determines the display format of the lesion information on the display 18 based on at least one of the observation conditions or lesion information. For example, the display format of the lesion information could be... Figure 5As shown, there exists a non-display format where lesion information is not displayed, either within the observation image display area RI that displays the endoscopic image or outside the observation image display area R0 that displays information other than the endoscopic image. Furthermore, as a display format for lesion information, such as... Figure 6 As shown, there is a display format for displaying the lesion information DI in at least one of the following: the observation image display area R[ displaying the endoscopic image, or the observation image display area outside RO displaying information other than the endoscopic image outside the observation image display area.

[0065] The following is a detailed explanation of the display format determination method based on the display format determination unit 74. The display format determination unit 74 determines that the display format when the moving speed is a first moving speed differs from the display format when the moving speed is a second moving speed, which is slower than the first moving speed. It is considered that a first moving speed, which is a high speed exceeding a certain speed threshold, and the endoscope tip 12d is moved at this speed, indicates that the endoscope is moving towards the target observation site and is not intended to acquire lesion information. Therefore, when the moving speed is the first moving speed, the display format determination unit 74 determines the display format of the lesion information as a non-display display format. Furthermore, it is considered that in dark environments, such as when the brightness of the observed object is below a brightness threshold, the detection of lesion information is unreliable. Therefore, when the brightness of the observed object is below a brightness threshold, the display format determination unit 74 also determines the display format of the lesion information as a non-display display format.

[0066] When the movement speed is a second movement speed and the brightness of the observed object is above a brightness threshold, the display format determination unit 74 determines the display format for the lesion information as a display format. The second movement speed is considered to be a low speed below a certain speed threshold, and the endoscope tip 12d is moved at this second movement speed to acquire lesion information. Since the types of lesion information to be acquired vary depending on the observation distance, it is preferable to use different display formats depending on the observation distance.

[0067] Specifically, when the observation distance is a first observation distance, the display format determination unit 74 determines different display formats based on the reliability of the lesion; when the observation distance is a second observation distance shorter than the first observation distance, it determines different display formats based on the diagnostic purpose. The first observation distance is preferably, for example, a distance for distant observation performed in screening or similar situations. The second observation distance is preferably, for example, a distance for close-up observation performed in situations such as lesion extent diagnosis or differential diagnosis.

[0068] The display format determination unit 74 preferably determines the display format as follows: when the observation distance is a first observation distance and the confidence level of the lesion is above a confidence level threshold, the lesion information is displayed on the display 18 frame by frame. For example, in this case... Figure 7 As shown, the confidence level of one of the lesion information DIs is displayed continuously in each frame. Additionally, in Figure 7 In this system, the confidence level of the lesion can be displayed in place of numerical values, or, in addition to numerical display, a graph can be used to display the confidence level outside the observation image display area (RO). Furthermore, lesion information can also be displayed within the observation image display area (RI). For example, lesion information can be visualized and overlaid on the observation image according to operator commands.

[0069] On the other hand, preferably, when the confidence level of the lesion is lower than a confidence threshold at the first observation distance, a first display format is determined as the display format. In this first display format, multiple specific frames before and after the frame with a confidence level lower than the confidence threshold are determined, and the lesion information is displayed on the display 18 according to a first calculation based on the lesion information of the multiple specific frames. Specifically, preferably, in the first display format, when the number of frames with a confidence level of a certain value or higher among the multiple frames is a certain number or more, the lesion information is displayed on the display 18. This is because, in cases such as when the confidence level of the lesion is lower than the confidence threshold, in order to suppress flickering caused by continuous display of lesion information and prevent lesions from being missed, the display of lesion information is prevented from becoming non-displayable.

[0070] For example, such as Figure 8 As shown, when the confidence level of the lesion in the 5th frame is "60" which is below a certain value (e.g., set to "80"), the 5th frame and the 1st to 4th frames preceding the 5th frame are determined as multiple specific frames. When the specific number used as the criterion for determining whether to display lesion information is set to 3 frames, since the confidence level of the 1st to 3rd frames among the 1st to 5th frames is above the certain value of "80", the number of frames with a confidence level above the certain value is the specific number "3 frames" or more. At this time, in the 5th frame, according to the first calculation processing based on the lesion information of the 1st to 5th frames, the lesion information is displayed on the display 18.

[0071] The content displayed as lesion information is preferably set to be the content obtained by performing a first calculation process, for example, calculating representative values ​​(average value, maximum value) of the credibility of the first to fifth frames. Figure 8In the image, the average confidence level of frames 1-5, "78", is displayed as lesion information (DI) outside the observation image display area (RO). In addition to displaying numerical information, lesion information can also be displayed graphically. Furthermore, lesion information can also be displayed within the observation image display area (RI). For example, lesion information can be visualized and overlaid on the observation image according to operator commands.

[0072] When the observation distance is a second observation distance and the diagnostic purpose is lesion extent diagnosis, the display format determination unit 74 determines a second display format as the display format. In this second display format, based on a second calculation process of lesion information based on multiple range diagnostic frames, lesion information related to lesion extent diagnosis is displayed on the display 18. In the second display format, it is preferable to determine the lesion extent based on the lesion information of multiple range diagnostic frames and display the lesion information on the display 18 using the lesion extent.

[0073] When the diagnostic objective is set to lesion extent diagnosis, the lesion information acquisition unit 72 calculates the confidence level of the lesion in each pixel or small region of the endoscopic image, and sets the lesion extent DRx based on the overall confidence level of the pixels or small regions above the extent threshold. When the display format determination unit 74 determines that there are 5 frames for multiple extent diagnosis, and wants to display lesion information related to the lesion extent as lesion information, such as... Figure 9 As shown, as the second operation, the average confidence level of small regions SR1 to SR5 in 5 frames is calculated, and the range of lesions for resetting is obtained by taking the average value as the range of small regions above the threshold.

[0074] Then, as Figure 10 As shown, as the second operation, the lesion extent DRx before reset is reset to the lesion extent DRy for reset. Then, RI is overlaid within the observation image display area to emphasize the portion corresponding to the lesion extent for reset. Simultaneously, it is preferable to display a representative value (average, etc.) of the confidence level in the lesion extent DRy outside the observation image display area at RO. Figure 10 The confidence level (XX) is displayed in the image. As a result, unevenness in the lesion extent in each frame can be suppressed, thus reducing flicker. Furthermore, the small region is preferably a region where the vertical pixels are set to multiple pixels. Also, the confidence level may not be displayed outside the observation image display area (RO). Furthermore, the display of lesion information using the lesion extent is preferably displayed at the interval of multiple diagnostic frames.

[0075] When the observation distance is a second observation distance and the diagnostic purpose is differential diagnosis, the display format determination unit 74 determines a third display format as the display format. In this third display format, based on a third calculation process of lesion information based on multiple differential diagnosis frames, lesion information related to differential diagnosis is displayed on the display 18. In the third display format, differential content is determined based on the lesion information of multiple differential diagnosis frames, and the lesion information is displayed on the display 18 using the differential content.

[0076] When the diagnostic objective is set as differential diagnosis, the lesion information acquisition unit 72 integrates the features of pixels or small regions in each endoscopic image and determines the severity or stage and confidence level of the lesion region on a frame-by-frame basis. For example, in the case of Barrett's esophagus, stages such as "Barrett's esophagus without dysplasia," "high-grade dysplasia," and "adenocarcinoma" exist, with a confidence level expressed as "adenocarcinoma: 60." Furthermore, in the case of colorectal cancer, stages such as "benign polyp," "adenoma," and "adenocarcinoma" exist, with a confidence level expressed as "benign polyp: 80."

[0077] Then, when the display format determination unit 74 determines that there are 5 frames for multiple differential diagnosis, such as... Figure 11 As shown, as the third operation, based on the stage discrimination results JD1 to JD5 and the confidence levels PB1 to PB5 of the 5 frames, the final stage discrimination result JDf and the confidence level PBf are calculated, and the final stage discrimination result JDf and the confidence level PBf are displayed on the display 18 as lesion information of the identification content.

[0078] For example, if the differential diagnosis is Barrett's esophagus, and 4 out of 5 frames in the stage discrimination results are "high-grade dysplasia," then "high-grade dysplasia" is taken as the final stage discrimination result JDf. Furthermore, the representative value (average, etc.) of the confidence level of the 4 frames classified as "high-grade dysplasia," "60," is taken as the final confidence level PBf. Then, as... Figure 12 As shown, in displaying lesion information DIJ using identification content, within the observation image display area RI, the region RJ included in the specific range of the final confidence level "60" is highlighted, while outside the observation image display area R0, "High dysplasia, confidence level: 60" is displayed. Alternatively, the confidence level can be displayed graphically. Furthermore, the confidence level may not be displayed outside the observation image display area RO. Moreover, the display of lesion information using identification content is preferably displayed periodically across multiple diagnostic frames.

[0079] Next, according to Figure 13The flowchart is explained below. If the operator switches the lesion information display mode via the operation mode switch 12f, the acquisition of observation conditions begins, and at the time point of acquiring the observation conditions, the acquisition of lesion information begins. The observation information includes at least one of the following: the movement speed of the endoscope 12, the observation distance between the endoscope 12 and the observed object, or the brightness of the observed object. The lesion information includes at least one of the following: the reliability of the lesion obtained from the endoscopic image or the diagnostic purpose.

[0080] If the acquisition of observation conditions and lesion information is completed, the display format determination unit 74 determines the display format of the lesion information on the display 18 based on at least one of the observation conditions or the lesion information. The display control unit 60 displays the lesion information on the display 18 according to the display format determined by the display format determination unit 74.

[0081] Furthermore, in the lesion information display mode, when automatically switching and emitting a first illumination light and a second illumination light with different emission spectra, the first illumination light is emitted in the first emission pattern, and the second illumination light is emitted in the second emission pattern. Thus, by switching and emitting the first illumination light and the second illumination light in frame units, the display image for displaying lesion information can be acquired based on the emission of the first illumination light, and the lesion information acquisition image for acquiring lesion information can be acquired based on the emission of the second illumination light.

[0082] Specifically, the first luminescent pattern is preferably as follows: Figure 14 The number of frames during the first illumination period shown is the same for the first A emission pattern in each of the first illumination periods, and as shown in the diagram. Figure 15 The number of frames shown during the first illumination period is any one of the different first B emission patterns during each first illumination period. Additionally, Figure 14 and Figure 15 In this context, the second illumination period refers to the period during which the second illumination light is emitted. Furthermore, the period is represented by the number of frames.

[0083] The second luminescent pattern is preferably as follows: Figure 14 The second A pattern, as shown, has the same number of frames during each second illumination period and the same emission spectrum of the second illumination light during each second illumination period. Figure 16 The second B pattern, as shown, has the same number of frames during each second illumination period and the emission spectrum of the second illumination light is different during each second illumination period. Figure 17 The second C pattern, shown where the number of frames differs during each second illumination period and the emission spectrum of the second illumination light is the same during each second illumination period, is as follows: Figure 18The second D pattern shown has different frame numbers during each second illumination period and different emission spectra of the second illumination light during each second illumination period. Additionally, the emission spectrum of the first illumination light may be the same or different during each first illumination period.

[0084] Preferably, the first illumination period is longer than the second illumination period, and the first illumination period is preferably two frames or more. For example, Figure 14 In this process, when the first luminous pattern is set to pattern 1A and the second luminous pattern is set to pattern 2A (the number of frames during the second illumination period is the same, and the emission spectrum of the second illumination light is the same), the first illumination period is set to 2 frames, and the second illumination period is set to 1 frame. The first illumination light is used to generate the display image displayed on the display 18, so it is preferable to obtain a bright image by illuminating the object being observed with the first illumination light.

[0085] For example, the first illumination light is preferably white light. On the other hand, since the second illumination light is used to acquire lesion information, it is preferable to obtain an image suitable for acquiring lesion information by illuminating the observed object with the second illumination light. For example, the second illumination light is preferably a short-wavelength narrowband light such as violet light.

[0086] In the above embodiments, the display format of the lesion information is determined in real time according to the observation conditions or lesion information. However, real-time performance can also be considered, and the display format of the lesion information can be predetermined according to the observation conditions or lesion information, and the display format corresponding to the acquired observation conditions or lesion information can be selected from the determined display format.

[0087] In the above embodiments, the hardware structure of the processing units that perform various processes, including the conventional observation image generation unit 62, special observation image generation unit 64, lesion information processing unit 66, central control unit 68, observation condition acquisition unit 70, lesion information acquisition unit 72, and display format determination unit 74, which are included in the light source processor 21, camera processor 45, image acquisition unit 50, DSP 52, noise reduction unit 54, image processing switching unit 56, and image processing unit 58, are various processors as shown below. These processors include general-purpose processors that execute software (programs) and function as various processing units, such as CPUs (Central Processing Units), processors whose circuit structures can be changed after manufacturing, such as FPGAs (Field Programmable Gate Arrays), which are programmable logic devices (PLDs), and processors with circuit structures specifically designed for performing various processes, such as dedicated circuits.

[0088] A processing unit can be composed of one of these various processors, or it can be composed of a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of CPUs and FPGAs). Furthermore, a single processor can constitute multiple processing units. As examples of a single processor constituting multiple processing units, there are two main approaches: firstly, represented by a client or server computer, where a combination of one or more CPUs and software constitutes a single processor, which functions as multiple processing units. Secondly, there are approaches such as System-on-a-Chip (SoC), where a processor that implements the overall system functionality including multiple processing units is used through a single IC (integrated circuit) chip. Thus, various processing units are constructed using one or more of the aforementioned processors as their hardware structure.

[0089] Furthermore, more specifically, the hardware structure of these various processors is a circuit composed of combined semiconductor components and other circuit elements. And the hardware structure of the storage section is a storage device such as an HDD (hard disk drive) or an SSD (solid state drive).

[0090] Symbol Explanation

[0091] 10-Endoscope system; 12-Endoscope; 12a-Insertion section; 12b-Operating section; 12c-Bend section; 12d-Front end section; 12e-Angle knob; 12f-Mode switching switch; 12g-Still image acquisition command section; 12h-Zoom operation section; 14-Light source device; 16-Processor device; 18-Display; 19-Operator interface; 20-Light source unit; 21-Processor for light source; 23-Optical path coupling unit; 25-Light guide; 30a-Illumination optical system; 30b-Camera optical system; 32-Illumination lens; 42-Objective lens. 43-Zoom lens, 44-Image sensor, 45-Image processor, 46-CDS / AGC circuit, 48-A / D converter, 50-Image acquisition unit, 52-DSP, 54-Noise reduction unit, 56-Image processing switching unit, 58-Image processing unit, 60-Display control unit, 62-Routine observation image generation unit, 64-Special observation image generation unit, 66-Lesion information processing unit, 68-Central control unit, 69-Still image storage memory, 70-Observation condition acquisition unit, 72-Lesion information acquisition unit, 74-Display format determination unit.

Claims

1. A processor device, wherein, The image processing processor performs the following controls: Acquire observation conditions including the endoscope's moving speed, the observation distance between the endoscope and the object being observed, and the brightness of the object being observed; At the time point when the observation conditions are obtained, lesion information is obtained including at least one of the confidence level of the lesion obtained from the endoscopic image and the diagnostic purpose; The display format of the lesion information on the monitor is determined based on at least one of the observation conditions and the lesion information. The lesion information is displayed on the monitor in the described display format; Based on the movement speed, the brightness, and the observation distance, a display format for showing the lesion information is determined, which may vary depending on the reliability or the diagnostic purpose. When the viewing distance is the first viewing distance, the image processing processor determines different display formats based on the confidence level. When the observation distance is a second observation distance that is shorter than the first observation distance, the image processing processor determines different display formats according to the diagnostic purpose.

2. The processor device according to claim 1, wherein, The image processing processor determines that the display format when the moving speed is a first moving speed is different from the display format when the moving speed is a second moving speed, which is slower than the first moving speed.

3. The processor device according to claim 2, wherein, The image processing processor determines, in at least one of the following cases—the moving speed being the first moving speed and the brightness being lower than a brightness threshold—that a non-display display format is used to not display the lesion information.

4. The processor device according to claim 2, wherein, When the moving speed is the second moving speed and the brightness is above the brightness threshold, the image processing processor determines the display format for displaying the lesion information.

5. The processor device according to claim 1, wherein, When the observation distance is the first observation distance and the confidence level is above a confidence level threshold, the image processing processor determines, as the display format, how to display the lesion information on the display in frames. When the observation distance is the first observation distance and the confidence level is lower than the confidence level threshold, the image processing processor determines a first display format as the display format. In the first display format, it determines multiple specific frames before and after the frame whose confidence level is lower than the confidence level threshold and displays the lesion information based on a first operation on the lesion information of the multiple specific frames.

6. The processor device according to claim 5, wherein, In the first display format, when the number of frames with high credibility among the plurality of specific frames is more than a certain number, the lesion information is displayed on the display.

7. The processor device according to claim 1, wherein, When the observation distance is the second observation distance and the diagnostic purpose is lesion range diagnosis, the image processing processor determines a second display format as the display format. In the second display format, based on a second calculation process of the lesion information based on multiple range diagnosis frames, the processor displays the lesion information related to the lesion range diagnosis. When the observation distance is the second observation distance and the diagnostic purpose is differential diagnosis, the image processing processor determines a third display format as the display format. In the third display format, based on a third operation processing of the lesion information based on multiple differential diagnosis frames, the lesion information related to the differential diagnosis is displayed.

8. The processor device according to claim 7, wherein, In the second display format, the lesion range is determined based on the lesion information of the plurality of range diagnostic frames, and the lesion information is displayed using the lesion range.

9. The processor device according to claim 7, wherein, In the third display format, the identification content is determined based on the lesion information of the plurality of differential diagnostic frames, and the lesion information is displayed using the identification content.

10. The processor device according to any one of claims 1 to 9, wherein, The display image used to show the lesion information is obtained based on the emission of the first illumination light, and the lesion information acquisition image used to acquire the lesion information is obtained based on the emission of the second illumination light, which has a different emission spectrum from the first illumination light.

11. A method of operating a processor device, wherein, The image processing processor performs the following controls: Acquire observation conditions including the endoscope's moving speed, the observation distance between the endoscope and the object being observed, and the brightness of the object being observed; At the time point when the observation conditions are obtained, lesion information is obtained including at least one of the confidence level of the lesion obtained from the endoscopic image and the diagnostic purpose; The display format of the lesion information on the monitor is determined based on at least one of the observation conditions and the lesion information. The lesion information is displayed on the monitor in the described display format; Based on the movement speed, the brightness, and the observation distance, a display format for showing the lesion information is determined, which may vary depending on the reliability or the diagnostic purpose. When the viewing distance is the first viewing distance, the image processing processor determines different display formats based on the confidence level. When the observation distance is a second observation distance that is shorter than the first observation distance, the image processing processor determines different display formats according to the diagnostic purpose.

Images