Image processing device, endoscope system, and image processing method

Abstract

The present invention provides an image processing device, an endoscope system, and an image processing method for stably displaying a determination result related to a disease using an endoscopic image. An image processing device (16) includes a processor, wherein the processor acquires a plurality of endoscopic images obtained by photographing an observation object through an endoscope (12), calculates a raw score related to the determination of the severity or stage of the disease of the observation object based on each endoscopic image, determines a final score based on the raw score, and controls the display of the final score and / or the change in the final score over time on a display (18) in real time.

Description

Technical Field

[0001] The present invention relates to an image processing device, an endoscope system, and an image processing method for supporting diagnosis using an endoscopic image captured by an endoscope. Background Art

[0002] In the medical field, diagnosis using endoscope systems, which include a light source device, an endoscope, and a processor device, is widely performed. In this type of diagnosis, computer-aided diagnosis (CAD) technology has been developed. It uses images obtained by an endoscope (hereinafter referred to as endoscopic images) to determine the stage of a specific disease in the subject, among other things. Using CAD, information such as the severity of a disease or scores associated with pathological findings can be calculated in real time or near real time and displayed on a monitor.

[0003] A technology for using CAD more conveniently has been disclosed. For example, an image analysis device capable of automatically distinguishing between super-magnified images and non-magnified images is known for using CAD in endoscopic images (Patent Document 1).

[0004] Previous technical literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2019-111040 Summary of the Invention

[0007] Technical issues to be solved by the invention

[0008] In real-time CAD using endoscopic images, the position of the observation object in the endoscopic image can easily change, making it difficult to obtain stable determination results. For example, when determining the stage of ulcerative colitis through image processing of endoscopic images, subtle changes in the observation position can lead to subtle changes in the determination results, resulting in unstable stage determinations. Therefore, to make CAD easier to use, it is desirable to obtain stable determination results.

[0009] In view of the above-mentioned circumstances, the present invention aims to provide an image processing device, an endoscope system, and an image processing method for stably displaying a determination result related to a disease using an endoscopic image.

[0010] Means for solving technical problems

[0011] The present invention is an image processing device including a processor. The processor acquires a plurality of endoscopic images of an observation object captured by an endoscope at different times, calculates a raw score related to the severity or stage of a disease of the observation object based on each endoscopic image, determines a final score based on the raw scores, and controls the display of the final score and / or its temporal changes on a display in real time.

[0012] Preferably, the processor calculates two or more different raw scores.

[0013] Preferably, the processor calculates a raw score based on a first feature value obtained by analyzing the endoscopic image.

[0014] Preferably, the first feature value is a value related to a surface layer vascular dense area, an intramucosal bleeding area, or an extramucosal bleeding area included in the endoscopic image.

[0015] Preferably, the processor executes a learned first machine learning model generated by inputting past endoscopic images associated with the raw scores into the machine learning model, and calculates the raw scores based on the endoscopic images.

[0016] Preferably, the processor determines the final score based on raw scores calculated from a plurality of endoscopic images acquired during a predetermined period before a point in time when the final score is determined.

[0017] Preferably, the processor determines the final score by performing a moving average or FIR filter processing or IIR filter processing on the plurality of raw scores.

[0018] Preferably, the processor determines the final score based on raw scores calculated just before or after the point in time when the final score is determined.

[0019] Preferably, the processor determines for each endoscopic image whether the endoscopic image is suitable for calculation of the raw score, and does not calculate a raw score for an endoscopic image determined to be unsuitable for calculation of a raw score.

[0020] Preferably, the processor determines whether the endoscopic image is suitable for calculation of a raw score based on a second feature value of the endoscopic image.

[0021] The second feature value is preferably a value related to at least one selected from the group consisting of halo distribution, spatial frequency distribution, brightness value distribution, shadow distribution, magnification index, and reflected light distribution of illumination light irradiated on the observation object in the endoscopic image.

[0022] The preferred processor executes a learned second machine learning model generated by inputting past endoscopic images that establish a correspondence association with whether they are suitable for calculating the original score into the machine learning model, and determines whether the endoscopic images are suitable for calculating the original score.

[0023] Preferably the processor determines the final score based on the raw scores excluding any calculations.

[0024] The preferred processor does not calculate the final score based on the raw scores of multiple endoscopic images acquired during a specified period, and when the number of raw scores other than those not calculated is less than a specified number, the number of raw scores not calculated is greater than a specified number, or the ratio of the number of raw scores not calculated to the number of raw scores based on multiple endoscopic images acquired during a specified period is greater than a specified value.

[0025] Preferably, the temporal change of the final score is displayed by at least one graph showing the relationship between the final score and the time when the final score was determined.

[0026] Preferably, the processor determines an observation target site included in the endoscopic image by performing image analysis on the endoscopic image, and the temporal change of the final score is displayed by at least one graph showing a relationship between the final score, a final score determination time, and the site.

[0027] Preferably, the processor is instructed to obtain a still image.

[0028] When there is an instruction, control is performed to display the final score and / or the change in the final score over time.

[0029] A preferred disease is ulcerative colitis.

[0030] Furthermore, the present invention provides an endoscope system comprising an endoscope for imaging an observation object and an image processing device including a processor. The processor acquires a plurality of endoscopic images of the observation object obtained at different times, calculates a raw score related to the severity or stage of a disease of the observation object based on each endoscopic image, determines a final score based on the raw scores, and controls the display of the final score and / or its temporal changes in real time on a display.

[0031] Furthermore, the present invention is an image processing method comprising: an image acquisition step for acquiring a plurality of endoscopic images of an observation object taken at different times; a raw score calculation step for calculating a raw score related to a determination of the severity or stage of a disease of the observation object based on each endoscopic image; a final score determination step for determining a final score based on the raw scores; and a display control step for controlling the display of the final score and / or the temporal change of the final score on a display in real time.

[0032] Effects of the Invention

[0033] According to the present invention, it is possible to stably display a determination result related to a disease using an endoscopic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is an external view of the endoscope system.

[0035] Figure 2 This is an external view of the operating unit of an endoscope.

[0036] Figure 3 This is a block diagram showing the functions of an endoscope system.

[0037] Figure 4 This is an explanatory diagram for explaining four-color LEDs included in the light source unit.

[0038] Figure 5 Graph showing the spectrum of violet light V, blue light B, green light G, and red light R.

[0039] Figure 6 This is a graph showing the spectrum of special light.

[0040] Figure 7 Graph showing the spectrum of special light including only purple light V.

[0041] Figure 8 This is a block diagram showing the functions of the score processing unit.

[0042] Figure 9 This is an explanatory diagram showing patterns of vascular structure that change according to the severity of ulcerative colitis.

[0043] Figure 10 It is a schematic diagram schematically showing a cross section of the large intestine.

[0044] Figure 11 This is an explanatory diagram explaining the classification of the superficial vascular dense area, the intramucosal bleeding area, and the extramucosal bleeding area by brightness value and spatial frequency.

[0045] Figure 12 This is an explanatory diagram illustrating the correlation between endoscopic images and raw score calculation in a time series manner.

[0046] Figure 13 This is an explanatory diagram for explaining display control of a final score based on a moving average of raw scores.

[0047] Figure 14 This is an explanatory diagram for explaining the display control of the final score displaying the part name.

[0048] Figure 15This is an explanatory diagram for explaining display control of a final score based on a previously obtained raw score.

[0049] Figure 16 This is an explanatory diagram for explaining display control of a final score based on a raw score obtained later.

[0050] Figure 17 This is an explanatory diagram illustrating the correlation between endoscopic images and raw score calculations that are not calculated using a time series.

[0051] Figure 18 This is an explanatory diagram for explaining the display control of the final score when the raw score that is not calculated is included.

[0052] Figure 19 This is an explanatory diagram for explaining display control of the final score indicating the discrimination result.

[0053] Figure 20 This is a flowchart showing a series of processes in the score display mode.

[0054] Figure 21 This is a block diagram showing a diagnosis support device.

[0055] Figure 22 This is a block diagram showing a medical work support device. DETAILED DESCRIPTION

[0056] exist Figure 1 In the embodiment, the endoscope system 10 includes an endoscope 12, a light source device 14, a processor device 16, a display 18, and a console 19. The endoscope 12 is optically connected to the light source device 14 and electrically connected to the processor device 16. The endoscope 12 includes an insertion portion 12a to be inserted into the body of an observation object, an operating portion 12b provided at the base end of the insertion portion 12a, and a bending portion 12c and a distal end portion 12d provided at the distal end side of the insertion portion 12a. The bending portion 12c is adjusted by operating the angle button 12e (refer to FIG. 1 ) of the operating portion 12b. Figure 2 The distal end portion 12d is directed toward a desired direction by the bending action of the bending portion 12c.

[0057] like Figure 2 As shown, the operating unit 12b includes, in addition to the angle knob 12e, a mode switch 12g for switching observation modes, a zoom operating unit 12h for changing the imaging magnification, and a still image acquisition instruction unit 12f for instructing still image acquisition. Furthermore, the observation mode switching operation, zoom operation, or still image acquisition instruction can be performed using the console 19 or a foot switch (not shown) in addition to the mode switch 12g or the scope switch of the still image acquisition instruction unit 12f.

[0058] The endoscope system 10 has three modes: normal observation mode, special observation mode, and score display mode. In normal observation mode, the observation object is illuminated with normal light and photographed, resulting in a normal image in natural colors displayed on the display 18. In special observation mode, the observation object is illuminated with special light of a wavelength band different from that of normal light and photographed, resulting in a special image that emphasizes specific structures displayed on the display 18. In score display mode, a score related to the severity or stage of the observation object's disease is determined based on the endoscopic image composed of the normal image or the special image. The determined score, its temporal changes, and / or the determination result are displayed on the display 18.

[0059] The severity of a disease is the degree of prognosis obtained through treatment. For example, in the case of ulcerative colitis, the disease is divided into three categories: mild, moderate, and severe. In the case of ulcerative colitis, the disease stage (disease stage) is divided into two categories: active and remission. Therefore, the severity determination result is any of mild, moderate, or severe, and the stage determination result is any of active, remission, or remission or non-remission. Furthermore, the score is a display that can identify the severity or stage of the disease under observation, and can be a numerical value or text. In this embodiment, the case where an image processing device determines whether ulcerative colitis is in remission or non-remission is described.

[0060] The processor device 16 connected to the endoscope 12 is an image processing device that executes the score display mode. The image processing device includes a processor. Programs related to the image signal acquisition unit 51, DSP 52, noise reduction unit 53, signal processing unit 55, and video signal generation unit 56 are stored in the memory of the image processing device. A control unit (not shown) composed of a processor executes these programs, thereby realizing the functions of the image signal acquisition unit 51, DSP 52, noise reduction unit 53, signal processing unit 55, and video signal generation unit 56. The score display mode can be executed by other components. For example, the image processing device functions can be provided in an external image processing system separate from the endoscope system 10, and the endoscopic image can be input to the external image processing system to execute the score display mode, with the results of the execution displayed on an external display connected to the external image processing system.

[0061] The processor device 16 is electrically connected to a display 18 and a console 19. The display 18 outputs and displays images of the observation object, scores, temporal changes in scores, determination results, and / or information accompanying the images of the observation object. The console 19 functions as a user interface for accepting input operations such as function settings. Furthermore, an external recording unit (not shown) for recording images or image information may also be connected to the processor device 16.

[0062] exist Figure 3 In the apparatus, the light source device 14 emits illumination light that is irradiated onto the object to be observed. The apparatus includes a light source unit 20 and a light source control unit 21 that controls the light source unit 20. The light source unit 20 is composed of, for example, a semiconductor light source such as a multi-color LED (Light Emitting Diode), a combination of a laser diode and a phosphor, or a halogen light source such as a xenon lamp. Furthermore, the light source unit 20 includes filters, etc., for adjusting the wavelength band of light emitted by the LEDs, etc. The light source control unit 21 controls the amount of illumination light by turning on / off each LED, etc., and adjusting the drive current and drive voltage of each LED, etc. Furthermore, the light source control unit 21 controls the wavelength band of illumination light by, for example, changing the filters.

[0063] like Figure 4 As shown, in this embodiment, the light source unit 20 includes four colors of LEDs: V-LED (Violet Light Emitting Diode: purple light emitting diode) 20a, B-LED (Blue Light Emitting Diode: blue light emitting diode) 20b, G-LED (Green Light Emitting Diode: green light emitting diode) 20c and R-LED (Red Light Emitting Diode: red light emitting diode) 20d.

[0064] like Figure 5 As shown, the V-LED 20a generates violet light V with a central wavelength of 405±10nm and a wavelength range of 380-420nm. The B-LED 20b generates blue light B with a central wavelength of 460±10nm and a wavelength range of 420-500nm. The G-LED 20c generates green light G with a wavelength range of 480-600nm. The R-LED 20d generates red light R with a central wavelength of 620-630nm and a wavelength range of 600-650nm. Furthermore, violet light V is a short-wavelength light used to detect areas with dense superficial blood vessels, intramucosal bleeding, or extramucosal bleeding, used in the fractional display mode. It is preferred that the central wavelength or peak wavelength include 410nm. Therefore, the endoscopic image used in the fractional display mode is preferably an image of the observation object illuminated by violet light V.

[0065] The light source control unit 21 controls the V-LED 20a, B-LED 20b, G-LED 20c, and R-LED 20d. In normal observation mode, the light source control unit 21 controls each LED 20a to 20d to emit normal light with a light intensity ratio of Vc:Bc:Gc:Rc among violet light V, blue light B, green light G, and red light R.

[0066] When the light source control unit 21 is in the special observation mode or fractional display mode, it controls each LED 20a to 20d so that the light intensity ratio of the short-wavelength purple light V to the blue light B, green light G, and red light R is Vs:Bs:Gs:Rs. The special light preferably emphasizes superficial blood vessels. Therefore, the light intensity ratio Vs:Bs:Gs:Rs of the special light preferably makes the light intensity of the purple light V greater than the light intensity of the blue light B. For example, Figure 6 As shown in FIG, the ratio of the light intensity Vs of the purple light V to the light intensity Bs of the blue light B is set to "4:1". Figure 7 As shown, regarding the special light, the light intensity ratio among the purple light V, blue light B, green light G, and red light R can be set to 1:0:0:0, and only the purple light V, which is light of a short wavelength, can be emitted.

[0067] In this specification, the term "light intensity ratio" includes the case where the ratio of at least one semiconductor light source is 0 (zero). Therefore, it also includes the case where any one, or two or more, of the semiconductor light sources are not illuminated. For example, if the light intensity ratio of violet light V, blue light B, green light G, and red light R is 1:0:0:0, then the light intensity ratio can still be achieved even when only one of the semiconductor light sources is illuminated and the other three are not illuminated.

[0068] Light emitted by each of the LEDs 20a to 20d passes through an optical coupling unit (not shown) composed of a reflector, lens, etc., and enters the light guide 41. The light guide 41 is built into the endoscope 12 and the universal cord (the cord that connects the endoscope 12 to the light source device 14 and the processor device 16). The light guide 41 transmits light from the optical coupling unit to the distal end portion 12d of the endoscope 12.

[0069] The distal end portion 12d of the endoscope 12 is provided with an illumination optical system 30a and an imaging optical system 30b. The illumination optical system 30a includes an illumination lens 42, through which illumination light propagated by a light guide 41 is irradiated onto the observation object. The imaging optical system 30b includes an objective lens 43, a zoom lens 44, and an imaging sensor 45. Various types of light, such as reflected light, scattered light, and fluorescence from the observation object, are incident on the imaging sensor 45 via the objective lens 43 and the zoom lens 44. As a result, an image of the observation object is formed on the imaging sensor 45. The zoom lens 44 is freely movable between the telephoto end and the wide-angle end by operating the zoom operation unit 12h, and the observation object imaged on the imaging sensor 45 is magnified or reduced.

[0070] The imaging sensor 45 is a color imaging sensor with one of the R (red), G (green), or B (blue) color filters applied to each pixel. It captures an observation object and outputs image signals in each of the RGB colors. A CCD (Charge Coupled Device) imaging sensor or a CMOS (Complementary Metal-Oxide Semiconductor) imaging sensor can be used as the imaging sensor 45. Furthermore, instead of the imaging sensor 45 with primary color filters, a complementary color imaging sensor with complementary color filters for C (cyan), M (magenta), Y (yellow), and G (green) can be used. When using a complementary color imaging sensor, image signals in the four colors of C, M, and G are output. Therefore, through complementary-to-primary color conversion, the four C, M, and G image signals are converted into three RGB image signals, resulting in the same RGB image signals as those of the imaging sensor 45. Alternatively, a monochrome sensor without color filters can be used in place of the imaging sensor 45.

[0071] The imaging sensor 45 is driven and controlled by an imaging control unit (not shown). The control by the imaging control unit varies depending on the mode. In normal observation mode or fractional display mode, the imaging control unit controls the imaging sensor 45 to capture an image of the observation object illuminated by normal light. Consequently, a Bc image signal is output from the B pixel of the imaging sensor 45, a Gc image signal is output from the G pixel, and an Rc image signal is output from the R pixel. In special observation mode or fractional display mode, the imaging control unit controls the imaging sensor 45 to capture an image of the observation object illuminated by special light. Consequently, a Bs image signal is output from the B pixel of the imaging sensor 45, a Gs image signal is output from the G pixel, and an Rs image signal is output from the R pixel.

[0072] The CDS / AGC (Correlated Double Sampling / Automatic Gain Control) circuit 46 performs correlated double sampling (CDS) or automatic gain control (AGC) on the analog image signal obtained by the imaging sensor 45 . The image signal passed through the CDS / AGC circuit 46 is converted into a digital image signal by an A / D (Analog / Digital) converter 48 . The A / D-converted digital image signal is input to the processor device 16 .

[0073] The processor device 16 includes an image signal acquisition unit 51 , a DSP (Digital Signal Processor) 52 , a noise reduction unit 53 , a memory 54 , a signal processing unit 55 , and a video signal generation unit 56 . The signal processing unit 55 includes a normal image generation unit 61 , a special image generation unit 62 , and a score processing unit 63 .

[0074] The image signal acquisition unit 51 acquires the digital image signal of the endoscopic image input from the endoscope 12. The acquired image signal is sent to the DSP 52. The DSP 52 performs various signal processing on the received image signal, including defect correction processing, offset processing, gain correction processing, linear matrix processing, gamma conversion processing, demosaicing processing, and YC conversion processing. In the defect correction processing, the signal of the defective pixel of the camera sensor 45 is corrected. In the offset processing, the dark current component is removed from the image signal subjected to the defect correction processing, and an accurate zero level is set. The gain correction processing adjusts the signal level of each image signal by multiplying the image signal of each color after the offset processing by a specific gain.

[0075] After gain correction, each color image signal undergoes linear matrix processing to improve color reproducibility. Gamma conversion is then used to adjust the brightness and chroma of each image signal. Demosaicing (also known as isotropy or synchronization) is performed on the gamma-converted image signal, and interpolation is used to generate signals for the missing colors in each pixel. Demosaicing ensures that all pixels have signals for each of the RGB colors. The DSP 52 performs YC conversion on each demosaiced image signal and outputs the luminance signal Y and the color difference signals Cb and Cr to the noise reduction unit 53.

[0076] The noise reduction unit 53 performs noise reduction processing based on, for example, a moving average method or a median filter method on the image signal subjected to the demosaicing processing by the DSP 52 . The image signal with reduced noise is stored in the memory 54 .

[0077] The signal processing unit 55 acquires the noise-reduced image signal from the memory 54. Furthermore, as necessary, it applies signal processing such as color conversion, color emphasis, and structure emphasis to the acquired image signal to generate a color endoscopic image of the observation object. Color conversion converts the image signal's color using a 3×3 matrix, grayscale conversion, and a 3D LUT (lookup table) process. Color emphasis is then applied to the image signal after color conversion. Structure emphasis emphasizes specific tissues and structures within the observation object, such as blood vessels and pit patterns. This is performed on the image signal after color emphasis.

[0078] The signal processing unit 55 includes a normal image generation unit 61, a special image generation unit 62, and a fractional processing unit 63. Depending on the selected mode, the signal processing unit 55 sends the image signal from the noise reduction unit 53 to one of the normal image generation unit 61, the special image generation unit 62, and the fractional processing unit 63. Specifically, for example, when the normal observation mode is selected, the image signal is input to the normal image generation unit 61. When the special observation mode is selected, the image signal is input to the special image generation unit 62. When the fractional display mode is selected, the image signal is input to the fractional processing unit 63.

[0079] The normal image generation unit 61 performs normal image processing on the input Rc, Gc, and Bc image signals for a single frame. This normal image processing includes color conversion processing such as 3×3 matrix processing, grayscale conversion, and three-dimensional LUT (Look Up Table) processing, as well as color emphasis processing and structure emphasis processing such as spatial frequency emphasis. The Rc, Gc, and Bc image signals that have undergone normal image processing are input to the video signal generation unit 56 as normal images.

[0080] The special image generation unit 62 performs special image processing on the input Rs, Gs, and Bs image signals for a single frame. This special image processing includes color conversion processing such as 3×3 matrix processing, grayscale conversion, and 3D LUT processing, as well as color emphasis processing and structure emphasis processing such as spatial frequency emphasis. The Rs, Gs, and Bs image signals that have undergone special image processing are input to the video signal generation unit 56 as special images.

[0081] The endoscopic images generated by the signal processing unit 55 are normal observation images when the observation mode is normal observation mode, and special observation images when the observation mode is special observation mode. Therefore, the content of the color conversion processing, color emphasis processing, and structure emphasis processing differs depending on the observation mode. In normal observation mode, the signal processing unit 55 applies the aforementioned various signal processing steps to render the observed object natural colors, generating normal observation images. In special observation mode, the signal processing unit 55 applies the aforementioned various signal processing steps to at least emphasize the blood vessels of the observed object, generating special observation images. In the special observation images generated by the signal processing unit 55, blood vessels (so-called superficial blood vessels) or blood located shallower within the observed object relative to the mucosal surface appear in a magenta-based color (e.g., brown), while blood vessels (so-called mid-deep blood vessels) located deeper within the observed object relative to the mucosal surface appear in a cyan-based color (e.g., green). Thus, the blood vessels or hemorrhage (blood) of the observed object are emphasized by the color difference compared to the pink-based mucosa.

[0082] The video signal generating unit 56 converts the normal image, special image, or final score determined by the score processing unit 63 output from the signal processing unit 55 into a video signal that can be displayed in full color on the display 18. The converted video signal is input to the display 18. As a result, the normal image, special image, final score, etc. are displayed on the display 18. Furthermore, when a still image acquisition instruction (freeze instruction or release instruction) is input by operating the still image acquisition instruction unit 12f, the signal processing unit 55 saves the generated endoscopic image in the image storage unit 75 (refer to FIG. Figure 8 ) or memory (not shown). The memory is an external storage device connected to the processor device 16 via a LAN (Local Area Network) or the like, for example, a PACS (Picture Archiving and Communication System, Image Storage and Communication System, Reference Figure 21 ) and other systems for archiving endoscopic images, such as file servers and NAS (Network Attached Storage).

[0083] The score processing unit 63 determines the final score and performs control to display the final score and / or the change in the final score over time on the display 18 in real time. Figure 8 As shown, the score processing unit 63 includes an image acquisition unit 71, a raw score calculation unit 72, a final score determination unit 73, and a display control unit 76. In addition, it may include an inappropriate image determination unit 74, an image storage unit 75, and a part determination unit 77.

[0084] In score display mode, the image acquisition unit 71 automatically acquires multiple endoscopic images obtained by the endoscope 12 at different times while capturing the observation object. Endoscopic images include normal observation images and special observation images. However, in this embodiment, the image acquisition unit 71 acquires special observation images that emphasize blood vessels, etc. The image acquisition unit 71 can retrieve endoscopic images from memory as needed. Endoscopic images acquired by the image acquisition unit 71 are sent to the raw score calculation unit 72, the unsuitable image determination unit 74, or the image storage unit 75.

[0085] The raw score calculation unit 72 calculates a raw score related to the determination of the severity or stage of the disease of the observation object based on the plurality of endoscopic images acquired by the image acquisition unit 71. The raw score related to the determination of the severity or stage of the disease of the observation object is set to a numerical value that can identify the severity or stage of the disease of the observation object contained in the endoscopic image. The raw score calculation unit 72 can include a first calculation unit 81 (refer to Figure 8 ) and the second calculation unit 82 (reference Figure 8) one or both of the following.

[0086] Preferably, the raw score calculation unit 72 calculates the raw score based on the feature quantity (first feature quantity) obtained by analyzing the endoscopic image in the first calculation unit 81. Examples of the feature quantity include feature quantities related to blood vessels, such as the number, thickness, length, number of branches, branch angles, distances between branch points, number of intersections, inclination, density, color, blood concentration, oxygen saturation, presence or absence of bleeding, bleeding area, or flow rate, and feature quantities related to the color of the mucous membrane.

[0087] For example, raw score feature quantities related to the severity or stage of ulcerative colitis are preferably quantities related to the dense superficial vascular areas, intramucosal bleeding areas, and extramucosal bleeding areas contained in endoscopic images. For the dense superficial vascular areas, the raw score is obtained by performing image analysis on the endoscopic image, extracting the subject's blood vessels from the endoscopic image using a frequency filter, etc., and counting the number of pixels in the dense vascular areas in the endoscopic image. Similarly, for the intramucosal or extramucosal bleeding areas, the subject's intramucosal or extramucosal bleeding is extracted from the endoscopic image using, for example, the G value relative to the R value proportional to the hemoglobin level, and counting the number of pixels in the intramucosal or extramucosal bleeding areas in the endoscopic image as the raw score.

[0088] As a method for counting the number of pixels of the surface blood vessel dense area, the intramucosal bleeding area, and the extramucosal bleeding area contained in the endoscopic image, the following method can also be used. First, the present inventors have found that Figure 9 As shown in (A) to (E), the vascular structure pattern of ulcerative colitis, which is a disease determined by this embodiment, changes as the severity worsens. When ulcerative colitis is relieved or does not occur, the pattern of the surface blood vessels 85 is regular ( Figure 9 (A)) or the degree to which the regularity of the pattern of the surface blood vessels 85 is disturbed to some extent ( Figure 9 On the other hand, when ulcerative colitis is not relieved and the severity is mild, the density of the surface blood vessels 85 is sparse ( Figure 9 (C). In addition, when ulcerative colitis is not resolved and is of moderate severity, intramucosal bleeding occurs86 ( Figure 9 (D)). In addition, when ulcerative colitis is not relieved and the severity is moderate to severe, extramucosal bleeding occurs 87 ( Figure 9 (E)). The raw score calculation unit 72 can calculate the raw score by using the pattern change of the above-mentioned vascular structure.

[0089] Here, the surface vascular dense area refers to the state where the surface blood vessels are winding and clustered. In the observation on the image, it means that many surface blood vessels surround the intestinal gland volume (crypt) (refer to Figure 10 ) around the mucosal area. Intramucosal bleeding refers to bleeding in the mucosa (refer to Figure 10 Bleeding that occurs within the mucosa must be differentiated from bleeding within the lumen. Intramucosal bleeding refers to bleeding that occurs within the mucosa and not within the lumen (lumen, folds, or foramina) as observed on the image. Extramucosal bleeding refers to a small amount of blood that enters the lumen, blood that can be visually detected even after rinsing the lumen from the lumen or mucosa in front of the endoscope, or intraluminal blood that oozes from bleeding mucosa.

[0090] The raw score calculation unit 72 classifies the surface blood vessel dense area, the mucosal bleeding area or the extramucosal bleeding area according to the frequency characteristics and brightness values ​​obtained from the special observation image. Figure 11 Classification is performed as shown. The density of surface blood vessels is represented by a low brightness value and a high frequency frequency characteristic. In intramucosal bleeding, the brightness value is represented by a medium brightness, and the frequency characteristic is represented by a medium frequency. In extramucosal bleeding, the brightness value is represented by a low brightness, and the frequency characteristic is represented by a low frequency. In addition, when the various structures of the special observation image are represented by brightness values ​​and frequency characteristics, in addition to the above-mentioned three areas of dense surface blood vessels, intramucosal bleeding areas, or extramucosal bleeding areas, blurred dark areas of special observation images or endoscope shadows (shadows formed in the center of the endoscope image when the front end 12d of the endoscope is moved along the lumen) are also included. Using the above classification, the density of surface blood vessels, intramucosal bleeding, and extramucosal bleeding required for determining whether ulcerative colitis has been relieved or not are extracted from the dense surface blood vessels area, intramucosal bleeding area, or extramucosal bleeding area.

[0091] Regarding the spatial frequency, the spatial frequency component distribution is calculated by applying a Laplace filter to the special observation image. Based on the spatial frequency component distribution, for example, when the standard deviation of the frequencies of the nine pixels near a specific pixel is less than a constant value, the specific pixel is set as a pixel belonging to the low-frequency region. The high-frequency region is extracted by Hessian analysis of the spatial frequency component distribution. In the intermediate frequency region, the portion obtained by removing the low-frequency region and the high-frequency region in the special observation image is set as the intermediate frequency region. In this way, the pixels of the special observation image are classified according to the spatial frequency and the brightness value, so that the number of pixels in the surface blood vessel dense area, the mucosal bleeding area, or the mucosal bleeding area can be calculated. As described above, since the severity or stage of the disease can be well determined by the above structure, the disease is preferably ulcerative colitis.

[0092] The raw scores calculated based on different first feature quantities are different types of raw scores. The raw score calculation unit 72 may calculate one type of raw score or two or more types of raw scores.

[0093] The raw score calculation unit 72 calculates the raw score based on the endoscopic image in the second calculation unit 82. The second calculation unit 82 has a first machine learning model that has been learned. The first machine learning model that has been learned is generated by inputting past endoscopic images that are associated with the raw score into the machine learning model. That is, it is a machine learning model that is generated by inputting past endoscopic images into the machine learning model and making it learn so as to correctly output the raw score that is associated with the raw score. Therefore, in addition to inputting past endoscopic images that are associated with the score into the machine learning model, learning also includes adjusting various parameters. As the raw score that is associated with the past endoscopic image, for example, it can also be a feature value, or the severity or stage of the disease of the observed object can be quantified. For example, the first machine learning model has two or more machine learning models corresponding to each feature value, and can calculate two or more raw scores.

[0094] The final score determination unit 73 determines the final score based on the raw scores. The raw scores calculated by the raw score calculation unit 72 are quantities relevant to determining the severity or stage of the disease and serve as indicators of the severity of the disease symptoms or the degree of progression of the disease. Therefore, the method for determining the final score can be adjusted according to the purpose of the determination. For example, when determining the severity or stage progression, the final score can be set based on the severity of the disease, with the highest raw score indicating a higher severity and the direction of disease progression, and the severity or stage can be determined based on the most advanced part of the disease. Examples of methods for determining the final score in this context include presetting thresholds for each of the three raw scores: the number of pixels in the superficial vascularized area, the number of pixels in the intramucosal hemorrhage area, or the number of pixels in the extramucosal hemorrhage area, and determining the final score as the raw score that exceeds the threshold. Alternatively, the final score can be determined based on the raw score that exceeds the threshold by the greatest margin among the three raw scores. On the other hand, for screening, for example, the thresholds can be set lower.

[0095] The display control unit 76 controls the display of the final score and / or the change in the final score over time on the display 18 in real time. For example, the final score can be displayed in the following ways: displaying the final score value in real time on the display 18; displaying the final score value in the form of a graph with the final score value on the vertical axis and the passage of time on the horizontal axis, and displaying the change in the final score over time in the form of a graph; or displaying the final score value and a predetermined threshold value based on a message indicating the severity or stage of the disease. Furthermore, the display control unit 76 also controls not to display the final score. For example, when the final score is less than a predetermined threshold value, the final score is not displayed. Furthermore, displaying in real time means displaying immediately, not strictly simultaneously.

[0096] When the position of the imaging subject and the observed object change, the endoscopic image obtained also changes, and the raw score calculated from the endoscopic image also changes. However, since the score processing unit 63, configured as described above, uses the raw score to determine the final score and then displays it on the display 18, changes in the raw score are not directly displayed on the display 18. By using the raw score as the final score, a more stable score is displayed on the display 18 even in real time. Furthermore, the final score is controlled not to be displayed depending on the value of the final score, so when the score is displayed, it is displayed as a more stable score. Furthermore, by displaying the changes in the final score over time in real time, even if the imaging subject position and the observed object change during a single observation, the user can clearly understand the changes in severity or stage caused by the imaging subject position, or the difference in the final score at the location with the highest severity or most advanced stage. Therefore, while focusing on observation, the user can also understand the location where the severity or stage of the disease has deteriorated the most during the observation. As described above, the image processing device prevents fluctuations in the final score even with subtle changes in the observed object, enabling more stable and durable score calculation and display of the final score. Therefore, the image processing apparatus contributes to preventing omission of lesions or easier observation by the user who performs observation.

[0097] The score processing unit 63 may include an unsuitable image discriminating unit 74. The unsuitable image discriminating unit 74 discriminates whether the endoscopic image is suitable or unsuitable for the calculation of the raw score. By the discrimination, the endoscopic image is set to be suitable or unsuitable. Endoscopic images are sometimes unsuitable as images for calculating the raw score. For example, in an endoscopic image, extreme values ​​may be calculated when calculating the raw score due to jitter caused by shooting the front end portion 12d of the endoscope while moving, lack of focus due to water droplets attached, more blurred parts due to tilt of the observed object, or almost no observed object due to shooting only distant views. Such endoscopic images for which an inappropriate score may be calculated are not suitable for calculating the raw score. The score processing unit 63 may include a first discriminating unit 83 (refer to Figure 8 ) and the second determination unit 84 (reference Figure 8 ) one or both of the following.

[0098] The first determination unit 83 determines whether the endoscopic image is suitable for raw score calculation based on a feature quantity (second feature quantity) obtained by analyzing the endoscopic image. The feature quantity in this case is preferably a quantity associated with at least one selected from the group consisting of halo distribution, spatial frequency distribution, brightness value distribution, shadow distribution, magnification index, and reflected light distribution of illumination light irradiating the observation object in the endoscopic image.

[0099] In endoscopic images, areas where halos occur or shadows created by the endoscope's lens hood, for example, are extremely bright or dark areas. Therefore, endoscopic images with many of these areas are not suitable for raw score calculation. Endoscopic images with many extremely bright or dark areas due to brightness or shadow distribution are also not suitable for raw score calculation. Endoscopic images with many areas of image blur due to spatial frequency distribution are also not suitable for raw score calculation, as they are jittery or out of focus. Furthermore, referring to the magnification index, at high magnifications, such as when calculating areas with dense superficial blood vessels, the imaging of blood vessels changes compared to unmagnified conditions, and the vascular density per unit area also changes. Therefore, it is best to consider magnification when calculating raw scores; otherwise, raw score calculation is not suitable. Furthermore, when the observed object changes to a different type, such as due to changes in the observation site, the reflected light distribution of the illumination light irradiating the object may change. Due to this change in reflected light distribution, endoscopic images with many extremely bright or dark areas are also not suitable for raw score calculation.

[0100] Furthermore, the values ​​calculated based on the different second feature quantities are used as different types of second feature quantities. The first determination unit 83 may calculate a single second feature quantity or two or more second feature quantities. The first determination unit 83 uses one or more second feature quantities to determine whether the endoscopic image is suitable for calculating a raw score.

[0101] The unsuitable image discriminating unit 74 discriminates whether the endoscopic image is suitable for calculating the raw score in the second discriminating unit 84. The second discriminating unit 84 includes a second machine learning model that has been learned. The learned second machine learning model is generated by inputting past endoscopic images that are associated with whether the image is suitable for calculating the raw score into the machine learning model. That is, the machine learning model is generated by inputting past endoscopic images into the machine learning model and causing it to learn so as to correctly output whether the image is suitable for calculating the raw score. Therefore, learning includes not only inputting past endoscopic images that are associated with whether the image is suitable for calculating the raw score into the machine learning model, but also adjusting various parameters.

[0102] Endoscopic images determined by the unsuitable image determination unit 74 to be suitable for raw score calculation are sent to the raw score calculation unit 72 for raw score calculation. Endoscopic images determined to be unsuitable for raw score calculation have their raw scores considered "not calculated." In other words, since the "not calculated" label is added to the raw scores, they are distinguished from endoscopic images for which raw scores have not yet been calculated.

[0103] In addition, it is preferable that the final score determination unit 73 determines the final score based on the raw scores calculated based on a plurality of endoscopic images acquired during a predetermined period before the time point of determining the final score. Figure 12 The determination of the final score in this case is explained in detail. Figure 12 In FIG, the upper part shows the process from the start of observation with an endoscope, and the lower part shows the state of the endoscopic image obtained. Observation is started with the endoscope, and endoscopic image acquisition 101 starts from the start time S of observation. The image signal acquisition unit 51 automatically performs endoscopic image acquisition 101 at a prescribed image acquisition interval a. Once the endoscopic image acquisition 101 is performed, the raw score of the endoscopic image is calculated. In addition, inappropriate image discrimination can be performed before the calculation. Figure 12 In the figure, endoscopic image acquisition 101 is represented by a filled circle, but only a portion of the circle is labeled to avoid complication. Similarly, only a portion of the image acquisition interval a is labeled to avoid complication. The final score is determined based on endoscopic images acquired during a period Δt before the final score determination time t, which is the time at which the final score is determined. This period Δt is a predetermined period.

[0104] The first endoscopic image acquired during the period Δt is an endoscopic image 121 acquired at time t-Δt. The endoscopic image 121 includes an intramucosal bleeding area 126 and a blood vessel dense area 127. Figure 12In the figure, the intramucosal bleeding area 126 and the densely vascularized area 127 are indicated by diagonal lines and hatching, respectively. However, to avoid complicating the figure, only some symbols are assigned. The endoscopic image 121 obtained by endoscopic image acquisition 101 at time t-Δt is determined by the unsuitable image determination unit 74 during period B to determine whether it is suitable for calculating the raw score. The determination result is obtained at the time of unsuitable image determination 102. The determination result is "suitable."

[0105] After the unsuitable image determination 102, the raw score calculation unit 72 calculates the raw score for the endoscopic image 121 during period C, obtaining the calculation result at the time of raw score calculation 103. The raw score calculation unit calculates raw scores for two types of images: the number of pixels in intramucosal and extramucosal bleeding areas (hereinafter referred to as the bleeding area pixel count) and the number of pixels in areas with dense blood vessels (hereinafter referred to as the dense area pixel count). The calculation results are "bleeding area pixel count: 100" and "dense area pixel count: 70."

[0106] Next, among the endoscopic images acquired during the period Δt, the second raw score is calculated for the endoscopic image 122 acquired at time t-Δt+4a. The endoscopic image 122 includes an intramucosal bleeding area 126, a densely vascularized area 127, and an extramucosal bleeding area 128. Figure 12 In the figure, extramucosal bleeding 128 is represented by a filled circle, but to avoid complication, only some symbols are assigned. The endoscopic image 122 obtained by endoscopic image acquisition 101 at time t-Δt+4a is determined by the unsuitable image determination unit 74 during period B to determine whether it is suitable for raw score calculation. The determination result is obtained at the time of unsuitable image determination 104. The determination result is "suitable."

[0107] After the unsuitable image determination 104, the raw score calculation unit 72 calculates the raw score for the endoscopic image 122 during period C, and obtains the calculation result at the time of raw score calculation 105. The raw score calculation unit calculates the raw scores for the two types of pixels: the number of pixels in the bleeding area and the number of pixels in the dense area, using the first calculation unit 81. The calculation results are "number of pixels in the bleeding area: 120" and "number of pixels in the dense area: 90."

[0108] Next, among the endoscopic images acquired during period Δt, the third raw score is calculated for endoscopic image 123 acquired at time t-Δt+8a. Because endoscopic image 123 was acquired while the endoscope was moving, it is an unsharp image 129 of the observed object, a blurred image, and the observed object cannot be identified. Endoscopic image 123, acquired by endoscopic image acquisition 101 at time t-Δt+8a, is determined by the unsuitable image determination unit 74 during period B to determine whether it is suitable for raw score calculation. The determination result is obtained at the time of unsuitable image determination 106. The determination result is "unsuitable."

[0109] After the unsuitable image determination 106, the raw score calculation unit 72 accepts the unsuitable image determination result as "unsuitable" and does not calculate the raw score. In other words, the raw score calculation result is "not calculated".

[0110] Next, among the endoscopic images acquired during period Δt, the fourth raw score is calculated for endoscopic image 124 acquired at time t-Δt+12a. Endoscopic image 124 includes an intramucosal bleeding area 126, a densely vascularized area 127, and an extramucosal bleeding area 128. The unsuitable image determination unit 74 determines whether endoscopic image 124, acquired during period B, is suitable for raw score calculation. The determination result is obtained at the time of unsuitable image determination 108. The determination result is "suitable."

[0111] After the unsuitable image determination 108, the raw score calculation unit 72 calculates the raw score for the endoscopic image 124 during period C, and obtains the calculation result at the time of raw score calculation 109. The raw score calculation unit calculates the raw scores for the two types of pixels: the number of bleeding area pixels and the number of dense area pixels, using the first calculation unit 81. The calculation results are "number of bleeding area pixels: 140" and "number of dense area pixels: 140."

[0112] Thus, based on the relationship between the image acquisition interval a, the period of inappropriate image determination by the inappropriate image determination unit 74, and the period of raw score calculation by the raw score calculation unit 72, the endoscopic images acquired during the period Δt are four endoscopic images: endoscopic images 121, 122, 123, and 124. The final score determination unit 73 preferably determines the final score based on the raw scores of these endoscopic images, excluding those that are "not calculated." Therefore, the final score determination unit 73 determines the final score based on the raw scores calculated for each of the three endoscopic images: 121, 122, and 124.

[0113] The final score determination unit 73 preferably determines the final score by performing a moving average, FIR (finite impulse response) filtering, or IIR (infinite impulse response) filtering on the multiple raw scores. The moving average is preferably a simple moving average, a weighted moving average, an exponential moving average, or a triangular moving average. When weighting is used in the moving average, a value that yields optimal results can be pre-set and used based on, for example, the observation site or endoscopic image acquisition conditions. By using a moving average, FIR filtering, or IIR filtering, the final score can be determined by minimizing the effects of noise, such as in raw scores with extremely distant values, among the multiple raw scores. This allows for stable final score determination.

[0114] At the final score determination time t, for example, when the final score is calculated by a simple moving average of the raw scores, if the raw score is the number of bleeding area pixels, the number of bleeding area pixels in the endoscopic images 121, 122, and 124 is averaged. That is, as shown in the following formula (1), the first type of final score is 120, which is the average of the number of bleeding area pixels in the endoscopic image 121 (100), the number of bleeding area pixels in the endoscopic image 122 (120), and the number of bleeding area pixels in the endoscopic image 124 (140). Similarly, as shown in the following formula (2), the second type of final score is 100, which is the average of the number of dense area pixels in the endoscopic image 121 (70), the number of dense area pixels in the endoscopic image 122 (90), and the number of dense area pixels in the endoscopic image 124 (140).

[0115] Final score (number of pixels in the bleeding area) = (100 + 120 + 140) / 3 = 120 (1)

[0116] Final score (number of pixels in dense area) = (70 + 90 + 140) / 3 = 100 (2)

[0117] The final score can be determined automatically or at an indicated time. For example, the final score determination time t can be set when the user acquires a still image using the still image acquisition instruction unit 12f (freeze button). In this case, when the still image acquisition instruction unit 12f is given, the display control unit 76 controls the display of the final score and / or its temporal changes. The observation object for which the user stores a still image often includes areas of interest. Determining the final score for such an area and displaying it on the display 18 facilitates a more appropriate diagnosis for the user, and is therefore preferred.

[0118] Next, the display control unit 76 controls the display 18 to display the final score and / or the change in the final score over time in real time. A preferred display control method is one that can stably display the final score determination result. For example, a method in which the display control unit 76 displays the final score with the highest numerical value among the final scores since the start of observation and updates the final score display when a final score greater than the previous numerical value is obtained, or a method in which the change in the final score over time is displayed, can be used.

[0119] When displaying the change in the final score over time, it is preferably displayed in a graphical form. The graphical form is preferably displayed using at least one graph showing the relationship between the final score and the time at which the final score was determined. When displaying the final score in a graphical form, it is preferably to pre-set a threshold for the final score value, and to display the final score on the graph only when the final score exceeds the threshold.

[0120] like Figure 13 As shown, the display control unit 76 controls whether to display the final score according to the threshold value at the final score determination time t described above. When there are two final scores, two graphs are displayed on the display 18. In the final score, the threshold value T1 of the number of pixels in the bleeding part is set to 800, and the threshold value T2 of the number of pixels in the dense part is set to 40. The final score of the number of pixels in the bleeding part at the final score determination time t is 120, which is smaller than the threshold value T1 of 800. Therefore, the number of pixels in the bleeding part at the final score determination time t is not plotted in the graph 131 of the number of pixels in the bleeding part. Moreover, the final score of the number of pixels in the dense part at the final score determination time t is 100, which is larger than the threshold value T2 of 40. Therefore, the number of pixels in the dense part at the final score determination time t is plotted with an auxiliary line 133 in the graph 132 of the number of pixels in the dense part as the final score, so that the numerical value can be grasped at a glance. In addition, in Figure 13 In the graphs 131 and 132 , the threshold value T1 or the threshold value T2 is indicated by a slash.

[0121] By controlling the display control unit 76 to display the final score and / or its temporal changes in real time on the display 18, the user can clearly understand which item is considered to be the most deteriorating, even while continuing to observe the subject. Furthermore, by graphically displaying the temporal changes, the user can also grasp the approximate location of the site considered to be the most deteriorating. Furthermore, by controlling the display of the final score using a threshold, only the items considered to be the most deteriorating are displayed, thereby making the display of the final score more stable.

[0122] Furthermore, as part of controlling the display of final scores using thresholds, when there are two or more final scores, the graph displayed on display 18 can be limited to a single graph. Specifically, for example, if one of the two final scores is above the threshold and the other is below the threshold, only the graph for the final score above the threshold is displayed. By strictly selecting and displaying the types of parameters such as the final score, the reliability and stability of the displayed final score is improved. Furthermore, the user can clearly grasp information useful for diagnosing severity or stage at a glance.

[0123] In addition, a part determination unit 77 (see FIG. 1 ) is provided for determining a part of an observation target included in an endoscopic image by performing image analysis on the endoscopic image. Figure 8 ), the time-varying changes of the final score can also be displayed by a graph showing the relationship between the final score and the time and part of the final score. Thus, the endoscopic image is automatically analyzed and the part name is displayed corresponding to the time-varying changes of the final score. Figure 14 For example, when observing the large intestine, while pulling the endoscope from the cecum located deep in the large intestine to screen, the region names 134 of "cecum, ascending colon, transverse colon, descending colon, sigmoid colon, rectum" are displayed. Displaying the region names 134 in this way allows for accurate understanding of the region where the disease is progressing during observation without omission, which is preferable.

[0124] In addition, the final score determination unit 73 determines the final score based on the raw score calculated immediately before or after the final score determination time point. Figure 15 and Figure 16 The determination of the final score in these cases is described in detail. Figure 15 As shown, in the method for determining the final score based on the raw score calculated immediately before the final score is determined, the final score is determined based on the raw score calculated immediately before the final score is determined, rather than using multiple raw scores. In other words, the raw score calculated immediately before the final score is determined is determined as the final score. The acquisition of endoscopic images and raw scores is the same as described above.

[0125] The raw scores calculated at the time of raw score calculation 109 based on the endoscopic image acquired at time t-Δt+12a, which were calculated immediately before the final score determination time t, are "Number of pixels in the bleeding area: 140" and "Number of pixels in the dense area: 140." Therefore, the final scores are "Number of pixels in the bleeding area: 140, number of pixels in the dense area: 140." The display of the final score using the threshold value based on the display control unit 76 is the same as described above. Therefore, as Figure 15As shown, a graph 132 of the number of dense portion pixels plotted as 140 is displayed on the display 18 as the final score.

[0126] And, as Figure 16 As shown, in the method for determining the final score based on the raw score calculated immediately after the final score is determined, the final score is determined not using multiple raw scores, but rather based on the raw score calculated immediately after the final score determination time t. In other words, the raw score calculated immediately after the final score determination time t is determined as the final score. This is the raw score calculated for the endoscopic image acquired at the final score determination time t. The endoscopic image acquisition and raw score acquisition are the same as described above.

[0127] The raw scores calculated at the time of the raw score calculation 111 based on the endoscopic image acquired at time t and immediately after the final score determination time t are "Number of pixels in the bleeding area: 160" and "Number of pixels in the dense area: 70", and thus the final scores are "Number of pixels in the bleeding area: 160, number of pixels in the dense area: 70". The display of the final score using the threshold value based on the display control unit 76 is the same as described above. Therefore, as Figure 16 As shown, a graph 132 in which the number of pixels in the dense portion is plotted as 70 is displayed on the display 18 as the final score.

[0128] The method of determining the final score based on the raw scores calculated immediately before or after the final score is determined is suitable for obtaining the final score of the currently observed subject and is preferred because it can stably display the latest final score.

[0129] Furthermore, when the user determines the final score at the time of acquiring a still image using the still image acquisition instruction unit 12f, by combining this with a method of determining the final score based on the raw score calculated immediately before or after the final score is determined, the final score based on the observation site for which the user has instructed to acquire a still image, which is the area of ​​interest, can be displayed quickly and stably. Therefore, the final score is preferably displayed at the moment the user desires to obtain information related to the diagnosis, thereby responding to the user's needs.

[0130] Furthermore, the final score determination unit 73 may determine whether to calculate the final score based on the number of raw scores that were not calculated by the inappropriate image determination unit 74 and the number of raw scores that were calculated, among the raw scores based on the plurality of endoscopic images acquired during the predetermined period. Specifically, the final score is set to "not calculated" if, among the raw scores based on the plurality of endoscopic images acquired during the predetermined period, the number of raw scores other than those not calculated is less than a predetermined number, the number of raw scores not calculated is greater than a predetermined number, or the ratio of the number of raw scores not calculated to the number of raw scores based on the plurality of endoscopic images acquired during the predetermined period is greater than a predetermined value.

[0131] use Figure 17 and Figure 18 The determination of the final score in these cases is described in detail. Figure 17 As shown, during a predetermined period Δt, endoscopic image acquisition 101 is automatically performed at image acquisition intervals a, acquiring endoscopic images at time t-Δt, time t-Δt+4a, time t-Δt+8a, and time t-Δt+12a. After acquiring the endoscopic images, the unsuitable image determination unit 74 determines, after period B, whether the endoscopic images acquired at each time are suitable for raw score calculation. Among the multiple endoscopic images, endoscopic images 122 and 123 acquired at time t-Δt+4a and time t-Δt+8a are unsharp images 129. Therefore, the raw score calculation unit 72 sets the raw scores for endoscopic images 122 and 123 to "not calculated," respectively.

[0132] In this embodiment, among the raw scores of a plurality of endoscopic images acquired during a predetermined period, when the number of raw scores other than those not calculated is 2 or less, the final score is set to "not calculated". Therefore, in this case, the plurality of endoscopic images is 4, the raw scores not calculated are 2, and the raw scores other than those not calculated are 2, which is equivalent to "the case where the number of raw scores other than those not calculated is 2 or less". Therefore, the final score at the final score determination time t is set to "not calculated". The display control unit 76 performs control so as not to display the final score not calculated on the display. Therefore, as Figure 18 As shown, in the display 18 , the final score is not plotted in either of the graphs 131 and 132 .

[0133] In addition, Figure 17In the case shown above, when the method of setting the final score to "not calculated" is adopted when the number of raw scores not calculated is a specified number or more, for example, when the specified number is set to 2, the number of raw scores not calculated is 2, and therefore the final score is set to "not calculated." Similarly, when the method of setting the final score to "not calculated" is adopted when the ratio of the number of raw scores not calculated to the number of raw scores based on a plurality of endoscopic images acquired during a specified period is equal to or greater than a specified value, for example, when the specified value is set to 0.5, the number of endoscopic images and raw scores acquired during period Δt is 4, of which the number of raw scores not calculated is 2, and the ratio is 2 / 4 (0.5), so the final score is set to "not calculated."

[0134] As described above, by determining whether to calculate the final score based on the number of raw scores that are not calculated and the number of raw scores that are calculated by the unsuitable image discrimination unit 74, the number or proportion of unpreferable raw scores is suppressed, the final score is appropriately determined, and the display of the score is stable, so it is preferred.

[0135] In addition, the final score can be displayed by a message notifying the observation subject of the result of the determination of the severity or stage of the disease. The severity or stage is determined by the final score. For example, the display control unit 76 uses a threshold value for controlling whether to display the final score. That is, in the final score, the severity or stage is determined by the threshold value T1 of the number of pixels in the bleeding part and the threshold value T2 of the number of pixels in the dense part. With respect to the severity, the case where the number of pixels in the bleeding part is above the threshold value T1 is set as a severe case, the case where the number of pixels in the bleeding part is less than the threshold value T1 and the number of pixels in the dense part is above the threshold value T2 is set as a moderate case, or the case where the number of pixels in the bleeding part is less than the threshold value T1 and the number of pixels in the dense part is less than the threshold value T2 is set as a mild case. With respect to the stage, the severe case where the number of pixels in the bleeding part is above the threshold value T1 and the moderate case where the number of pixels in the bleeding part is less than the threshold value T1 and the number of pixels in the dense part is above the threshold value T2 is set as pathological non-remission, or the mild case where the number of pixels in the bleeding part is less than the threshold value T1 and the number of pixels in the dense part is less than the threshold value T2 is set as pathological remission.

[0136] like Figure 19 As shown, the message notifying the severity or stage determination result is provided by, for example, displaying a message 135 on a portion of display 18. Message 135 may be set to "severe," "moderate," or "mild" for severity, in addition to "non-remission," and "remission" or "non-remission" for stage. Displaying the final score as a message notifying the severity or stage determination result is preferred because it allows the user to obtain information supporting the severity or stage diagnosis at a glance without obstructing the user's view.

[0137] Then, along Figure 20 The flowchart shown here illustrates the series of steps in the score display mode. When the mode is switched to score display mode, special light is irradiated onto the observation object. The endoscope 12 captures the observation object illuminated by the special light (step ST110), acquiring an endoscopic image as a special image at a specific point in time. In this flowchart, the final score is automatically acquired at predetermined intervals. The image acquisition unit 71 acquires the special image from the endoscope 12 (step ST120).

[0138] The special image is sent to the inappropriate image determination unit 74, which determines whether it is suitable for raw score calculation. If the special image is determined to be suitable (Yes in step ST130), a raw score is calculated based on the special image (step ST140). If the special image is determined to be unsuitable (No in step ST130), the raw score is considered "not calculated" (step ST150).

[0139] The final score is determined based on the raw score calculation results (step ST160). Once the final score is determined, the display control unit 76 controls the display of the final score (step ST170). The display is controlled to show the final score (step ST180). If the observation is complete (yes in step ST190), the observation is completed. If the observation is not complete (no in step ST190), the process returns to endoscopic image acquisition.

[0140] Furthermore, while the above-described embodiment applies the present invention to an endoscope system that processes endoscopic images, the present invention can also be applied to a medical image processing system that processes medical images other than endoscopic images. In this case, the medical image processing system includes the image processing device of the present invention. Furthermore, the present invention can also be applied to a diagnostic support device that uses medical images to assist users in diagnosis. Furthermore, using medical images, the present invention can also be applied to a medical service support device that supports medical services such as diagnostic reporting.

[0141] For example, Figure 21 As shown in FIG, the diagnosis support apparatus 201 uses a combination of medical imaging equipment such as a medical image processing system 202 and a PACS (Picture Archiving and Communication Systems) 203. Figure 22As shown, a medical service support device 210 is connected to various examination devices, such as a first medical image processing system 211, a second medical image processing system 212, ..., and an Nth medical image processing system 213, via an arbitrary network 214. The medical service support device 210 receives medical images from the first to Nth medical image processing systems 211, 212, ..., 213 and provides medical service support based on the received medical images.

[0142] In the above-described embodiment, the hardware configuration of the processing units (processing units) included in the processor device 16 that perform various processes, such as the image signal acquisition unit 51, DSP 52, noise reduction unit 53, signal processing unit 55, and video signal generation unit 56, is composed of the various processors described below. These processors include general-purpose processors (CPUs) that execute software (programs) to function as various processing units; processors (PLDs) (programmable logic devices) whose circuit configuration can be modified after manufacturing, such as FPGAs (field programmable gate arrays); and processors (special-purpose circuits) with circuit configurations specifically designed to perform various processes.

[0143] A processing unit can be composed of one of these various processors, or a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a CPU and an FPGA). In addition, multiple processing units can be composed of one processor. As an example of a method of composing multiple processing units by one processor, there is a method in which a processor is composed of a combination of one or more CPUs and software, such as a computer such as a client or server, and the processor functions as multiple processing units. There is a method in which a processor is used to implement the functions of the entire system including multiple processing units by a single IC (Integrated Circuit) chip, such as a system on chip (SoC). In this way, various processing units are composed of one or more of the above-mentioned various processors as a hardware structure.

[0144] Furthermore, more specifically, the hardware structure of these various processors is an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

[0145] The present invention can also be implemented in another embodiment as described below.

[0146] In the processor device,

[0147] Acquire multiple endoscopic images of the observation object at different times by an endoscopic device,

[0148] Calculate a raw score related to the severity or stage of the disease of the observed subject from each endoscopic image,

[0149] The final score is determined based on the raw scores.

[0150] Control is performed to display the final score and / or the change in the final score over time on the display in real time.

[0151] Explanation of symbols

[0152] 10-Endoscope system, 12-Endoscope, 12a-Insertion portion, 12b-Operation portion, 12c-Bending portion, 12d-Front end portion, 12e-Angle button, 12f-Still image acquisition instruction portion, 12g-Mode switching switch, 12h-Zoom operation portion, 14-Light source device, 16-Processor device (Image processing device), 18-Display, 19-Console, 20-Light source portion, 20a-V-LED, 20b-B-LED, 20c-G-LED, 20d-R-LED, 21-Light source control portion, 30a-Illumination Bright optical system, 30b-imaging optical system, 41-light guide, 42-illumination lens, 43-objective lens, 44-zoom lens, 45-imaging sensor, 46-CDS / AGC circuit, 47-A / D converter, 51-image signal acquisition unit, 52-DSP, 53-noise reduction unit, 54-memory, 55-signal processing unit, 56-video signal generation unit, 61-normal image generation unit, 62-special image generation unit, 63-score processing unit, 71-image acquisition unit, 72-raw score calculation unit, 73-final score determination unit, 74 - Unsuitable Image Determination Unit, 75 - Image Storage Unit, 76 - Display Control Unit, 77 - Part Determination Unit, 81 - First Calculation Unit, 82 - Second Calculation Unit, 83 - First Determination Unit, 84 - Second Determination Unit, 85 - Surface Blood Vessels, 86 - Intramucosal Bleeding, 87 - Extramucosal Bleeding, 101 - Endoscopic Image Acquisition, 102, 104, 106, 108, 110, 111 - Unsuitable Image Determination, 103, 105, 107, 109 - Raw Score Calculation, 121-125 - Endoscopic Images, 126 - Intramucosal Bleeding 127-Densely vascularized area, 128-Extramucosal bleeding area, 129-Unclear image, 131, 132-Chart, 133-Auxiliary line, 134-Part name, 135-Message, 201-Diagnosis support device, 202-Medical image processing system, 203-PACS, 210-Medical service support device, 211-1st medical image processing system, 212-2nd medical image processing system, 213-Nth medical image processing system, 214-Network, t-Final score determination time, ST110~ST190-Steps.

Claims

1. An image processing device comprising a processor, The processor performs the following processing: acquiring, through the endoscope, a plurality of endoscopic images obtained by photographing an observation object at different times; calculating a raw score related to determination of the severity or stage of the disease of the observation subject based on each of the endoscopic images; determining a final score based on the raw scores; performing control to display the change of the final score over time on a display in real time, The processor determines, for each of the endoscopic images, whether the endoscopic image is suitable for calculation of the raw score. not calculating the raw score for the endoscopic image determined to be unsuitable for calculating the raw score, The processor determines the final score based on the raw scores calculated based on the plurality of endoscopic images acquired during a specified period before the time point of determining the final score, and does not calculate the final score when the number of raw scores other than those not calculated is less than a specified number, the number of raw scores not calculated is greater than a specified number, or the ratio of the number of raw scores not calculated to the number of raw scores based on the plurality of endoscopic images acquired during the specified period is greater than a specified value.

2. The image processing apparatus according to claim 1, wherein: The processor calculates two or more different raw scores.

3. The image processing apparatus according to claim 1, wherein: The processor calculates the raw score based on a first feature value obtained by analyzing the endoscopic image.

4. The image processing apparatus according to claim 2, wherein: The processor calculates the raw score based on a first feature value obtained by analyzing the endoscopic image.

5. The image processing apparatus according to claim 3, wherein: The first feature value is a value related to a surface blood vessel dense area, an intramucosal bleeding area, or an extramucosal bleeding area included in the endoscopic image. The image processing apparatus according to claim 1 , wherein: The processor executes a learned first machine learning model generated by inputting the past endoscopic image associated with the raw score into a machine learning model, and calculates the raw score based on the endoscopic image.

7. The image processing apparatus according to claim 2, wherein: The processor executes a learned first machine learning model generated by inputting the past endoscopic image associated with the raw score into a machine learning model, and calculates the raw score based on the endoscopic image.

8. The image processing apparatus according to any one of claims 1 to 7, wherein: The processor determines the final score by performing a moving average, FIR filter processing, or IIR filter processing on a plurality of the raw scores.

9. The image processing apparatus according to any one of claims 1 to 7, wherein: The processor determines the final score based on the raw score calculated just before or after a point in time when the final score is determined.

10. The image processing apparatus according to any one of claims 1 to 7, wherein: The processor determines whether the endoscopic image is suitable for calculation of the raw score based on a second feature value of the endoscopic image. The image processing apparatus according to claim 10 , wherein: The second feature value is a value related to at least one selected from the group consisting of halo distribution, spatial frequency distribution, brightness value distribution, shadow distribution, magnification index, and reflected light distribution of illumination light irradiated on the observation object of the endoscopic image.

12. The image processing apparatus according to claim 10, wherein: The processor executes a learned second machine learning model generated by inputting the past endoscopic image that establishes a corresponding association with whether it is suitable for calculating the original score into the machine learning model, and determines whether the endoscopic image is suitable for calculating the original score.

13. The image processing apparatus according to any one of claims 1 to 7, wherein: The processor determines the final score based on the raw score excluding the calculation.

14. The image processing apparatus according to any one of claims 1 to 7, wherein: The temporal change of the final score is displayed by at least one graph showing the relationship between the final score and the time when the final score was determined.

15. The image processing apparatus according to any one of claims 1 to 7, wherein: The processor determines a portion of the observation object included in the endoscopic image by performing image analysis on the endoscopic image. The temporal change of the final score is displayed by at least one graph showing the relationship between the final score, the time when the final score was determined, and the site.

16. The image processing apparatus according to any one of claims 1 to 7, wherein: The processor performs an instruction to acquire a still image, When the instruction is given, control is performed to display the change in the final score over time.

17. The image processing apparatus according to any one of claims 1 to 7, wherein: The disease is ulcerative colitis.

18. An endoscope system comprising an endoscope for capturing an image of an observation object and an image processing device including a processor. The processor acquires a plurality of endoscopic images of the observation object captured at different times, calculates a raw score related to determination of the severity or stage of the disease of the observation object based on each of the endoscopic images, determines a final score based on the raw scores, and controls displaying the final score or a change in the final score over time on a display in real time. The processor determines, for each of the endoscopic images, whether the endoscopic image is suitable for calculation of the raw score. not calculating the raw score for the endoscopic image determined to be unsuitable for calculating the raw score, The processor determines the final score based on the raw scores calculated based on the plurality of endoscopic images acquired during a specified period before the time point of determining the final score, and does not calculate the final score when the number of raw scores other than those not calculated is less than a specified number, the number of raw scores not calculated is greater than a specified number, or the ratio of the number of raw scores not calculated to the number of raw scores based on the plurality of endoscopic images acquired during the specified period is greater than a specified value.

19. An image processing method, comprising: an image acquisition step of acquiring a plurality of endoscopic images obtained by photographing the observation object at different times; a raw score calculation step of calculating a raw score related to determination of severity or stage of the disease of the observation subject based on each of the endoscopic images; a final score determination step, determining a final score based on the raw scores; and a display control step of performing control to display the final score or the change in the final score over time on a display in real time; In the raw score calculation step, for each of the endoscopic images, whether the endoscopic image is suitable for calculation of the raw score is determined, and the raw score is not calculated for the endoscopic image determined to be unsuitable for calculation of the raw score. In the final score determination step, the final score is determined by the raw scores calculated based on the plurality of endoscopic images acquired during a specified period before the time point of determining the final score, and the final score is not calculated when the number of raw scores other than those not calculated is less than a specified number, the number of raw scores not calculated is greater than a specified number, or the ratio of the number of raw scores not calculated to the number of raw scores based on the plurality of endoscopic images acquired during the specified period is greater than a specified value.

Images