Method for generating correction data and device for generating correction data

Abstract

Method for generating correction data, performed by a computer, the method comprising: a data acquisition step for capturing recorded image data by mapping an indicator that is related to a predetermined disease; a graphical representation step to display a real image data point that corresponds to the captured image data in a predetermined color space, which is assigned to the disease according to a color component of the data point; a calculation step for calculating a correction value to correct pixel values ​​that make up an image captured via an electronic endoscope, based on a distance between the image data point and a predetermined target point in the color space; and a storage step to save the correction value, characterized in that, in the acquisition step, first recorded image data are captured by imaging a first indicator which has a first color that is a color of biological tissue, if a symptom stage is a predetermined first stage in relation to the disease, and second recorded image data are captured by imaging a second indicator which has a second color that is a color of biological tissue, if a symptom stage is a predetermined second stage in relation to the disease; In the graphical representation step, the first and second data points, which correspond to the captured first and second recorded image data, are displayed in the color space according to the color components of the first and second data points and In the calculation step, the correction value is calculated based on a distance between the first data point and the predetermined first target point in the color space and a distance between the second data point and the predetermined second target point in the color space.

Description

Technical field

[0001] The invention relates to a method for generating correction data and a device for generating correction data. State of the art

[0002] A lesion site typically has a different color than normal mucosa. In recent years, performance improvements in color endoscopes have made it possible for users to identify and diagnose lesion sites whose color differs only slightly from that of normal tissue. However, extensive training under the guidance of an expert is necessary to be able to accurately distinguish lesion sites from normal tissue and make a diagnosis based on a slight color difference in an image acquired by an electronic endoscope. Even an experienced user may not be able to easily identify and diagnose a lesion site based on a slight color difference, thus requiring careful work.

[0003] In light of this, document JP 2014-18332A (hereinafter referred to as "Patent Document 1") is an example of a document describing a device for evaluating a lesion site visible in a captured image, with the aim of facilitating the diagnosis of the lesion site by a user.In particular, the device described in patent document 1 subjects the pixels representing an image captured by an electronic endoscope to a hue enhancement process to apply non-linear gain to the pixel data, widens the dynamic range near the boundary of an area containing pixel data undergoing lesion site determination, converts the hue-enhanced pixel data from an RGB space defined by the three primary colors RGB into a predetermined color space, such as the HSI or HSV color space, to obtain hue and saturation data, identifies the pixels belonging to the lesion site based on the acquired hue and saturation data, and calculates an evaluation value (lesion site index) based on the number of pixels identified as belonging to the lesion site.

[0004] A method according to the preamble of claim 1 and a device according to the preamble of claim 7 are known from publication JP 2016-15 995 A. Corresponding subsequently published prior art is known from publications DE 11 2016 000 095 B4 and DE 11 2016 004 454 T5. Summary of the invention

[0005] In the pixel data acquired by an electronic endoscope, the RGB values ​​for a white object do not correspond to the pixel values ​​in their actual state. For this reason, the pixel data undergoes white balance correction. In the device described in patent document 1, a lesion index is calculated based on the color data acquired from a captured image, and it is therefore assumed that improving the color representation of a captured image by means of white balance correction leads to an improvement in the calculation accuracy of the lesion index.

[0006] However, if the same lesion site is imaged using different electronic endoscope systems, cases are possible in which significant differences arise in the calculated result of the lesion site index or similar results, even if the lesion site appears visually identical. In light of this, the inventors of the present invention, through thorough and careful examination, have discovered that, primarily due to individual differences in the optical components (optical performance) of the electronic endoscopes, an error (variation) remains in the color values ​​used for calculating the result, which cannot be eliminated by white balance correction, and that this error is a reason for the impairment of precision in the result calculation.

[0007] The present invention was achieved in view of the circumstances described above, and an object of the present invention is to provide a method according to claim 1 for generating correction data and a device according to claim 7 for generating correction data in order to minimize deviations in the result values ​​when the same lesion site is imaged with different endoscope systems.

[0008] A method for generating correction data according to a first aspect of the present invention, carried out by a computer, comprises: a capture step for capturing recorded image data by mapping an indicator associated with a predetermined disease; a graphic representation step for displaying a real image data point corresponding to the captured image data in a predetermined color space, which is assigned to the disease according to a color component of the data point; a calculation step for calculating a correction value for correcting pixel values ​​that constitute an image captured via an electronic endoscope, based on a distance between the image data point and a predetermined target point in the color space; and a storage step for storing the correction value.

[0009] Furthermore, in the method according to the first aspect of the present invention, in the acquisition step, first recorded image data are acquired by mapping a first indicator which has a first color that is a color of biological tissue when a symptom stage corresponds to a predetermined first stage with respect to the disease, and second recorded image data are acquired by mapping a second indicator which has a second color that is a color of biological tissue when a symptom stage corresponds to a predetermined second stage with respect to the disease; in the graphic representation step, first and second data points corresponding to the acquired first and second recorded image data are represented in the color space according to the color components of the first and second data points;and in the calculation step, the correction value is calculated based on a distance between the first data point and the predetermined first target point in the color space and a distance between the second data point and the predetermined second target point in the color space.

[0010] Furthermore, the procedure for generating correction data includes: a step to accept user input to identify an initial symptom level; and a step to inform the user about an indicator corresponding to the assumed symptom level.

[0011] Furthermore, in the method for generating correction data according to the first aspect of the present invention, a matrix coefficient is calculated as a correction value in the calculation step, which achieves a minimum sum value of the distance between the first data point and the first target point and the distance between the second data point and the second target point.

[0012] Furthermore, in the method for generating correction data according to the first aspect of the present invention, the color space is a two-dimensional color space comprising an R-component axis and a G-component axis that is orthogonal to the R-component axis.

[0013] Furthermore, in the method for generating correction data according to the first aspect of the present invention, the first color is a color of a biological tissue when the symptom level with respect to the disease is highest, and the first target point is a point on an axis that has a high correlation with hemoglobin pigments in the color space.

[0014] Furthermore, in the method for generating correction data according to the first aspect of the present invention, the second color is a color of a biological tissue that is healthy with respect to the disease, and the second target point is a point on an axis that has a high correlation with a hue of a mucous membrane at a body height in the color space.

[0015] Furthermore, a device for generating correction data according to a second aspect of the invention comprises: a detection means for capturing recorded image data by mapping an indicator related to a predetermined disease; a graphic representation means for displaying a real image data point corresponding to the captured image data in a predetermined color space, which is assigned to the disease according to a color component of the data point; a calculation means for calculating a correction value for correcting pixel values ​​that constitute an image captured via an electronic endoscope, based on a distance between the image data point and a predetermined target point in the color space; and a storage means for storing the correction value.

[0016] According to the aspects of the present invention, it is possible to provide a method for generating correction data and a device for generating correction data in order to minimize deviations in the result values ​​when the same lesion site is imaged with different endoscope systems. Brief description of the drawings Fig. Figure 1 is a block diagram illustrating an embodiment of an electronic endoscope system according to an embodiment of the present invention. Fig. Figure 2 is a flowchart of a special image generation method that is carried out in a special mode according to an embodiment of the present invention. Fig. Figure 3 shows a diagram depicting an RG plane in which the correspondence pixel points are shown according to an embodiment of the present invention. Fig. Figure 4 shows a diagram that displays reference axes in the RG plane. Fig. Figure 5 shows an example of a screen display that is shown in a monitor display screen in a special mode according to an embodiment of the present invention. Fig. Figure 6 shows a flowchart illustrating the calibration process performed in a calibration mode according to an embodiment of the present invention. Fig. Figure 7 shows a diagram to support the description of the calibration process according to Fig. 6. Description of the exemplary implementations

[0017] In the following, embodiments of the present invention are described with reference to the drawings. It should be noted that an electronic endoscope system (computer) is used as an example of an embodiment of the present invention in the following description. Design of the electronic endoscope system 1

[0018] Fig. Figure 1 shows a block diagram illustrating an embodiment of an electronic endoscope system 1 according to an embodiment of the present invention. As shown in Fig. As shown in Figure 1, the electronic endoscope system 1 is a system specific for medical use and comprises an electronic endoscope 100, a processor 200 and a monitor 300.

[0019] The processor 200 comprises a control unit 202 and a timer 204. The control unit 202 executes various programs stored in the memory 222 and performs overall control of the electronic endoscope system 1. The control unit is also connected to an operator panel 218. The control unit 202 modifies the operation of the electronic endoscope system 1 and the parameters for different operating modes in accordance with instructions entered by a user via the operator panel 218. An example of a user-entered instruction is an instruction to change the operating mode of the electronic endoscope system 1. In the present embodiment, the operating mode includes a normal mode, a special mode, and a calibration mode.The timer 204 outputs a clock pulse which serves to regulate the timing of the operation of parts with circuits in the electronic endoscope system.

[0020] A lamp 208 is activated by a power supply element 206 and subsequently emits white light L. The lamp 208 is a high-intensity lamp, such as a xenon lamp, a halogen lamp, a mercury lamp, or a metal halide lamp. The white light L emitted by the lamp 208 is focused by a converging lens 210 and limited to a suitable amount of light by an aperture 212. The lamp 208 can be replaced by a light-emitting semiconductor element, such as an LD (laser diode) or an LED (light-emitting diode). A light-emitting semiconductor element has characteristics such as lower energy consumption and lower heat emission than other light sources and therefore has the advantage that bright images can be captured while energy consumption and heat emission are minimized.The ability to capture bright images leads to an improvement in the accuracy of the inflammation marker assessment described later. The light-emitting semiconductor element is not limited to being provided in the processor 200 and can also be located in the electronic endoscope 100. For example, the light-emitting semiconductor element can be located in the distal end section of the electronic endoscope 100.

[0021] A motor 214 is mechanically connected to the aperture 212 via a transmission mechanism, such as a (swivel) arm and a gearbox, which are not shown. The motor is, for example, a DC motor and is driven by a drive 216. The aperture 212 is operated by the motor 214, and the opening angle is adjusted to set the images displayed on the screen of a monitor 300 to a suitable brightness. The amount of white light L emitted by the lamp 208 is limited by the opening angle of the aperture 212. The reference value corresponding to the suitable image brightness is set according to an instruction for adjusting the intensity, which the user executes at the control panel 218. The light control circuit for carrying out the intensity adjustment by controlling the drive 216 is a known circuit that is not described in detail here.

[0022] The white light L, which passes through the aperture 212, is focused at the entrance of the end surface of an LCB element 102 (LCB: Light carrying bundle) and enters the LCB element 102.

[0023] After the white light L has passed through the LCB element, it exits the LCB element via an end face of the LCB element, which is positioned at the front end face of the electronic endoscope 100. It passes through a light distribution lens 104 and illuminates the biological tissue. Light returning from the biological tissue illuminated by white light L passes through an objective lens 106 and forms an optical image on the light-receiving surface of a solid-state image sensor 108.

[0024] The solid-state image sensor 108 is a single-plate CCD (charge-coupled device) color image sensor comprising a checkerboard-pattern color filter. The solid-state image sensor 108 accumulates charge according to the amount of light of an optical image formed from pixels on the light-sensing surface. It generates yellow Ye, cyan Cy, green G, and magenta Mg pixel data and outputs the pixel data. The solid-state image sensor is not limited to being a CCD image sensor and can be replaced by a CMOS (complementary metal oxide semiconductor) image sensor or by another image-generating device. The solid-state image sensor 108 can be an element comprising a primary color filter (filter in Bayer configuration).

[0025] A control signal processing circuit 112 is provided in the connection area of ​​the electronic endoscope 100. Pixel data from pixels that have detected biological tissue illuminated by white light L are transmitted via the solid-state image sensor 108 to the control signal processing circuit 112 in a full-frame cycle. The pixel data input from the solid-state image sensor 108 is output by the control signal processing circuit 112 to a signal processing circuit 220 of the processor 200. The terms "full frame" and "half-frame" can be used interchangeably in the following description. In the present embodiment, the full-frame cycle and the half-frame cycle are 1 / 30 of a second and 1 / 60 of a second, respectively.

[0026] The control signal processing circuit 112 accesses a memory 114 and reads out specific information relating to the electronic endoscope 100. This specific information, stored in memory 114, includes, for example, the number of pixels, the sensitivity, the operable frame rate, and the model number of the solid-state image sensor 108. The control signal processing circuit 112 outputs this specific information from memory 114 to a system controller 202.

[0027] The control unit 202 generates control signals for performing various calculations based on specific information relating to the electronic microscope. The control unit 202 uses these generated control signals to manage the operation and timing of various circuits within the processor 200, in order to perform controls suitable for the endoscope connected to the processor 200.

[0028] A timing controller 204 supplies a clock signal to the control signal processing circuit 112 according to a timing operation performed by the system controller 202. In accordance with the clock signal supplied by the timing controller, the control signal processing circuit 112 controls the drive of the solid-state image sensor according to a timing specification that is synchronized with the frame rate of the images processed by the processor 200. Operation in normal mode

[0029] The following describes signal processing processes in the 200 processor during normal operating mode.

[0030] The signal processing circuit 220, which includes the processor 200, has a pre-processing circuit 220A, a processing circuit 220B, an output circuit 220C, a correction circuit 220D, a result circuit 220E and a mapping circuit 220F.

[0031] The pre-processing circuit 220A performs demosaicing on the RAW pixel data received from the control signal processing circuit 112 in the full-frame cycle, it performs conversions to RGB pixel data, it performs color matrix processing, as well as white balance corrections, hue enhancement corrections and the like, and transmits the resulting pixel data to the processing circuit 220B.

[0032] The processing circuit 220B performs enhancement processing, gamma corrections and similar operations on pixel data received from the pre-processing circuit 220A, generates normal color image data and transmits the image data to the output circuit 220C.

[0033] Output circuit 220C performs enhancement processing, such as Y / C separation and color difference correction, on the color image data received from processing circuit 220B and converts it into a predetermined video format signal. The converted video format signal is output to monitor 300. Accordingly, normal color image data of the biological tissue is displayed on the monitor 300's screen. Operation in special mode

[0034] The following describes signal processing operations in the 200 processor in its special operating mode.

[0035] Fig. Figure 2 shows a flowchart of a special image generation procedure that is executed in the special mode. The special image generation and processing in Fig. 2 starts at the time when the operating mode of the electronic endoscope system is changed to the special mode. S11 in FIG. 2 (Input of the pixel data of the current image)

[0036] In this processing step S11, pixel data for each pixel of the current image is received from the pre-processing circuit 220A. The pixel data of each pixel undergoes signal processing by the pre-processing circuit 220A and is then transferred to the processing circuit 220B and the correction circuit 220D. S12 in FIG. 2 (representation in the RG plane)

[0037] Fig. Figure 3 is a diagram that conceptually depicts the processes in the 220D correction circuit and shows an RG plane (two-dimensional color space) defined by an R-axis and a G-axis, which are perpendicular to each other. The R-axis represents the R-component (R-pixel values), and the G-axis represents the G-component (G-pixel values).

[0038] In processing step S12, pixel data for each pixel in RGB space, defined by the three primary colors RGB (three-dimensional pixel data composed of three types of color components), is converted (vertically projecting transformation) into RG pixel data (two-dimensional pixel data composed of two types of color components). As conceptually described in Fig. As shown in Figure 3, the pixel data for each pixel from the RGB color space is projected into the RG plane as a function of the R and G pixel values. For clarity, the points corresponding to the pixels projected into the RG plane are referred to as correspondence pixel points. It should be noted that the correspondence pixel points in Fig. 3 are only shown for some and not all pixel points.

[0039] In this way, in processing step S12, pixel data from the RGB space (three-dimensional data) are projected perpendicularly onto the RG plane, and the corresponding pixel points (two-dimensional data) are the base of vertical lines drawn from the points in the RGB space that correspond to the pixel data onto the RG plane. S13 in FIG. 2 (Defining the reference axis)

[0040] In this processing step S13, a reference axis is placed in the RG plane using the correction circuit 220D, which is necessary for calculating the degree of inflammation of a predetermined disease, such as gastritis. Fig. Figure 4 shows a diagram to support the description of the reference axis.

[0041] Due to influences such as the hemoglobin pigment, the red component (R) is dominant over the other components (green and B components) in the patient's body cavity being imaged. The more intense the inflammation, the more pronounced the redness (R) typically becomes relative to the other hues (green and B components). However, the hue in images acquired within body cavities changes depending on imaging conditions that affect brightness (e.g., the level of white light illumination). For example, shadowed areas not reached by the white light appear black (achromatic, with R, G, and B component values ​​close to zero), while areas where the white light is intensely incident and specularly reflected appear white (achromatic, with R, G, and B component values ​​at or near 255).In other words, even if the same inflamed abnormal area is imaged, the pixel value in the image of the abnormal area will be higher the more intense the white light is applied to it. For this reason, the pixel value, depending on the white light illuminance (L), can take on a value that has no correlation with the degree of inflammation.

[0042] Generally, normal, non-inflamed areas within a body cavity are adequately covered with mucous membrane. In contrast, abnormal, inflamed areas within a body cavity are not adequately covered with mucous membrane. The higher the degree of inflammation at an abnormal site, such as a lesion, the thinner the mucous membrane becomes. Mucous membranes are generally white but have a slightly yellowish tint, and this tint (yellow tint) appears in an image and varies in darkness / lightness (mucous membrane thickness). Therefore, the darkness / lightness of the mucous membrane is also intended to serve as an indicator for assessing the degree of inflammation.

[0043] In light of this, in processing step S13, as in Fig. Figure 4 shows a straight line passing through (50,0) and (255, 76) in the RG plane being set as a reference axis, and a straight line passing through (0,0) and (255, 192) is also set as a reference axis. For clarity, the first axis is called the "hemoglobin variation axis AX1" and the second axis is called the "mucosal variation axis AX2".

[0044] The in Fig. Figure 4 shows the result obtained by the inventor of the present invention through the analysis of a large number of example images of body cavities. The example images used in the analysis included images of various levels of inflammation, encompassing images of inflammation at the highest symptom level (examples of images of inflammation at the most severe level) and examples of inflammation at the lowest symptom level (examples of images of areas considered essentially normal). For the clarity of the diagram, only a portion of the points obtained as results of the analysis are shown in the example of the Fig. Figure 4. The number of points actually obtained as results of the analysis is far higher than the number shown. Fig. 4 points shown.

[0045] As described above, the R-component is higher relative to the other components (G-component and B-component) the higher the degree of inflammation is at an abnormal site. For this reason, an axis is drawn at the boundary line that divides regions where points are distributed and undistributed, and which is closer to the R-axis than to the G-axis, and which is the boundary line that is defined in Fig. The axis passing through (50,0) and (255,76) is defined as the axis that has a high correlation with a lesion site corresponding to the highest symptom level (an inflamed (abnormal) site with the highest symptom level). This axis is the hemoglobin variation axis AX1. Points corresponding to inflamed sites with the highest symptom level, acquired under different image acquisition conditions (e.g., white light illumination L), lie on the hemoglobin variation axis AX1.

[0046] On the other hand, the closer a point is to a normal point, the more intense the G-component (or B-component) is relative to the R-component. For this reason, there is an axis on the boundary line that divides regions where points are distributed and undistributed, and which is closer to the G-axis than to the R-axis, and which is the boundary line that is in Fig. The axis passing through (0,0) and (255, 192) is defined as the axis that has a high correlation with a lesion site corresponding to the lowest symptom level (an inflamed (abnormal) site with the lowest symptom level, considered to be essentially normal (healthy)). This axis is the mucosal variation axis AX2. Points corresponding to inflamed sites with the lowest symptom level (considered to be essentially normal) and acquired under different image acquisition conditions (e.g., white light illumination L) lie on the mucosal variation axis AX2.

[0047] Furthermore, the inflamed area with the highest symptom level is accompanied by bleeding. Conversely, an inflamed area with the lowest symptom level is essentially normal and therefore sufficiently covered with mucous membrane. For this reason, it can be understood that the in Fig. The four points shown in the RG plane are distributed in a region enclosed between the axis with the highest correlation with blood (hemoglobin pigment) and the axis with the highest correlation with mucosal hue. Therefore, of the two boundary lines separating regions in which points are distributed and undistributed, the one closer to the R-axis (higher R-component) corresponds to the axis representing an inflamed area with the highest symptom level (hemoglobin variation axis), and the boundary line closer to the G-axis (higher G-component) corresponds to the axis representing an inflamed area with the lowest systemic level (mucosal variation axis). S14 in FIG. 2 (Selection of pixels of interest)

[0048] In processing step S14, a pixel-of-interest is selected from all pixels by the correction circuit 220D, according to a predetermined sequence. S15 in FIG. 2 (Correction of pixel data)

[0049] The correction circuit 220D stores a correction matrix coefficient calculated in the calibration mode described later. In processing step S15, the pixel data (R,G) of the pixel of interest selected in processing step S14 (pixel of interest selection) are corrected by the correction circuit 220D using the correction matrix coefficient to minimize variations in the results when the same lesion site is imaged with different endoscope systems (in other words, individual differences between electronic endoscopes). The correction matrix coefficient is described in detail in the section describing operation in calibration mode. • Examples of the correction matrix (RnewGnew)=(M11M12M21M22)(RG) R neu Corrected pixel data of the pixel-of-interest (R component) G neu Corrected pixel data of the pixel-of-interest (G component) M 11 - M 22 Cooreal matrix coefficients R Uncorrected pixel data of the pixel-of-interest (R component) G Uncorrected pixel data of the pixel of interest (G component) S16 in FIG. 2 (angle calculation)

[0050] After the processing steps S14 (selection of the pixel-of-interest) and S15 (correction of the pixel data) have been carried out by the correction circuit 220D on all pixels of the current image, the result circuit 220E calculates an angle to calculate the degree of inflammation for the pixel data (R). neu , G neu) for each pixel obtained during the correction in processing step S15 (Correction of Pixel Data). In particular, this processing step S16 concerns the processing to calculate an angle θ for each pixel formed by the hemoglobin variation axis AX1 and a line segment LS, which represents the correspondence pixel point (R). neu , G neu ) and an intersection point (reference point) O' of the hemoglobin axis of variation AX1 with the mucosal axis of variation AX2 (see Fig. 3) connects. The reference point O' has the coordinates (-150, -75). S17 in FIG. 2 (normalization processing)

[0051] While the brightness of an image of a body cavity varies depending on the level of white light (L), the hue of the image is influenced by individual differences such as the location depicted, the state of inflammation, etc. However, the hue of an inflamed area with the highest symptom level in the retinal plane changes approximately along the hemoglobin variation axis AX1, and the hue of an inflamed area with the lowest symptom level changes approximately along the mucosal variation axis AX2. It is also concluded that the hue of an image taken at a location with a moderate symptom level changes with the same tendency.Specifically, when a pixel correspondence point, corresponding to an inflamed area, changes depending on the illuminance with white light L, a shift occurs in the direction of the azimuth angle, with the reference point O' serving as the origin. In other words, when a pixel correspondence point, corresponding to an inflamed area, changes depending on the illuminance with white light L, the distance to the reference point O' changes, while the angle θ remains constant. This means that the angle θ is a parameter that is essentially unaffected by changes in the brightness of the captured image.

[0052] The lower the angle θ, the more intense the R component is relative to the G component, meaning that the symptom level of the inflamed area is higher. Conversely, the higher the angle θ, the more intense the G component is relative to the R component, meaning that the symptom level of the inflamed area is lower.

[0053] Based on this, in processing step S17, the result circuit 220E normalizes the angle θ for each pixel of the current image, so that a value of 255 is taken when the angle θ is zero and a value of 0 is taken when the angle θ is zero. MAX is. θ MAX This corresponds to the angle formed by the hemoglobin axis of variation AX1 and the mucosal axis of variation AX2. Accordingly, an inflammation grade is calculated, falling within the range of 0 to 255 (8-bit information). S18 in FIG. 2 (Calculation of an inflammation assessment value)

[0054] In this processing step S18, the result circuit 220E calculates the average value of the inflammation level of all pixels of the current image (or calculates an integrated value of the inflammation level of all pixels) as the overall inflammation score of the captured image and generates a data display for the calculated inflammation score (example of a data display: Result: OO) S19 in FIG. 2 (Determining the display color in the color chart image)

[0055] In the present embodiment, it is possible to display a color map image created by mosaicking a captured image into display colors corresponding to the degree of inflammation. To enable the display of a color map image, a table of correspondences between inflammation degree values ​​and predetermined display colors is stored in a memory region of the result circuit 220E. In this table, for example, a display color is assigned to a group of five values. For instance, yellow is assigned to an inflammation degree range with values ​​from 0 to 5, various display colors are each assigned to groups of five higher values ​​based on the color order in the color wheel, and red is assigned to the value range 250 to 255.

[0056] In processing step S19, the display color in the color map image for each pixel of the current image is determined by the imaging circuit 220F in such a way that, based on the table described above, it corresponds to the color corresponding to the inflammation level value obtained in processing step S17 (normalization processing). S20 in FIG. 2 (Generation of color map image data)

[0057] In processing step S20, the imaging circuit 220F converts the color data of each pixel of the current image into data for the display colors that were determined in processing step S19, and generates color map image data formed from pixels to be displayed in the converted display colors. S21 in FIG. 2 (overlay processing)

[0058] In processing step S21, output circuit 220C sets a coefficient as the degree of superposition of a normal color image based on normal color image data received from processing circuit 220B and a color map image based on color map image data generated in processing step S20, and adds the original image data (normal color image data) and the latter image data (color map image data) based on the coefficient.

[0059] The coefficient setting can be changed via a user command. For example, if it is desired to display the normal color image more strongly, the coefficient for the normal color image data can be increased, and if it is desired to display the color map image more strongly, the coefficient for the color map image data can be increased. S22 in FIG. 2 (final determination)

[0060] In processing step S22, it is determined whether the operating mode of the electrical endoscope system 1 has been changed to a mode other than the specific mode. If it is determined that the operating mode has not been changed to another mode (S22: NO), the special image generation processing procedure is terminated. Fig. 2 returns to step S11 (input of the pixel data of the current image). However, if it is detected that the operating mode has been changed to a different mode (S22: YES), the special image generation processing ends. Fig. 2. Example of a screen display

[0061] The output circuit 220C generates display data for a superimposed image, which contains the normal color image and the color chart image, based on the image data that was processed during the addition processing in processing step S21 (superimposition processing) in Fig. The output circuit 220C performs masking processing to mask the peripheral areas of the monitor 300's screen display (periphery of the image display areas) and further generates screen display data in which the inflammation assessment value is superimposed on the mask area generated during the masking processing. The output circuit 220C converts the generated screen display data into a predetermined video format signal and transmits the signal to the monitor 300.

[0062] Fig. Figure 5 shows an example of a screen display in special mode. As in Fig. As shown in Figure 5, the screen display of the monitor 300 comprises the intracavitary captured image (superimposed image in which the normal image and the color map image are superimposed) in the central area, and a masked screen area surrounding the image display area.

[0063] The screen display mode of the captured image is not limited to a superimposed image of a standard color image and a color chart image. For example, there are display modes in which the standard color image and the color chart image are shown side-by-side on the same screen, and others in which only the color chart image is displayed. In the first case, the standard color image and the color chart image can be displayed at the same size; alternatively, it is possible to display either the standard color image or the color chart image as the main image, with the other displayed as a sub-image that is smaller than the main image.

[0064] In this way, according to the present invention, there is no need to perform complex color space transformation processes or non-linear computational processes such as color enhancement or the like, and an inflammation score (here, a value that correlates with the increase / decrease of the hemoglobin pigment at the imaged site) is calculated only by performing simple computational operations. In other words, the amount of hardware resources required to calculate the inflammation score is significantly reduced. Furthermore, the inflammation score does not vary substantially depending on the imaging conditions that affect the brightness of the intracavitary images (e.g., the degree of illumination with irradiation light), and therefore the user can make a more objective and accurate diagnosis regarding inflammation. Operation in calibration mode

[0065] The operation of the electronic endoscope system 1 in calibration mode is described below. Fig. Figure 6 shows a flowchart of the calibration processing performed in calibration mode. Furthermore, it shows Fig. 7. A diagram to support the description of the calibration processing in Fig. 6. The calibration process according to Fig. 6 is executed, for example, at the time of factory delivery, and begins when the user switches the mode of the electronic endoscope system 1 to calibration mode.

[0066] Preparatory steps can be carried out by a clerk before the calibration process. In particular, the clerk can perform the white balance of the electronic endoscope system 1 using a gray card or similar.

[0067] Once the white balance correction has been performed, the clerk places the electronic endoscope system 1 into a calibration device and starts calibration software on a terminal (PC) connected to the processor 200.

[0068] The technician then makes brightness adjustments to an image captured by the electronic endoscope 100. For example, the technician manually adjusts the aperture of the 212 aperture so that the brightness value of the captured image, achieved by focusing on a subject for brightness adjustment, matches a target brightness value. The brightness value of the captured image can be checked using the calibration software.

[0069] Once the brightness adjustment of the captured image is complete, the operator positions a first indicator within the viewing angle of the electronic system 100 by inserting the first indicator into the calibration device. Here, the first indicator is a color indicator, which corresponds to the color of biological tissue when the symptom level is at its highest with respect to a predetermined disease. In the present embodiment, the first indicator is a plate-shaped component colored in a shade corresponding to a predetermined point (the first target point P, described below). T1 ) on the hemoglobin variation axis AX1 in the RG plane. S31 in FIG. 6 (illustration of the first indicator)

[0070] In processing step S31, the first indicator (surface, colored in the first color) is recorded by the electronic endoscope 100 based on operating input from the clerk, and the resulting recorded image (RAW data format, YUV format or similar) is entered into the PC. S32 in FIG. 6 (illustration of the second indicator)

[0071] The operator then positions a second indicator firmly within the field of view of the electronic system 100 by inserting the second indicator into the calibration device. Here, the indicator is a second-color indicator, which is the color of biological tissue that is healthy with respect to a predetermined disease. In the present embodiment, the first indicator is a plate-shaped part colored in a color corresponding to a predetermined point (the first target point P, described later). T2) on the mucosal variation axis AX2 in the RG plane.

[0072] In processing step S32, the second indicator (surface, colored in the second shade) is imaged by the electronic endoscope 100 based on operator input, and the resulting image (RAW data format, YUV format, etc.) is entered into the PC. Using the calibration device, the operator can image the first and second indicators under identical conditions. By performing processing step S31 (image of the first indicator) and this processing step S32, image data is obtained by imaging indicators related to a predetermined disease.

[0073] S33 in Fig. 6 (Calculation of the first real image data point P) D1) The execution of processing step S33 begins as a result of processing input from the clerk or automatically after a predetermined number of images have been taken (here: two images).

[0074] In processing step S33, the calibration software calculates a first real image data point P. D1 as the real measurement point of the first indicator based on the recorded image data for the first indicator, which was recorded in processing step S31 (image of the first indicator). For example, the average value of the pixels (e.g., 200x200 pixels) in the central area of ​​the recording of the first indicator is considered the first real data point P. D1 calculated. S34 in FIG. 6 (calculation of the second real data point P) D2 )

[0075] In processing step S34, the calibration software calculates a second real image data point P. D2as the real measurement point of the second indicator based on the recorded image data for the second indicator, which was recorded in processing step S32 (image of the second indicator). For example, in the same way as for the first real data point, the average value of the pixels (e.g., 200x200 pixels) in the central area of ​​the recording of the second indicator is considered the second real data point P. D2 calculated. S35 in FIG. 6 (plotting of real imaging data on RG plane)

[0076] In processing step S35, the calibration software sets the first real data point P D1 and the second real data point P D2 in the RG level, which is related to a target disease (e.g. gastritis), as shown here in Fig. Figure 7 is shown. During processing step S35, the real image data points are represented in a predetermined color space, which is assigned to a predetermined disease according to the color components of the real image data points. S36 in FIG. 6 (calculation of the correction matrix coefficient)

[0077] As in Fig. As shown in 7, a first target point P is T1 , which corresponds to the first real image data point P D1 corresponds to being set on the hemoglobin variation axis AX1, and a second target point P T2 , which corresponds to the second real image data point P D2 The corresponding value is located on the mucosal axis of variation AX2. The first target point P T1 corresponds to the first color of the first indicator, and the second target point P T2 corresponds to the second color of the second indicator.

[0078] In processing step S36, the calibration software uses the least squares method or a similar method to calculate the correction matrix coefficient that provides the smallest sum of the distance between the first real data point P. D1 and the first destination P T1 (first distance Δ1) and the distance between the second real data point P D1 and the second destination P T2 (second distance Δ2) is calculated. In the execution of processing step S36, a correction value is calculated to correct the values ​​of the pixels that make up an image captured by an electronic endoscope, based on the distance between real data points and predetermined target data points in a color space.

[0079] Furthermore, the correction matrix coefficient can be calculated by applying the following expression: • Calculation example of the correction matrix coefficient (M11M12M21M22)=(REF11REF12REF21REF22)(MEA11MEA12MEA21MEA22)−1 M 11 - M 22 Correction matrix coefficient REF 11 , REF 21 First destination P T1 REF 12 , REF 22 Second destination P T2 MEA 11 , MEA 21 First real data point P D1 MEA 12 , MEA 22 Second real data point P D2

[0080] As shown below, the above expression is derived by transforming an expression for correction to correction targets (target points) by applying a correction matrix to measured values ​​(real data points). (REF11REF12REF21REF22)=(M11M12M21M22)(MEA11MEA12MEA21MEA22) (REF11REF12REF21REF22)(MEA11MEA12MEA21MEA22)−1=(M11M12M21M22)(MEA11MEA12MEA21MEA22)(MEA11MEA12MEA21MEA22)−1 (REF11REF12REF21REF22)(MEA11MEA12MEA21MEA22)−1=(M11M12M21M22) S37 in FIG. 6 (saving of correction matrix coefficient)

[0081] In processing step S37, the correction matrix coefficient calculated in processing step S36 (calculation of the correction matrix coefficient) is stored in the correction circuit 220D of processor 200. Accordingly, the Fig. The calibration process shown in section 6 is complete.

[0082] When performing calibration procedures on electronic endoscope systems, approximately the same values ​​are achieved (values ​​that approximate the first target point and the second target point in each of the electronic endoscope systems) when the first indicator and the second indicator, which are associated with a target disease, are mapped by the electronic endoscope systems, and for this reason the inflammation assessment values ​​that are ultimately calculated also take on approximately the same values.

[0083] In other words, according to the present embodiment, by limiting the correction target (in particular, by defining predetermined colors as correction targets related to a predetermined disease), it is advantageously possible to eliminate errors that remain in color data used when calculating the assessment score for the target disease (mainly variations arising from individual differences between the optical components of electronic endoscopes 100). For this reason, the accuracy of the assessment score is increased.

[0084] Furthermore, the electronic endoscope system according to the present embodiment achieves effects and problem solutions such as those described below in applicable technical fields.

[0085] Firstly, the electronic endoscope system according to the present embodiment is a diagnostic tool for the early detection of an inflammatory disease.

[0086] Secondly, according to the design of the present embodiment, it is possible to display the degree of inflammation on a screen or to enlarge the image in an area affected by the inflammation, so that the user can detect mild inflammations that are difficult to see. In particular, mild inflammation is difficult to distinguish from a normal area, and for this reason the effects on the assessment of mild inflammations achieved by the design of the present embodiment are considerable.

[0087] Thirdly, according to the design of the present embodiment, it is possible to provide the user with an objective assessment value in the form of an evaluation of the degree of inflammation, thus making it possible to reduce differences in diagnosis among users. In particular, a significant advantage arises from the ability to provide a user with little experience with an objective assessment value, which is achieved through the design of the present embodiment.

[0088] Fourthly, according to the design of the present embodiment, the image processing load is reduced, making it possible to perform real-time displays of images of an inflamed area.

[0089] Fifthly, according to the design of the present embodiment, the processing load of the calculation of the assessment value is reduced, making it possible to display a color map image (an image showing the degree of inflammation) and a normal-color image side by side or in a combined manner without delay. Therefore, it is possible to display a color map image without increasing the inspection time, thus avoiding any increase in patient stress.

[0090] The site to be examined in the present embodiment is, for example, a respiratory organ or similar, or a digestive organ or similar. "Respiratory organ or similar" includes, for example, the lungs, ears, nose, and throat. "Digestive organ or similar" includes, for example, the large intestine, small intestine, stomach, duodenum, and uterus. It is assumed that the electronic endoscope according to the present embodiment has even more remarkable effects when the target of examination is the large intestine. Specific reasons for this are given below.

[0091] The colon is susceptible to diseases that can be assessed using inflammation as a reference point, and the benefit of detecting inflamed areas is greater than in other organs. In particular, the inflammation score achieved in the present embodiment is effective as an indicator of inflammatory bowel disease (IBD), exemplified by ulcerative colitis. No cure for ulcerative colitis is currently known, so the use of the electronic endoscope system in the configuration of the present embodiment is highly effective for early diagnosis and delaying disease progression.

[0092] The large intestine is a narrower and longer organ than the stomach or similar structures, and the resulting images have greater depth and are darker with increasing depth. According to the design of the present embodiment, it is possible to minimize deviations in the results caused by changes in image brightness. When the electronic endoscope system according to the present embodiment is applied to the examination of the large intestine, the effects of the present embodiment are correspondingly significant. In other words, the electronic endoscope system according to the present embodiment is preferably an electronic endoscope system intended for respiratory organs or an electronic endoscope system intended for digestive organs, and it is particularly an electronic endoscope system intended for the large intestine.

[0093] Although mild inflammations are generally difficult to diagnose, the present embodiment makes it possible to avoid situations where the user overlooks a mild inflammation, for example, by displaying the inflammation assessment score on the screen. In particular, the criteria for diagnosing mild inflammation are not clear, and this is a factor that leads to a large number of individual differences among users. In this context, too, the embodiment makes it possible to provide the user with an objective assessment score, thus reducing the variation in diagnoses caused by individual differences.

[0094] The embodiment described above is applicable to the calculation of a rating value that relates not only to the degree of inflammation, but also to cancer, polyps, and various other lesions accompanied by a color change, and advantageous effects similar to those described above can also be achieved in these other cases. In other words, the rating value of the embodiment is preferably a rating value for a lesion accompanied by a color change, and it includes a rating value for at least one inflammation, cancer, or polyp.

[0095] An exemplary embodiment of the present invention has been described above. The embodiments of the present invention are not limited to the embodiment described above, and various modifications can be made without departing from the scope of the technical idea of ​​the present invention. For example, suitable combinations of embodiments and the like are explicitly given as examples in this description, and obvious embodiments and the like are included in the embodiments of the present invention.

[0096] In the embodiment described above, an operator selects an indicator with a first color (the color of biological tissue when the symptom level is highest with respect to a predetermined disease) as the first indicator, and selects an indicator with a second color (the color of biological tissue that is healthy with respect to a predetermined disease) as the second indicator. Therefore, in the embodiment described above, the calibration is performed with higher precision the closer the colors are to the first and second colors (e.g., correction targets) in the color space. In other words, the further the colors are from the correction targets in the color space (e.g., a color unlikely to be present in inflammation, such as light blue), the lower the calibration accuracy.

[0097] Accordingly, it is sufficient for the case worker to select, as an indicator to be placed in the calibration device, an indicator that corresponds to the symptom level that can be detected with particularly high precision when using the electronic endoscope system 1. For example, if it is desired to detect mild inflammation with high precision, the user must select as the first indicator an indicator in the color that the biological tissue has when mild inflammation is present.

[0098] The more detailed the indicators provided, the more difficult it is for a user to select a suitable indicator. For this reason, Control Panel 202 can implement a control mechanism so that the indicator corresponding to a specific symptom level is displayed on the monitor 300 screen, or communicated via audio playback (e.g., the user is informed about the indicator) when a user command to determine a symptom level is entered via a peripheral device (such as a keyboard). This allows the user to select a suitable indicator from several options.

[0099] Furthermore, in the above embodiment, the inflammation assessment score is calculated using the R-component and the G-component (RG two-dimensional color space) of each pixel. However, in another embodiment, it is possible to use an RB two-dimensional color space or a three-dimensional color space such as HIS, HSV, or Lab instead of the RG two-dimensional color space to calculate a score related to a target disease associated with that color space, which differs from the above embodiment (e.g., gastric atrophy or large gastric tumors). In this case, the correction matrix coefficient is calculated using indicators and target points that differ from those of the above embodiment.

[0100] Different types of correction matrix coefficients, corresponding to different types of target diseases, can be stored in the 200D correction circuit of the 200 processor. By changing the correction matrix coefficient according to the disease to be diagnosed, it is possible to perform a stable (with minor variation due to individual differences) scoring calculation for the corresponding disease.

Claims

[1] A method for generating correction data carried out by a computer, the method comprising: an acquisition step of acquiring captured image data by imaging an indicator related to a predetermined disease; a graphical display step for displaying a real image data point corresponding to the acquired captured image data in a predetermined color space associated with the disease according to a color component of the data point; a calculation step for calculating a correction value for correcting pixel values ​​constituting an image captured via an electronic endoscope based on a distance between the image data point and a predetermined target point in the color space; and a storage step for saving the correction value, characterized byin that in the acquiring step, first captured image data is acquired by imaging a first indicator having a first color that is a color of a biological tissue when a symptom level is a predetermined first level with respect to the disease, and second captured image data is acquired by imaging a second indicator having a second color that is a color of a biological tissue when a symptom level is a predetermined second level with respect to the disease; in the graphical display step, first and second data points corresponding to the acquired first and second recorded image data are displayed in the color space according to color components of the first and second data points, and in the calculation step, the correction value is calculated based on a distance between the first data point and the predetermined first target point in the color space and a distance between the second data point and the predetermined second target point in the color space. [2] A method for generating correction data according to claim 1, further comprising: a step for accepting a user input to identify a first symptom level; and a step to inform the user about an indicator corresponding to the assumed symptom level. [3] A method for generating correction data according to claim 1 or 2, wherein in the calculating step, a matrix coefficient is calculated as a correction value which achieves a smallest sum value of the distance between the first data point and the first target point and the distance between the second data point and the second target point. [4] A method for generating correction data according to any one of claims 1 to 3, wherein the color space is a two-dimensional color space including an R component axis and a G component axis orthogonal to the R component axis. [5] The method for generating correction data according to one of claims 2 to 4, where the first color is a color of a biological tissue when the symptom level is highest in relation to the disease, and where the first target point is a point on an axis that has a high correlation with hemoglobin pigments in color space. [6] Method for generating correction data according to one of claims 2 to 5, where the second color is a color of a biological tissue that is healthy with respect to the disease and wherein the second target point is a point on an axis having a high correlation with a hue of a mucous membrane in a body cavity in the color space. [7] Device for generating correction data comprising: a detecting means for detecting captured image data by imaging an indicator related to a predetermined disease; a graphical display means for displaying a real image data point corresponding to the acquired captured image data in a predetermined color space associated with the disease according to a color component of the data point; a calculation means for calculating a correction value for correcting pixel values ​​constituting an image captured via an electronic endoscope based on a distance between the image data point and a predetermined target point in the color space; and a storage means for storing the correction value, characterized by that the detection means detects first captured image data by imaging a first indicator having a first color that is a color of a biological tissue when a symptom level corresponds to a predetermined first level with respect to the disease, and detects second captured image data by imaging a second indicator having a second color that is a color of a biological tissue when a symptom level corresponds to a predetermined second level with respect to the disease; the graphical representation means represents first and second data points corresponding to the acquired first and second recorded image data in the color space according to color components of the first and second data points, and the calculating means calculates the correction value based on a distance between the first data point and the predetermined first target point in the color space and a distance between the second data point and the predetermined second target point in the color space. [8] The correction data generating apparatus according to claim 7, further comprising: a means of accepting a user input to indicate a symptom level and a means of informing the user of an indicator corresponding to the perceived symptom level. [9] The correction data generating apparatus according to claim 7 or 8, wherein the calculating means calculates, as the correction value, a matrix coefficient that achieves a smallest sum value of the distance between the first data point and the first target point and the distance between the second data point and the second target point. [10] The correction data generating apparatus according to any one of claims 7 to 9, wherein the color space is a two-dimensional color space including an R component axis and a G component axis orthogonal to the R component axis. [11] The correction data generating apparatus according to any one of claims 8 to 10, wherein the first target point is a point on an axis having a high correlation with hemoglobin pigments in the color space. [12] The correction data generating apparatus according to any one of claims 8 to 11, wherein the second target point is a point on an axis having a high correlation with a color tone of a mucous membrane in a body cavity in the color space.

Images