Image processing device, image processing method, and program

Abstract

To make it easier to observe the changes in potential lesions over time. [Solution] In the image processing apparatus 10, the processing unit 11 acquires a first representative brightness value, which is a representative brightness value of the area surrounding the lesion candidate in the first image in which the lesion candidate was captured, and acquires a second representative brightness value, which is a representative brightness value of the area surrounding the lesion candidate in the second image in which the lesion candidate was captured at a different time than the first image. Based on the first representative brightness value and the second representative brightness value, the processing unit 11 corrects the brightness in the area containing the lesion candidate in the second image so that the second representative brightness value in the corrected second image is equal to or approaches the first representative brightness value.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] A technique for observing the body of a subject using a captured image of the subject and diagnosing the subject is known. For example, Patent Document 1 discloses an apparatus for evaluating skin diseases using image data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique for diagnosing a subject as described above, in order to observe the temporal change of a lesion candidate to be observed, there are cases where it is desired to compare and observe lesion candidates captured at different times. However, when the shooting times are different, it is difficult to completely match the shooting conditions such as the surrounding brightness and color tone at the time of shooting. Therefore, it is difficult to compare in detail the lesion candidates captured at different times, and there is a problem that it hinders the detailed observation of the temporal change of the lesion candidate itself.

[0005] The present invention is for solving the above problems, and an object thereof is to provide an image processing apparatus, an image processing method, and a program capable of facilitating the observation of the temporal change of a lesion candidate.

Means for Solving the Problems

[0006] To achieve the above objective, a first aspect of the image processing apparatus according to the present invention includes a processing unit that acquires a first representative luminance value, which is a representative value of the luminance in the area surrounding the lesion candidate in a first image in which the lesion candidate is captured; acquires a second representative luminance value, which is a representative value of the luminance in the area surrounding the lesion candidate in a second image in which the lesion candidate is captured at a different time than the first image; and corrects the luminance in the area including the lesion candidate in the second image based on the first representative luminance value and the second representative luminance value, so that the second representative luminance value in the corrected second image becomes equal to or approaches the first representative luminance value.

[0007] To achieve the above objective, a second aspect of the image processing apparatus according to the present invention includes a processing unit that acquires a first color representative value, which is a representative value of the color in the area surrounding the lesion candidate in a first image in which the lesion candidate is captured; acquires a second color representative value, which is a representative value of the color in the area surrounding the lesion candidate in a second image in which the lesion candidate is captured at a different time than the first image; and corrects the color in the area including the lesion candidate in the second image based on the first color representative value and the second color representative value, so that the second color representative value in the corrected second image becomes equal to or approaches the first color representative value. [Effects of the Invention]

[0008] According to the present invention, it is possible to easily observe the changes in candidate lesions over time. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating the schematic of the image processing system according to Embodiment 1. [Figure 2] This is a block diagram showing the configuration of the image processing apparatus according to Embodiment 1. [Figure 3] (a) and (b) are diagrams showing examples of new and past images according to Embodiment 1, respectively. [Figure 4] (a) and (b) are diagrams showing examples of target regions extracted from a new image and a past image according to Embodiment 1, respectively. [Figure 5] This figure shows an example of a histogram of luminance in the target region according to Embodiment 1. [Figure 6] This figure shows an example of how lesion candidates are displayed on the image processing device according to Embodiment 1. [Figure 7] This flowchart shows the processing flow performed by the image processing device according to Embodiment 1. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals. The image processing system 1 according to Embodiment 1 is a medical support system for diagnosing candidate lesions present in the body of a subject U based on captured images of the subject U. In particular, the image processing system 1 is a system for capturing candidate lesions of the subject U multiple times at different times and comparing and observing the multiple images obtained to confirm whether or not there are changes in the candidate lesions over time. As shown in Figure 1, the image processing system 1 comprises an imaging device 5 and an image processing device 10.

[0011] The imaging device 5 is a device that acquires an image of the subject U by photographing the subject U using light of an appropriate wavelength such as visible light, infrared light, ultraviolet light, etc. The imaging device 5 includes, although not shown in the figures, a lens that focuses incident light, an image sensor that receives the light focused by the lens, and a readout circuit that reads out the light received by the image sensor. The image sensor is equipped with an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) and generates an image of the subject U. The readout circuit is equipped with an A / D (Analog / Digital) converter and converts the analog signal representing the image captured by the image sensor into digital data and outputs it to the image processing device 10.

[0012] The images captured by the imaging device 5 are medical images used for medical purposes and are used to diagnose potential lesions present in the body of subject U. More specifically, the imaging device 5 captures the same subject U multiple times at different times, obtaining multiple images of the same potential lesion present in the body of subject U at different times. These multiple images are used to diagnose the changes in the potential lesion over time, in other words, to diagnose how the potential lesion has changed over time.

[0013] The image processing device 10 is a device operated by a user, such as a personal computer or tablet terminal. Here, the user is a diagnostician, such as a doctor or medical professional, who diagnoses potential lesions. The image processing device 10 performs image processing on the captured image taken by the imaging device 5 of potential lesions present on the body of subject U. As shown in Figure 2, the image processing device 10 comprises a processing unit 11, a storage unit 12, an operation unit 13, a display unit 14, and a communication unit 15.

[0014] The processing unit 11 comprises a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). The CPU is a central processing unit equipped with a microprocessor, etc., that performs various processes and calculations. In the processing unit 11, the CPU reads the control program stored in the ROM and controls the operation of the entire image processing device 10 while using the RAM as work memory. The processing of the processing unit 11 may be performed by a single CPU or by multiple CPUs. Furthermore, the processing unit 11 may also be equipped with image processing processors such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit).

[0015] The memory unit 12 is a non-volatile memory such as flash memory or a hard disk. The memory unit 12 stores programs and data executed by the processing unit 11, and data generated by the processing unit 11. The operation unit 13 is equipped with input devices such as a keyboard, mouse, or touch panel, and accepts user input. The display unit 14 is equipped with a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display, and displays various images under the control of the processing unit 11. The communication unit 15 is equipped with a communication interface for communicating with external devices of the image processing device 10. For example, the communication unit 15 communicates with external devices such as the imaging device 5 in accordance with well-known communication standards such as LAN (Local Area Network) or USB (Universal Serial Bus).

[0016] The processing unit 11 functionally comprises an image acquisition unit 111, a pre-processing unit 112, a cropping unit 113, a correction unit 114, and an image output unit 115. In the processing unit 11, the CPU functions as each of these units by reading a program stored in ROM into RAM and executing that program to control it. In the processing unit 11, one CPU may function as each of the units, or multiple CPUs may jointly function as each of the units.

[0017] The image acquisition unit 111 acquires a captured image of the subject U captured by the imaging device 5. By capturing the subject U, the imaging device 5 acquires, as captured images, for example, a new image Ic shown in FIG. 3(a) and a past image Ip shown in FIG. 3(b). The new image Ic is a captured image of the subject U captured at the first time. In contrast, the past image Ip is a captured image of the same subject U captured at the second time, which is a time before the first time. Here, the time difference between the first time and the second time is an appropriate length of time difference required for observing the temporal change of the lesion candidate, such as several days, several weeks, several months, several years, etc. The new image Ic and the past image Ip are images of the body of the same subject U captured, but since they are captured at different times, the imaging conditions such as the posture of the subject U at the time of capture, the angle of view of the imaging device 5, and the ambient brightness are different.

[0018] More specifically, the new image Ic and the past image Ip shown in FIGS. 3(a) and 3(b) are images captured from the back of a wide range of the skin (muscle) of the upper body including the neck, shoulders, arms, etc. of the subject U. In the new image Ic and the past image Ip, the region where the skin of the body of the subject U is captured is referred to as the subject region A1, and the region other than the subject region A1 is referred to as the background region A0. In the new image Ic and the past image Ip shown in FIGS. 3(a) and 3(b), four lesion candidates B1 to B4 are captured within the subject region A1. Here, a lesion candidate means a location where a pathological change may occur in the body of the subject U, in other words, a location where there is a possibility of some disease occurring in the body of the subject U. That there is a possibility of a pathological change occurring means that as long as there is such a possibility, the pathological change may actually occur, or even if the detailed diagnosis result shows that there is actually no pathological change. In the following, the region where the lesion candidate in the captured image (new image Ic or past image Ip) is captured may also be simply referred to as "lesion candidate". The image acquisition unit 111 communicates with the imaging device 5 via the communication unit 15 to acquire from the imaging device 5 the new image Ic and the past image Ip in which the lesion candidates B1 to B4 existing in the body of the same subject U are captured at different times.

[0019] Returning to FIG. 2, the preprocessing unit 112 performs preprocessing on the new image Ic and the past image Ip acquired by the image acquisition unit 111. Here, the preprocessing is a process executed prior to the correction process so that the correction unit 114 described later can appropriately execute the correction process. The preprocessing unit 112 first identifies a subject region A1 in which the skin of the subject U, which is the body of the subject, is photographed from each of the new image Ic and the past image Ip. Specifically, the preprocessing unit 112 analyzes the pixel values of each pixel included in each of the new image Ic and the past image Ip, and based on physical features such as the color of the skin and the shape of the body part, identifies the subject region A1 from each of the new image Ic and the past image Ip.

[0020] After identifying the subject region A1, next, the preprocessing unit 112 detects lesion candidates from the identified subject region A1 in each of the new image Ic and the past image Ip. In other words, the preprocessing unit 112 detects a portion in which there may be a pathological change in the body of the subject U from the subject region A1 in which the skin of the subject U is photographed. To detect lesion candidates, the preprocessing unit 112 can use a known method related to image identification. Generally, since the luminance of the lesion candidates is relatively low and the luminance in the regions other than the lesion candidates is relatively high, the preprocessing unit 112 detects a region in the subject region A1 where the pixel value is relatively small compared to the surroundings, that is, a relatively dark region, as a lesion candidate. Specifically, in the examples of the new image Ic and the past image Ip shown in FIGS. 3(a) and (b), the preprocessing unit 112 detects four lesion candidates B1 to B4 from each of the new image Ic and the past image Ip.

[0021] When lesion candidates B1 to B4 are detected, the pre-processing unit 112 performs positional alignment (matching) of the lesion candidates based on the positions of lesion candidates B1 to B4 in the new image Ic and the positions of lesion candidates B1 to B4 in the past image Ip. Specifically, even if the same body part of the same subject U is photographed, the posture of the subject U at the time of shooting, the field of view of the shooting device 5, etc. may differ if the shooting times are different. Therefore, the positions of lesion candidates B1 to B4 in the new image Ic and the positions of lesion candidates B1 to B4 in the past image Ip do not perfectly match, and a positional shift occurs. To correct such a positional shift, the pre-processing unit 112 performs geometric deformation (geometric correction) on at least one of the images, new image Ic or past image Ip, based on the relative positional relationship between lesion candidates B1 to B4 captured in new image Ic and lesion candidates B1 to B4 captured in past image Ip.

[0022] The preprocessing unit 112 may use any method for aligning such lesion candidates. For example, the preprocessing unit 112 can use the non-rigid deformation TPS-RPM (Thin Plate Spline Robust Point Matching) algorithm. In this way, the preprocessing unit 112 performs non-rigid deformation on at least one of the new image Ic and the past image Ip so that the same lesion candidate is in the same position in both the new image Ic and the past image Ip. In this way, the preprocessing unit 112 aligns lesion candidates B1 to B4 between the new image Ic and the past image Ip and establishes correspondences between identical lesion candidates.

[0023] Returning to Figure 2, the cropping unit 113 crops the target region to be observed from the new image Ic and past image Ip, which have been pre-processed by the pre-processing unit 112. Here, the target region is a portion of the new image Ic and past image Ip that includes a candidate lesion that the user wishes to observe in detail. Specifically, the cropping unit 113 displays at least one of the images from the new image Ic and past image Ip, which have been pre-processed by the pre-processing unit 112, on the display unit 14. While viewing the display unit 14, the user operates the operation unit 13 to select a candidate lesion that they wish to observe in detail from among the candidate lesions B1 to B4 captured in the new image Ic and past image Ip. Based on this user operation, the cropping unit 113 selects a candidate lesion to be observed from among the candidate lesions B1 to B4. Then, the cropping unit 113 crops the target region, which is the region containing the selected candidate lesion, from each of the new image Ic and past image Ip.

[0024] In the following explanation, we will use the case where lesion candidate B1 is selected by the user from lesion candidates B1 to B4 as an example, but the same explanation can be applied when other lesion candidates are selected. When lesion candidate B1 is selected, the cutting unit 113 cuts out the target region Sc, which is the region containing the selected lesion candidate B1, from the new image Ic, as shown in Figure 4(a). Furthermore, as shown in Figure 4(b), the cutting unit 113 cuts out the target region Sp, which is the region containing the selected lesion candidate B1, from the past image Ip. Here, the target regions Sc and Sp are rectangular regions that include the region of the selected lesion candidate B1 and its surrounding region. When lesion candidate B1 is selected, the cutting unit 113 sets rectangular regions in the X and Y directions, respectively, with a size of several times to about 10 times the size of lesion candidate B1, based on the position of lesion candidate B1 (e.g., the centroid position), in both the new image Ic and the past image Ip. Then, the cutting unit 113 cuts out the set regions as the target regions Sc and Sp.

[0025] Furthermore, the pre-processing described above ensures that identical lesion candidates are associated between the new image Ic and the past image Ip. Therefore, even if the user selects a lesion candidate captured in one of the images, the cropping unit 113 can identify the same lesion candidate from among the lesion candidates B1 to B4 captured in the other image and crop the target region containing the identified lesion candidate.

[0026] Returning to Figure 2, the correction unit 114 corrects the brightness in at least one of the new image Ic and past image Ip based on the brightness in the target regions Sc and Sp extracted by the cropping unit 113. Specifically, the pre-processing by the pre-processing unit 112 described above corrects the positional shift of lesion candidates B1 to B4 between the new image Ic and past image Ip, thus correcting differences in the posture of the subject U and the field of view of the imaging device 5 when the new image Ic and past image Ip were taken. On the other hand, since the new image Ic and past image Ip were taken at different times, the shooting conditions such as the brightness around the imaging device 5 at the time of shooting do not perfectly match, and pre-processing cannot correct for such differences in shooting conditions. Different shooting conditions hinder detailed comparison and observation of the new image Ic and past image Ip. To avoid this and make it easier to compare and observe the new image Ic and past image Ip, the correction unit 114 corrects the brightness in at least one of the new image Ic and past image Ip. Here, the correction unit 114 may correct either the new image Ic or the past image Ip, but the following explanation will use the case where the past image Ip is corrected as an example. The new image Ic, whose brightness is not corrected by the correction unit 114, corresponds to the first image, and the past image Ip, whose brightness is corrected by the correction unit 114, corresponds to the second image.

[0027] To correct the brightness and color in the past image Ip, the correction unit 114 first converts the pixel values ​​of each pixel within the target regions Sc and Sp extracted by the extraction unit 113 into brightness and color components. Here, the brightness component is the component that indicates the degree of brightness in the image, and the color component is the component that indicates the hue in the image. For example, in the Lab color space (L*a*b* color space), the L component (L* component) corresponds to the brightness component, and the a component (a* component) and b component (b* component), which indicate chromaticity (hue and saturation), correspond to the color components. As another example, in the YUV color space, the Y component corresponds to the brightness component, and the U component and V component, which represent color difference, correspond to the color components. Below, we will explain using the case where the pixel values ​​of each pixel in the new image Ic and the past image Ip are converted to the L component, a component, and b component in the Lab color space as an example. However, the same explanation can be given using the YUV color space instead of the Lab color space.

[0028] For example, if the pixel values ​​of each pixel in the new image Ic and the past image Ip are represented in the RGB (Red, Green, Blue) color model, the correction unit 114 converts the pixel values ​​of each pixel into L, a, and b components according to a known conversion formula between the RGB color model and the Lab color space. Alternatively, if the pixel values ​​of each pixel in the new image Ic and the past image Ip are represented in the CMYK (Cyan, Magenta, Yellow, Keyplate) color model, the correction unit 114 converts the pixel values ​​of each pixel into L, a, and b components according to a known conversion formula between the CMYK color model and the Lab color space. As a result, the correction unit 114 converts the pixel values ​​of each pixel contained in the target area Sc,Sp into luminance and hue components. In the RGB color model or CMYK color model, luminance and hue components are distributed across multiple components. By converting the pixel values ​​into luminance and hue components in this way, it becomes easier to correct for differences in ambient brightness during shooting.

[0029] Next, the correction unit 114 obtains a representative value of brightness in the peripheral region of lesion candidate B1 in the new image Ic, and a representative value of brightness in the peripheral region of lesion candidate B1 in the past image Ip. Here, the peripheral region of lesion candidate B1 corresponds to the region other than lesion candidate B1 within the target regions Sc and Sp. The correction unit 114 obtains representative values, which are representative values ​​of brightness for multiple pixels in the region other than lesion candidate B1, from each of the target regions Sc and Sp cut out by the cutting unit 113. Specifically, the correction unit 114 obtains the average value Yc of brightness in the peripheral region, which is the region other than lesion candidate B1 within the target region Sc, as the representative value of brightness in the new image Ic. The correction unit 114 also obtains the average value Yp of brightness in the peripheral region, which is the region other than lesion candidate B1 within the target region Sp, as the representative value of brightness in the past image Ip. The average value Yc is an example of a first representative brightness value, and the average value Yp is an example of a second representative brightness value.

[0030] More specifically, the correction unit 114 generates a frequency distribution of the brightness of multiple pixels contained within each of the target regions Sc and Sp extracted by the cropping unit 113, i.e., a histogram. As an example, the correction unit 114 generates a brightness histogram shown in Figure 5 from the brightness of each pixel contained within the target region Sc. In the histogram shown in Figure 5, the horizontal axis represents the brightness value, and the vertical axis represents the number of pixels. Generally, the brightness of candidate lesions is relatively low, and the brightness of areas other than candidate lesions is relatively high, so the brightness histogram for the target region Sc shows two peaks. The peak on the lower brightness side corresponds to the brightness of candidate lesion B1 within the target region Sc, and the peak on the higher brightness side corresponds to the brightness of the surrounding areas other than candidate lesion B1 within the target region Sc.

[0031] The correction unit 114 sets a threshold TH based on this luminance histogram. The correction unit 114 then determines that luminances below the threshold TH correspond to the luminance in the area of ​​lesion candidate B1, and luminances above the threshold TH correspond to the luminance in the surrounding area of ​​lesion candidate B1. Here, the correction unit 114 can use known methods to set the threshold TH. For example, the correction unit 114 calculates the threshold TH using Otsu's binarization method. Specifically, the correction unit 114 calculates the threshold TH that minimizes the variation in luminance within each group and maximizes the variation in luminance between groups when the luminance of each pixel contained within the target area Sc is divided into two groups by the threshold TH. As a result, the correction unit 114 sets the luminance corresponding to the trough of the two peaks as the threshold TH, for example, as shown in Figure 5. After calculating the threshold TH in this way, the correction unit 114 calculates the average value Yc of luminances above the threshold TH within the target area Sc as a representative value of the luminance in the surrounding area of ​​lesion candidate B1.

[0032] Furthermore, the correction unit 114 performs the same processing on the target area Sp as it does on the target area Sc to calculate the average value Yp. Specifically, the correction unit 114 sets a threshold TH based on the brightness histogram within the target area Sp, and calculates the average value Yp of brightness levels above the threshold TH within the target area Sp as a representative value of brightness in the area surrounding the candidate lesion B1.

[0033] When the average values ​​Yc and Yp are obtained, the correction unit 114 sets a correction value for the brightness in the past image Ip, which is the target of correction, based on the average values ​​Yc and Yp. Specifically, the correction unit 114 calculates a ratio p (=Yc / Yp) which is the value obtained by dividing the average value Yc obtained from the new image Ic by the average value Yp obtained from the past image Ip, which is the target of correction, and sets the calculated ratio p as the correction value.

[0034] When a correction value is set, the correction unit 114 uses the set correction value to correct the brightness of at least a portion of the past image Ip that includes the candidate lesion B1. In other words, the correction unit 114 corrects for differences in imaging conditions between the new image Ic and the past image Ip in order to make it easier to compare and observe the candidate lesion B1 captured in the new image Ic and the past image Ip, which were taken at different times. Specifically, the correction unit 114 multiplies the brightness of each pixel in the target region Sp extracted from the past image Ip by the correction value ratio p (=Yc / Yp). If the brightness value of a certain pixel in the target region Sp before correction is represented as L, the correction unit 114 calculates the brightness value L' of that pixel in the corrected target region Sp as "L'=L×p". When the correction unit 114 corrects the brightness of each pixel in the target region Sp, it performs a process of multiplying the brightness value by a ratio p for each of the multiple pixels included in the target region Sp, thereby uniformly multiplying the brightness of each pixel by p.

[0035] To explain in more detail, in the examples of Figures 4(a) and (b), the overall brightness in the subject area A1 is lower in the target area Sc than in the target area Sp. This corresponds to the case where the surrounding environment at the time of acquisition was darker in the new image Ic than in the past image Ip. In this case, the average brightness Yc obtained from the new image Ic will be smaller than the average brightness Yp obtained from the past image Ip, so the ratio p will be a value less than 1. Therefore, the correction unit 114 reduces the overall brightness of each pixel in the lesion candidate B1 and its surrounding area within the target area Sp.

[0036] Specifically, the correction unit 114 corrects the image of the target region Sp shown on the left side of the lower panel of Figure 4(b) to the image shown on the right side of the lower panel of Figure 4(b). As a result, the average brightness Yp in the peripheral region of lesion candidate B1 within the corrected target region Sp becomes equal to the average brightness Yc in the peripheral region of lesion candidate B1 within the target region Sc shown in Figure 4(a). In other words, the overall brightness in the corrected target region Sp becomes equal to the overall brightness in the uncorrected target region Sc, thus correcting the difference in imaging conditions related to brightness between target regions Sc and Sp.

[0037] In contrast, although not shown in the diagram, if the average value Yc is greater than the average value Yp, the ratio p will be greater than 1. Therefore, the correction unit 114 increases the overall brightness of each pixel in the lesion candidate B1 and its surrounding area within the target area Sp. In this way, the correction unit 114 corrects the brightness of each pixel within the target area Sp so that the average value Yp in the corrected target area Sp is equal to the average value Yc in the target area Sc. This corrects for differences in imaging conditions related to brightness, making it easier for diagnosticians to compare lesion candidate B1 images taken at different times. The correction unit 114 may, if necessary, convert the image of the target area Sp after brightness correction back to the RGB color model or CMYK color model, which are the pixel value formats before conversion.

[0038] Returning to Figure 2, the image output unit 115 outputs an image of the candidate lesion B1 captured in the new image Ic, and an image of the candidate lesion B1 captured in the past image Ip, after correction by the correction unit 114 has been applied. The image output unit 115 displays the image of the target region Sc extracted from the new image Ic and the image of the target region Sp extracted from the past image Ip and after the brightness has been corrected by the correction unit 114 on the display unit 14, for example, as shown in Figure 6. At this time, the image output unit 115 displays the two images side by side on the display screen so that the diagnostician can easily compare and observe the two images. The image output unit 115 may also output these images to an external device via the communication unit 15 and display them on the display unit of the external device.

[0039] In this display screen, the brightness of each pixel in the target area Sp is corrected by the correction unit 114. Therefore, diagnosticians such as doctors and medical professionals can compare and observe lesion candidate B1 captured in new image Ic and past image Ip, which were taken at different times, under equivalent shooting conditions. This allows for detailed diagnosis of changes in lesion candidate B1 over time, leading to a highly accurate diagnosis.

[0040] Next, with reference to Figure 7, the processing flow performed by the image processing device 10 will be described. The processing shown in Figure 7 is performed at appropriate timings for a user, such as a doctor, to diagnose potential lesions in the subject U. The processing shown in Figure 7 is an example of an image processing method. First, the processing unit 11 functions as an image acquisition unit 111 and acquires a new image Ic and a past image Ip, which are captured images of potential lesions in the subject U to be diagnosed (step S1). Specifically, the processing unit 11 communicates with the imaging device 5 and acquires the new image Ic, which has just been captured by the imaging device 5, and the past image Ip, which was captured at a time prior to that, from the imaging device 5. If the new image Ic and past image Ip are already stored in the storage unit 12, the processing unit 11 may read the new image Ic and past image Ip from the storage unit 12.

[0041] When a new image Ic and a past image Ip are acquired, the processing unit 11 functions as a pre-processing unit 112 and performs pre-processing on the acquired new image Ic and past image Ip (step S2). Specifically, the processing unit 11 identifies subject area A1 from both the new image Ic and the past image Ip, and detects lesion candidates from among the identified subject area A1. Then, the processing unit 11 aligns the lesion candidates between the new image Ic and the past image Ip.

[0042] After preprocessing is performed, the processing unit 11 selects a candidate lesion to be observed from among multiple candidate lesions captured in the new image Ic and the past image Ip, respectively, according to the user's operation (step S3). Then, the processing unit 11 functions as an extraction unit 113 and extracts target regions Sc and Sp, which include the selected candidate lesion, from the preprocessed new image Ic and past image Ip, for example, as shown in Figures 4(a) and (b) (step S4).

[0043] Once the target regions Sc and Sp are extracted, the processing unit 11 functions as a correction unit 114 until step S9, converting the pixel values ​​of each pixel in each target region Sc and Sp into brightness and color (step S5). Then, the processing unit 11 obtains representative brightness values ​​in the surrounding areas of the lesion candidate in each target region Sc and Sp (step S6). Specifically, for each of the target regions Sc and Sp extracted in step S4, the processing unit 11 generates a brightness histogram as shown in Figure 5, and sets a threshold TH based on the histogram. Then, the processing unit 11 obtains the average value Yc of brightness above the threshold TH within the target region Sc, and the average value Yp of brightness above the threshold TH within the target region Sp, as representative brightness values.

[0044] Once a representative value of luminance is obtained, the processing unit 11 sets a correction value based on the obtained representative value (step S7). Specifically, the processing unit 11 calculates a ratio p between the average value Yc and the average value Yp as the correction value. Then, the processing unit 11 corrects the luminance of the target region Sp extracted from the past image Ip to be corrected using the set correction value (step S8). Specifically, the processing unit 11 multiplies the luminance of each pixel within the target region Sp by the ratio p calculated in step S7. This corrects the luminance of each pixel within the target region Sp overall, as shown in the lower part of Figure 4(b), for example.

[0045] When brightness is corrected, the processing unit 11 functions as an image output unit 115 and displays the image of the lesion candidate captured in the new image Ic and the image of the lesion candidate captured in the past image Ip after correction by the correction unit 114 has been applied on the display screen of the display unit 14 (step S9). The processing unit 11 displays the image of the target region Sc and the image of the target region Sp after brightness correction by the correction unit 114 on the display unit 14, for example, as shown in Figure 6. This allows the diagnostician to compare and observe lesion candidates captured in two images taken at different times under equivalent shooting conditions.

[0046] As described above, the image processing device 10 according to Embodiment 1 acquires an average value Yc, which is a representative value of the brightness in the area surrounding the lesion candidate, in a new image Ic in which the lesion candidate was captured, and acquires an average value Yp, which is a representative value of the brightness in the area surrounding the lesion candidate, in a past image Ip in which the lesion candidate was captured at a time earlier than the new image Ic. Using correction values ​​based on the average values ​​Yc and Yp, the image processing device 10 corrects the brightness in the target area Sp within the past image Ip so that the average value Yp in the corrected past image Ip is equal to the average value Yc. In this way, the image processing device 10 according to Embodiment 1 corrects the brightness in the past image Ip so that the representative value of the brightness is equal between the new image Ic and the past image Ip, and can correct for differences in shooting conditions between two images taken at different times, in other words, differences in elements other than the lesion candidate. As a result, the diagnostician is less likely to be misled by elements other than the lesion candidate and can focus more on changes in the lesion candidate itself. As a result, diagnosticians can more easily compare and observe candidate lesions captured in new images Ic and past images Ip, and can more easily observe changes in candidate lesions over time.

[0047] Next, Embodiment 2 will be described. Descriptions of the same configuration and functions as in Embodiment 1 will be omitted as appropriate. In Embodiment 1, the correction unit 114 corrected the brightness of each pixel in the target region Sp so that the average value Yp in the corrected target region Sp was equal to the average value Yc in the target region Sc. In contrast, in Embodiment 2, the correction unit 114 corrects the color tone in the target region Sp in the same way as the brightness correction described in Embodiment 1, either in lieu of or in addition to the brightness correction. Here, color tone corresponds to the a and b components in the Lab color space and to the U and V components in the YUV color space. The correction process for color tone can be similarly described in the brightness correction process described above by replacing "brightness" with "color tone".

[0048] Specifically, in Embodiment 2, the correction unit 114 converts the pixel values ​​of each pixel within the target regions Sc and Sp extracted by the cropping unit 113 into luminance and color components. The correction unit 114 then obtains a first color representative value, which is a representative value of the color in the area surrounding the lesion candidate B1 in the new image Ic, and a second color representative value, which is a representative value of the color in the area surrounding the lesion candidate B1 in the past image Ip. To this end, the correction unit 114 generates a color histogram of multiple pixels contained within each of the target regions Sc and Sp extracted by the cropping unit 113. The correction unit 114 then sets a threshold TH based on the generated color histogram. Once the threshold TH is set, the correction unit 114 calculates the average value Yc of the color corresponding to the area surrounding the lesion candidate B1, among the color values ​​above the threshold TH and the color values ​​below the threshold TH in the target region Sc, as the first color representative value. Similarly, the correction unit 114 calculates the average value Yp of the color that corresponds to the area surrounding the lesion candidate B1, among the color values ​​above the threshold TH and the color values ​​below the threshold TH in the target area Sp, as the second color representative value.

[0049] When the average values ​​Yc and Yp of the color tone are obtained, the correction unit 114 sets a color tone correction value for the past image Ip to be corrected based on the average values ​​Yc and Yp. Specifically, the correction unit 114 calculates a ratio p (=Yc / Yp) which is obtained by dividing the average value Yc obtained from the new image Ic by the average value Yp obtained from the past image Ip to be corrected, and sets the calculated ratio p as the color tone correction value. Once the correction value is set, the correction unit 114 uses the set correction value to correct the color tone of at least a portion of the past image Ip to be corrected, including the lesion candidate. Specifically, the correction unit 114 multiplies the color tone of each pixel in the target region Sp extracted from the past image Ip by the correction value ratio p (=Yc / Yp). As a result, the correction unit 114 corrects the color of each pixel in the target region Sp within the past image Ip so that the average value Yp in the past image Ip after color correction approaches the average value Yc.

[0050] The correction unit 114 performs this color correction processing on the a component and b component, which correspond to the color in the Lab color space. Furthermore, if the YUV color space is used instead of the Lab color space, the correction unit 114 performs this color correction processing on the U component and V component, respectively, in the YUV color space. Thus, in Embodiment 2, instead of or in addition to correcting the brightness of each pixel in the target area Sp, the color of each pixel in the target area Sp is corrected so that the representative value of the color is the same between the new image Ic and the past image Ip. This makes it possible to correct the differences in shooting conditions related to color between the new image Ic and the past image Ip. By correcting the differences in shooting conditions related to color as well as shooting conditions related to brightness in this way, it becomes easier for the diagnostician to compare lesion candidates photographed at different times, and thus easier to diagnose changes in lesion candidates over time.

[0051] Next, Embodiment 3 will be described. Descriptions of the same configuration and functions as in Embodiment 1 will be omitted as appropriate. In Embodiments 1 and 2, the correction unit 114 set a ratio p (=Yc / Yp), which is the value obtained by dividing the average value Yc by the average value Yp, as the correction value. The correction unit 114 then corrected the brightness or color in the past image Ip by multiplying the brightness or color value of each pixel within the target area Sp by the ratio p. In contrast, in Embodiment 3, the correction unit 114 sets a difference d (=Yc-Yp), which is the value obtained by subtracting the average value Yp from the average value Yc, as the correction value. The correction unit 114 then corrected the brightness or color in the past image Ip by adding the difference d to the brightness or color value of each pixel within the target area Sp.

[0052] To explain in more detail, if the brightness or color value of a pixel in the target area Sp before correction is represented as L, the correction unit 114 calculates the brightness or color value L' of that pixel in the corrected target area Sp as "L' = L + d". The correction unit 114 then performs a process to add a difference d to each of the multiple pixels included in the target area Sp, thereby uniformly adding or subtracting an offset to the brightness or color of each pixel.

[0053] For example, if the average value Yc of brightness or color obtained from the new image Ic is smaller than the average value Yp of brightness or color obtained from the past image Ip, the difference d will be a negative value. In this case, the correction unit 114 reduces the overall brightness or color of each pixel in the lesion candidate B1 and its surrounding area within the target area Sp. Conversely, if the average value Yc is larger than the average value Yp, the difference d will be a positive value. In this case, the correction unit 114 increases the overall brightness or color of each pixel in the lesion candidate B1 and its surrounding area within the target area Sp.

[0054] Thus, the correction unit 114 can correct the brightness or color of each pixel in the target area Sp so that the average value Yp in the corrected target area Sp is equal to the average value Yc, not only by using the ratio p of the average values ​​Yc and Yp, but also by using the difference d between the average values ​​Yc and Yp. This makes it possible to standardize the shooting conditions regarding brightness or color between the new image Ic and the past image Ip, making it easier for the diagnostician to compare lesion candidates captured at different times and to diagnose changes in lesion candidates over time.

[0055] Although embodiments of the present invention have been described above, these embodiments are merely examples, and the scope of application of the present invention is not limited thereto. That is, the embodiments of the present invention can be applied in various ways, and all embodiments are included within the scope of the present invention. For example, in the above embodiment, the correction unit 114 obtained the average values ​​Yc and Yp as representative values ​​of brightness or color in the peripheral region of the lesion candidate. However, the correction unit 114 is not limited to using the average values ​​Yc and Yp as representative values ​​of brightness or color in the peripheral region of the lesion candidate; it may also use the mode, median, etc. Furthermore, in the above embodiment, the correction unit 114 identified the brightness or color in the peripheral region of the lesion candidate based on a brightness or color histogram. However, the correction unit 114 is not limited to this; for example, it may identify the brightness or color in the peripheral region of the lesion candidate from its position in the image.

[0056] In the above embodiment, the correction unit 114 corrected the brightness or color of the target area Sp by using the ratio p or difference d of the average values ​​Yc and Yp as a correction value, so that the average value Yp, which is a representative value of brightness or color in the corrected target area Sp, is equal to the average value Yc, which is a representative value of brightness or color in the target area Sc. However, the correction unit 114 is not limited to correcting the brightness or color so that the two representative values ​​are equal after correction, but may also correct the brightness or color so that the two representative values ​​are closer together after correction. In other words, the correction unit 114 may correct the brightness in the target area Sp in the past image Ip so that the second representative brightness value after correction is closer to the first representative brightness value. Also, the correction unit 114 may correct the color in the target area Sp in the past image Ip so that the second representative color value after correction is closer to the first representative color value. For example, if the second representative luminance value is greater than the first representative luminance value, the correction unit 114 reduces the luminance of each pixel in the target area Sp so that the corrected second representative luminance value approaches the first representative luminance value. Conversely, if the second representative luminance value is smaller than the first representative luminance value, the correction unit 114 increases the luminance of each pixel in the target area Sp so that the corrected second representative luminance value approaches the first representative luminance value. The same applies to color. Thus, the correction by the correction unit 114 does not necessarily have to be equal to the two representative values ​​after correction, as long as the difference between the two representative values ​​after correction of luminance or color is smaller than before correction. Even if the two representative values ​​after correction are not equal, by correcting the luminance or color so that the two representative values ​​approach each other, the shooting conditions of two images taken at different times can be made closer. Therefore, it becomes easier to compare and observe lesion candidates captured in two images taken at different times, and it becomes easier to diagnose changes in lesion candidates over time.

[0057] In the above embodiment, the correction unit 114 corrected the brightness or color of the target region Sp, which is the region containing the candidate lesion in the past image Ip, using the past image Ip as the correction target. However, the correction unit 114 may also correct the brightness or color of the target region Sc, which is the region containing the candidate lesion in the new image Ic, using the new image Ic as the correction target. When the new image Ic is used as the correction target, the new image Ic corresponds to the second image, and the past image Ip corresponds to the first image. Furthermore, the correction unit 114 may correct the brightness or color of both the new image Ic and the past image Ip, as long as the representative values ​​of the two images after correction become equal or close to each other. In that case, either the new image Ic or the past image Ip can be the first image or the second image.

[0058] In the above embodiment, the new image Ic and past image Ip were images of the skin of subject U, and were images used to diagnose candidate lesions present on the skin of subject U. However, the new image Ic and past image Ip may be images of parts other than the skin of subject U, as long as they are images used to diagnose changes in candidate lesions over time. Furthermore, the new image Ic and past image Ip are not limited to images taken with visible light, infrared light, or ultraviolet light, but may also be X-ray images, ultrasound images, etc. Thus, the new image Ic and past image Ip may be images of any part of the body and taken by any method, as long as the imaging conditions differ depending on when the images were taken.

[0059] In the above embodiment, the image processing device 10 had the parts shown in Figure 2. However, the parts of the image processing device 10 are not limited to being provided in a single device, but may be located in separate, independent devices. For example, any of the functions of the image acquisition unit 111, pre-processing unit 112, cropping unit 113, correction unit 114, and image output unit 115 in the processing unit 11 may be provided in a different device from the other functions. In that case, multiple devices providing each part can be collectively called an image processing device. Also, in the above embodiment, the shooting device 5 was a separate device from the image processing device 10. However, the shooting device 5 may be included in the image processing device 10. In other words, the image processing device 10 may form a single unit together with the shooting device 5, or it may be located in a separate location from the shooting device 5. When the image processing device 10 is integrated with the shooting device 5 to form a single unit, that integrated unit may be called an image processing device.

[0060] In the above embodiment, the processing unit 11 functioned as the respective parts shown in Figure 2 by the CPU executing a program stored in the ROM or storage unit 12. However, the processing unit 11 may be dedicated hardware. Dedicated hardware includes, for example, a single circuit, a composite circuit, a programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. If the processing unit 11 is dedicated hardware, each function of each part may be realized by separate hardware, or the functions of each part may be realized together by a single piece of hardware. Furthermore, some of the functions of each part may be realized by dedicated hardware, and other parts by software or firmware. In this way, the processing unit 11 can realize the above-mentioned functions by hardware, software, firmware, or a combination thereof.

[0061] It is also possible to make an existing computer, such as a personal computer or cloud server, function as the image processing device 10 by applying the program that defines the operation of the image processing device 10 described above to that computer. Furthermore, the method of distributing such a program is arbitrary; for example, it may be distributed by storing it on a computer-readable recording medium such as a CD-ROM (Compact Disk ROM), DVD (Digital Versatile Disk), MO (Magneto Optical Disk), or memory card, or it may be distributed via a communication network such as the Internet.

[0062] Although preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and substitutions can be made to the embodiments described above without departing from the scope of the claims. [Explanation of Symbols]

[0063] 10…Image processing device, 11…Processing unit, B1~B4…Candidate lesions

Claims

1. A first representative brightness value, which is a representative brightness value in the area surrounding the candidate lesion in the first image in which the candidate lesion was captured, is obtained. A second representative brightness value, which is a representative brightness value in the area surrounding the candidate lesion in a second image taken at a different time than the first image, is obtained. The brightness in the region containing the candidate lesion in the second image is corrected based on the first and second brightness representative values ​​so that the second brightness representative value in the corrected second image becomes equal to or close to the first brightness representative value. An image processing apparatus characterized by comprising a processing unit.

2. The aforementioned processing unit, When correcting the brightness in the region containing the candidate lesion in the second image, the value obtained by dividing the first brightness representative value by the second brightness representative value is multiplied by the brightness in the region containing the candidate lesion. The image processing apparatus according to feature 1.

3. The aforementioned processing unit, When correcting the brightness in the region containing the candidate lesion in the second image, the value obtained by subtracting the first brightness representative value by the second brightness representative value is added to the brightness in the region containing the candidate lesion. The image processing apparatus according to feature 1.

4. The aforementioned processing unit, Based on the brightness histogram in the region containing the candidate lesion in the first image, the first representative brightness value is obtained. A second representative brightness value is obtained based on the brightness histogram in the region containing the candidate lesion in the second image. The image processing apparatus according to any one of claims 1 to 3.

5. The aforementioned processing unit, A first representative color value, which is a representative color value of the color in the area surrounding the candidate lesion in the first image, is obtained. A second representative color value, which is a representative color value of the surrounding area of ​​the candidate lesion in the second image, is obtained. The color of the region containing the candidate lesion in the second image is corrected based on the first color representative value and the second color representative value so that the second color representative value in the corrected second image becomes equal to or close to the first color representative value. The image processing apparatus according to any one of claims 1 to 3.

6. A first color representative value, which is a representative value of the color in the area surrounding the candidate lesion in the first image in which the candidate lesion was captured, is obtained. A second representative color value, which is a representative color value of the area surrounding the candidate lesion in a second image taken at a different time than the first image, is obtained. The color of the region containing the candidate lesion in the second image is corrected based on the first color representative value and the second color representative value so that the second color representative value in the corrected second image becomes equal to or close to the first color representative value. An image processing apparatus characterized by comprising a processing unit.

7. Computers A first representative brightness value, which is a representative brightness value in the area surrounding the candidate lesion in the first image in which the candidate lesion was captured, is obtained. A second representative brightness value, which is a representative brightness value in the area surrounding the candidate lesion in a second image taken at a different time than the first image, is obtained. The brightness in the region containing the candidate lesion in the second image is corrected based on the first and second brightness representative values ​​so that the second brightness representative value in the corrected second image becomes equal to or close to the first brightness representative value. An image processing method characterized by the following:

8. Computers A first color representative value, which is a representative value of the color in the area surrounding the candidate lesion in the first image in which the candidate lesion was captured, is obtained. A second representative color value, which is a representative color value of the area surrounding the candidate lesion in a second image taken at a different time than the first image, is obtained. The color of the region containing the candidate lesion in the second image is corrected based on the first color representative value and the second color representative value so that the second color representative value in the corrected second image becomes equal to or close to the first color representative value. An image processing method characterized by the following:

9. Computers, A first representative brightness value, which is a representative brightness value in the area surrounding the candidate lesion in the first image in which the candidate lesion was captured, is obtained. A second representative brightness value, which is a representative brightness value in the area surrounding the candidate lesion in a second image taken at a different time than the first image, is obtained. The brightness in the region containing the candidate lesion in the second image is corrected based on the first and second brightness representative values ​​so that the second brightness representative value in the corrected second image becomes equal to or close to the first brightness representative value. A program designed to function as a processing unit.

10. Computers, A first color representative value, which is a representative value of the color in the area surrounding the candidate lesion in the first image in which the candidate lesion was captured, is obtained. A second representative color value, which is a representative color value of the area surrounding the candidate lesion in a second image taken at a different time than the first image, is obtained. The color of the region containing the candidate lesion in the second image is corrected based on the first color representative value and the second color representative value so that the second color representative value in the corrected second image becomes equal to or close to the first color representative value. A program designed to function as a processing unit.

Images