Image processing device, image processing method, and program

Abstract

To suppress an influence of a protective film.SOLUTION: An image processing device comprises: detection means 21 which detects a predetermined mark added to a protective film affixed to a lens of a photographing apparatus from an image acquired by the photographing apparatus; specifying means 22 which in a case where the image in which the mark is detected is a polarization photographing image acquired by polarization imaging, specifies the orientation of the protective film relative to the lens, based on a position where the mark in the polarization photographing image is detected; and adjustment means 25 which adjusts the polarization photographing image in accordance with the specified orientation, so as to reproduce an image photographed by the photographing apparatus without the protective film.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The disclosure of this specification relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Conventionally, as an apparatus for photographing an image of an affected part used for image diagnosis by contact photography, a dermoscope and a camera equipped with a dermoscope (dermoscopy camera) are known (see Patent Document 1). Patent Document 1 describes a technique of interposing an adapter between the dermoscope and the affected part. Some dermoscopy cameras directly contact the lens of the dermoscope with the affected part for contact photography.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In such a dermoscopy camera, when photographing by directly contacting the lens with the affected part, for the purpose of protecting the lens surface, a protective film may be attached to the lens surface. In this case, the image photographed through the protective film has its brightness and color tone changed due to the optical characteristics of the protective film, resulting in an image different from that without the protective film. The above problems apply equally to cameras other than dermoscopy cameras as long as a protective film is attached to the lens surface.

[0005] Based on the above circumstances, an object according to one aspect of the present invention is to provide a technique for suppressing the influence of the protective film.

Means for Solving the Problems

[0006] An image processing apparatus according to an aspect of the present invention includes: a detection unit that detects a predetermined mark attached to a protective film attached to a lens of the imaging device from an image acquired by the imaging device; a specification unit that specifies the orientation of the protective film with respect to the lens based on the position where the mark is detected in the polarized light captured image when the image in which the mark is detected is a polarized light captured image acquired by polarized light imaging; and an adjustment unit that adjusts the polarized light captured image according to the specified orientation so as to reproduce an image captured by the imaging device without the protective film.

Advantages of the Invention

[0007] According to the above aspect, the influence of the protective film can be suppressed.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

[0009] <First Embodiment> As shown in FIGS. 1 and 2, the dermoscopy camera 10 is a photographing device with a dermoscope, which is a magnifying glass that can non-invasively observe a lesion generated on the skin while reducing the reflected light from the skin surface. Further, the dermoscopy camera 10 is also a computer that performs image processing on the acquired image. The image acquired by the dermoscopy camera 10 is used, for example, for image diagnosis of skin diseases.

[0010] In the dermoscopy camera 10, contact photography is performed with the lens 11 close to the affected part A of the skin S. In some cases, the lens 11 may be directly brought into contact with the affected part for photographing. Therefore, for the purpose of protecting the lens 11, as shown in FIGS. 1 and 2, photographing is performed with the protective film 12 attached to the surface of the lens 11 that comes into contact with the affected part. Such a protective film 12 is configured as a disposable film in consideration of hygiene, and is replaced each time it is used.

[0011] As a specific method for reducing the influence of reflected light from the skin surface, in the dermoscopy camera 10 shown in FIG. 1, among well-known photographing methods, it corresponds to a photographing method (hereinafter referred to as polarized light photographing) using at least a polarizing filter.

[0012] As shown in Fig. 2, the dermoscopy camera 10 irradiates the affected area A with light from the light source 13 through the lens 11, and the reflected light reflected from the affected area A is detected by the image sensor 16. In particular, in the setting of performing polarization imaging by the crossed polarization method, the dermoscopy camera 10 arranges a polarizer 14, which is a polarization filter, between the light source 13 and the lens 11, and an analyzer 15, which is a polarization filter, between the lens 11 and the image sensor 16. The polarizer 14 and the analyzer 15 are arranged such that their transmission axes are orthogonal to each other. Thereby, in polarization imaging, linearly polarized light is irradiated onto the affected area A through the polarizer 14 provided in the illumination optical path, and the specularly reflected light whose polarization state is maintained among the reflected light is blocked by the analyzer 15. For this reason, it is possible to obtain a skin image suitable for image diagnosis with the specularly reflected light (specular reflection light) on the skin surface reduced.

[0013] As described above, the protective film 12 shown in Fig. 3 is attached to protect the lens 11. For this reason, the protective film 12 has a shape that can cover the surface of the lens 11, for example, a circular shape as shown in Fig. 3. Further, as shown in Fig. 3, a mark M is provided on the protective film 12. The mark M is for attaching the protective film 12 in the correct orientation when attaching the protective film 12 to the lens 11. By providing the mark M, the user can easily attach the protective film 12 to the lens 11 in the correct orientation.

[0014] The protective film 12 is a birefringent material made of a polymer material or the like, and has an ordinary axis and an extraordinary axis (collectively referred to as the principal axis) with different refractive indices. The linearly polarized light incident on the protective film 12 in polarization imaging is converted into different elliptically polarized lights depending on the magnitude of the retardation of the protective film 12 (the product of the birefringence, which is the refractive index difference between the directions of the ordinary axis and the extraordinary axis, and the thickness of the protective film 12) and the direction of the principal axis (ordinary axis and extraordinary axis). For this reason, when the orientation of the protective film 12 attached to the lens 11 changes, the state of the light (polarization state and wavelength intensity distribution) passing through the protective film 12 is different, and as a result, in polarization imaging, the obtained image also changes.

[0015] However, in the birefringent material, there is a direction called the optical axis that does not cause birefringence in the incident light. By orienting this optical axis in the propagation direction of the incident light, it is possible to avoid changes in the polarization state. Therefore, especially when polarization imaging is performed, when attaching the protective film 12 to the lens 11, it is desirable to adjust the orientation of the protective film 12 so that the polarization state is not changed by the protective film 12.

[0016] Since the protective film 12 is provided with a mark M, the user can easily attach the protective film 12 to the lens 11 in the correct orientation. Specifically, by attaching the mark M in a predetermined direction (in this example, vertically upward) of the lens 11 (the dermoscopy camera 10), it can be attached to the lens 11 in the correct orientation where the incident light is not affected by birefringence. Thereby, even when frequently replacing the protective film 12 and using the dermoscopy camera 10, changes in the characteristics of the dermoscopy camera 10 can be avoided.

[0017] In the dermoscopy camera 10, the mark M provided on the protective film 12 is used not only as a guide for the user to attach the protective film 12 to the lens 11, but also effectively used in the image processing of the captured images performed after attaching the protective film 12 to the lens 11.

[0018] Hereinafter, with reference to FIGS. 4 and 5, the image processing using the mark M performed by the dermoscopy camera 10 and the configuration of the dermoscopy camera 10 for performing the image processing will be described. The image processing performed by the dermoscopy camera 10 is performed when the processor 20 (see FIG. 2) of the dermoscopy camera 10 operates as each functional element (detection means 21, identification means 22, selection means 23, adjustment means 25, acquisition means 24) shown in FIG. 5 by executing a predetermined program.

[0019] When the dermoscopy camera 10 takes a picture with the protective film 12 attached, for example, a captured image 1 as shown in FIG. 4 is obtained. In this captured image 1, within the imaging range 110 of the skin S through the lens 11, in addition to the affected part A (strictly speaking, the image of the affected part), a mark M (strictly speaking, the image of the mark) is included.

[0020] The detection means 21 of the processor 20 detects a predetermined mark M attached to the protective film 12 attached to the lens 11 of the processor 20 from the captured image 1 acquired by the dermoscopy camera 10 which is an imaging device.

[0021] The detection means 21 further determines whether or not the protective film 12 is attached to the lens 11 based on the detection result. When the protective film 12 is attached to the lens 11, the mark M is detected. Therefore, the detection means 21 determines that the protective film 12 is attached to the lens 11 when the mark M is detected. When the protective film 12 is not attached to the lens 11, the mark M is not detected. Therefore, the detection means 21 determines that the protective film 12 is not attached to the lens 11 when the mark M is not detected.

[0022] Note that the method by which the detection means 21 detects the mark M from the captured image 1 is not particularly limited and can be detected using any image processing. The detection means 21 may detect the mark M using, for example, a machine learning model that has previously machine-learned the mark M, that is, a machine learning has been performed so as to output the presence or absence of the mark M for the input image.

[0023] When the image in which the mark M is detected is an image acquired by polarized light imaging (hereinafter referred to as a polarized light imaging image), the specifying means 22 of the processor 20 specifies the orientation of the protective film 12 with respect to the lens 11 based on the position where the mark M is detected in the polarized light imaging image. Since the position of the mark M in the polarized light imaging image changes according to the orientation in which the protective film 12 is attached to the lens 11, the specifying means 22 specifies the orientation of the protective film 12 from the position of the mark M in the polarized light imaging image.

[0024] The selection means 23 of the processor 20 selects one conversion matrix based on the orientation specified by the specifying means 22 from a plurality of pre-prepared conversion matrices. Each of the plurality of conversion matrices is a conversion matrix for converting a polarization imaging image obtained by the dermoscopy camera 10 with a protective film by polarization imaging into a polarization imaging image obtained by the dermoscopy camera 10 without a protective film by polarization imaging.

[0025] The method for creating such a conversion matrix is not particularly limited. For example, it may be created by the following procedure. First, prepare a color chart for image correction as the imaging object. Then, the color chart is polarization-imaged with the protective film 12 attached to the lens 11 and without the protective film attached to obtain polarization imaging images. Note that the imaging with the protective film 12 attached to the lens 11 is performed a plurality of times while changing the orientation of the protective film 12. Thereby, a plurality of polarization imaging images (images with film) obtained with the protective film 12 attached in different orientations and a polarization imaging image (image without film) obtained with the protective film 12 not attached are obtained.

[0026] Thereafter, a conversion matrix for image adjustment is created for each orientation by the least squares method from the pairs of the pixel values of each color of the color chart of the image with film in each orientation and the pixel values of each color of the color chart of one image without film. The conversion matrix is created after converting the pixel values of each color in the RGB color space into the pixel values in the Lab color space. That is, the conversion matrix obtained by the selection means 23 is a matrix for converting the pixel values on the Lab color space. When protective films with different characteristics (such as thickness and material) can be used, it is desirable to generate a plurality of conversion matrices by the above method for each protective film with different characteristics in advance.

[0027] When the acquisition means 24 of the processor 20 detects that the image in which the mark M is detected is an image acquired by means other than polarized photography (hereinafter referred to as a non-polarized photography image), the dermoscopy camera 10 acquires an image with the protective film 12 by non-polarized photography and acquires a conversion matrix for converting it into an image acquired without the protective film 12 by non-polarized photography.

[0028] The method for creating such a conversion matrix is not particularly limited, but it may be created by the same procedure as the conversion matrix for polarized photography described above. However, in non-polarized photography, since it is assumed that the influence of the orientation of the protective film 12 on the image is minor, it is not necessary to create a conversion matrix for each orientation of the protective film 12. For example, one conversion matrix may be created from a non-polarized photography image (image with film) acquired by non-polarized photography with the protective film 12 attached in the correct orientation and a non-polarized photography image (image without film) acquired by non-polarized photography with the protective film 12 not attached.

[0029] When the image in which the mark M is detected is a polarized photography image, the adjustment means 25 of the processor 20 adjusts the polarized photography image according to the orientation specified by the specifying means 22 so as to reproduce the image taken by the dermoscopy camera 10 without the protective film 12. Specifically, the adjustment means 25 adjusts the polarized photography image based on the conversion matrix selected by the selection means 23 according to the orientation specified by the specifying means 22.

[0030] When the image in which the mark M is detected is a non-polarized photography image, the adjustment means 25 adjusts the non-polarized photography image regardless of the orientation of the protective film 12 with respect to the lens 11 so as to reproduce the image taken by the dermoscopy camera 10 without the protective film 12. Specifically, the adjustment means 25 adjusts the non-polarized photography image based on the conversion matrix acquired by the acquisition means 24.

[0031] Note that, as described above, the conversion matrix used by the acquisition means 24 is a matrix for performing conversion in the Lab color space. Therefore, the adjustment means 25 first converts the image in the RGB color space into an image in the Lab color space, and then converts the converted image in the Lab color space using the conversion matrix acquired by the selection means 23. And finally, the adjustment means 25 converts the obtained image in the Lab color space back into an image in the RGB color space, thereby reproducing the image captured by the dermoscopy camera 10 without the protective film 12.

[0032] According to the dermoscopy camera 10 configured as described above, the influence of the protective film 12 can be suppressed. More specifically, when shooting is performed with the protective film 12 attached to the lens 11, the captured image is adjusted and output so as to approach the image captured without the protective film 12 being attached. Therefore, it is possible to obtain substantially the same image whether the protective film 12 is attached or not.

[0033] Also, in the dermoscopy camera 10, in consideration of hygiene, for example, a new protective film is reattached to the lens surface for each shooting. Since the birefringence of the protective film has different effects on the incident light depending on the orientation of the protective film, when the protective film is reattached and the orientation of the protective film changes, the influence of the protective film on the image in polarization photography using polarization also changes. However, in the dermoscopy camera 10, when a polarization photography image is input, the angle of the protective film 12 is specified and adjustment for the polarization photography image corresponding to the specified angle is performed. Therefore, the influence of the difference in the angle of the protective film 12 can also be eliminated. The dermoscopy camera 10 that outputs a constant result as an image regardless of the presence or absence of such a protective film 12 is suitable for ensuring the accuracy of diagnosis in image diagnosis performed based on a unified standard for the captured image.

[0034] In addition, in the dermoscopy camera 10, a conversion matrix created from images obtained with and without the protective film 12 is used to adjust the polarized light captured image, so that the influence of the protective film 12 can be accurately compensated. Further, a plurality of conversion matrices with different attachment angles of the protective film 12 are prepared in advance, and an optimal conversion matrix is selected according to the attachment angle of the protective film 12 at the time of shooting the image to be converted, so that the influence of the attachment angle of the protective film 12 can be more accurately eliminated.

[0035] Furthermore, in the dermoscopy camera 10, in addition to adjusting the polarized light captured image taken with the protective film 12, the non-polarized light captured image is also adjusted. Thereby, the influence of the protective film 12 can be eliminated regardless of the shooting method. Also, even in non-polarized light shooting, by using a conversion matrix in the same manner as in polarized light shooting, the influence of the protective film 12 can be accurately compensated. Also, in non-polarized light shooting, it is not necessary to prepare a plurality of conversion matrices like in polarized light shooting, so the labor required for prior preparation is also reduced.

[0036] The image processing shown in FIG. 6 is started when the dermoscopy camera 10 performs shooting. When shooting is performed with the dermoscopy camera 10, the processor 20 first acquires a captured image 1 created based on the signal acquired from the image sensor 16 (step S11), and performs a process of detecting the mark M from the acquired captured image 1 (step S12).

[0037] When the processor 20 does not detect the mark M (NO in step S13), it is presumed that the protective film 12 is not attached to the lens 11. Therefore, the processor 20 then skips the processes of steps S14 to S17 and step S19 described later, saves the image acquired in step S11 (step S18), and ends the process of FIG. 6. That is, when the protective film 12 is not attached, the dermoscopy camera 10 saves the captured image without performing special image processing for eliminating the influence of the protective film 12.

[0038] On the other hand, when the processor 20 detects the mark M (step S13 YES), the processor 20 determines whether the captured image 1 is a polarized captured image (step S14). The method for determining whether the captured image 1 is a polarized captured image is not particularly limited. For example, the processor 20 may determine whether the captured image 1 is a polarized captured image based on whether the current setting of the dermoscopy camera 10 is set to perform polarized imaging. Further, the processor 20 may determine whether the captured image 1 is a polarized captured image from the meta information (e.g., EXIF information) attached to the captured image 1 at the time of creation of the captured image 1.

[0039] When it is determined that the captured image 1 is a polarized captured image (step S14 YES), the processor 20 specifies the orientation of the protective film 12 from the position of the mark M detected in step S12 (step S15). The orientation of the protective film 12 may be calculated, for example, as an angle θ from the origin through which the optical axis of the lens 11 passes, based on the coordinates in the captured image 1 in which the mark M is detected, as shown in FIG. 7. FIG. 7 shows an example of calculating the angle with respect to the horizontal direction of the image.

[0040] When the orientation of the protective film 12 is specified, the processor 20 selects a conversion matrix for polarized imaging based on the specified orientation (step S16). A plurality of conversion matrices are stored in the dermoscopy camera 10 in advance, and the processor 20 selects one conversion matrix from these based on the orientation specified in step S15. For example, when the conversion matrices are stored at intervals of 30° such as angle θ = 0°, 30°, 60°, 90°, ···, the processor 20 may select the conversion matrix with the angle closest to the angle calculated in step S15.

[0041] When the conversion matrix for polarized imaging is selected, the processor 20 adjusts the captured image 1 obtained in step S11 based on the conversion matrix selected in step S16 (step S17). Here, the processor 20 converts the captured image 1 using the conversion matrix for polarized imaging to reproduce a polarized captured image captured without the protective film 12.

[0042] After that, the processor 20 saves the adjusted image (step S18) and ends the process of FIG. 6. That is, when polarized light imaging is performed with the protective film 12 attached, the dermoscopy camera 10 performs adjustment on the captured image 1 according to the orientation of the protective film 12, reproduces the image captured without the protective film 12, and saves the reproduced image.

[0043] If it is determined in step S14 that the captured image 1 is not a polarized light captured image (step S14 NO), the processor 20 acquires a conversion matrix for non-polarized light imaging (step S19). Since only one conversion matrix for non-polarized light imaging is stored in advance, the processor 20 may acquire that one conversion matrix.

[0044] When the conversion matrix for non-polarized light imaging is selected, the processor 20 adjusts the captured image 1 acquired in step S11 based on the conversion matrix acquired in step S19 (step S17). Here, the processor 20 converts the captured image 1 using the conversion matrix for non-polarized light imaging to reproduce the image captured without the protective film 12.

[0045] After that, the processor 20 saves the adjusted image (step S18) and ends the process of FIG. 6. That is, when non-polarized light imaging is performed with the protective film 12 attached, the dermoscopy camera 10 performs adjustment on the captured image 1 regardless of the orientation of the protective film 12, reproduces the image captured without the protective film 12, and saves the reproduced image.

[0046] As described above, by the processor 20 executing the image processing shown in FIG. 6, the dermoscopy camera 10 can suppress the influence of the protective film 12. Specifically, when the protective film 12 is attached, the brightness and color tone of the obtained image change. However, when the protective film 12 is attached to the lens 11, the dermoscopy camera 10 automatically detects this fact from the captured image 1 and performs image conversion to compensate for the influence of the protective film 12 using a conversion matrix. Also, in polarized light photography, the orientation of the protective film 12 also affects the image. However, when polarized light photography is performed with the protective film 12 attached to the lens 11, the dermoscopy camera 10 automatically detects the orientation of the protective film 12 and performs image conversion to compensate for the influence of the protective film 12 using a conversion matrix corresponding to the orientation of the protective film 12. Therefore, according to the dermoscopy camera 10, the influence of the protective film 12 can be suppressed, so that the user can obtain a certain image regardless of the presence or absence and orientation of the protective film 12 without being aware of the presence of the protective film 12.

[0047] <Second Embodiment> The dermoscopy camera according to this embodiment is configured to display a warning to the user instead of adjusting the image when it is not possible to reproduce an image without the protective film 12 from an image with the protective film 12 attached with sufficient accuracy, but to adjust the image when it is possible to reproduce with sufficient accuracy.

[0048] The dermoscopy camera according to this embodiment is different from the dermoscopy camera 10 in that it includes a processor 20a shown in FIG. 8 instead of the processor 20 shown in FIG. 5. Other points are the same as those of the dermoscopy camera 10. Hereinafter, the same reference numerals are used for the same components as those of the dermoscopy camera 10 included in the dermoscopy camera according to this embodiment.

[0049] As shown in FIG. 8, the processor 20a is different from the processor 20 in that it includes a determination means 26 and a warning means 27. The processor 20a is the same as the processor 20 in that it includes a detection means 21, a specification means 22, a selection means 23, an acquisition means 24, and an adjustment means 25.

[0050] The determination means 26 of the processor 20a determines whether an image captured by the dermoscopy camera without the protective film 12 can be reproduced from the polarized captured image based on the orientation of the protective film 12 specified by the specification means 22.

[0051] Although there are innumerable orientations of the protective film 12, the number of conversion matrices prepared for polarized photography actually provided in the dermoscopy camera is finite. For this reason, in most cases, the orientation of the protective film 12 and the orientation corresponding to each of the plurality of conversion matrices do not exactly match. It is expected that the reproducibility of the image captured without the protective film 12 will decrease as the difference between the orientation of the protective film 12 and the orientation corresponding to the conversion matrix increases. Therefore, the determination means 26 determines whether the decrease in reproducibility is within an allowable range. Specifically, for example, the determination means 26 compares the angle θ calculated by the specification means 22 with the angles corresponding to the plurality of conversion matrices, and determines whether the minimum value of the difference exceeds a predetermined threshold value (for example, 10°). If the minimum value of the difference is within the threshold value, the determination means 26 determines that the image captured by the dermoscopy camera without the protective film 12 can be reproduced. On the other hand, when the minimum value of the difference exceeds the threshold value, the determination means 26 determines that the image captured by the dermoscopy camera without the protective film 12 cannot be reproduced.

[0052] The warning means 27 of the processor 20a issues a warning based on the result of the determination by the determination means 26. For example, when it is determined that an image captured by the dermoscopy camera cannot be reproduced without the protective film 12, the warning means 27 issues a warning. The warning is displayed, for example, as a message on a display (not shown) of the dermoscopy camera. The content of the message is not particularly limited, but may indicate that the influence of the protective film 12 may appear in the image, that it is desirable to adjust the orientation of the protective film 12 correctly, etc. By displaying such a message, the user can recognize that the captured image is affected by the protective film 12. Also, the user can recognize that the orientation of the protective film 12 is incorrect. Therefore, since the user can be prompted to reattach the protective film 12, an image in which the influence of the protective film 12 is eliminated can be obtained again by reattaching the protective film 12.

[0053] Hereinafter, with reference to FIG. 9, a specific example of shooting and image processing performed by the dermoscopy camera according to the present embodiment will be described while paying attention to the differences from the image processing of FIG. 6.

[0054] The process shown in FIG. 9 starts when the dermoscopy camera according to the present embodiment shifts to the shooting mode. When shifting to the shooting mode and live shooting is started (step S21), the processor 20a performs a process of detecting a mark using the live image (step S22). If no mark is detected from the live image (step S23 NO), the processor 20a directly performs still image shooting to acquire an image (step S33), and then saves the acquired image (step S30), and ends the process of FIG. 9.

[0055] When a mark is detected from the live image (step S23 YES), the processor 20a determines whether the shooting method is polarized shooting (step S24). If it is not polarized shooting (step S24 NO), the processor 20a acquires a conversion matrix for non-polarized shooting (step S32), and then performs shooting with the current settings to acquire an image (step S28). Further, image adjustment is performed on the acquired image using the conversion matrix acquired in step S32 (step S29), the adjusted image is saved (step S30), and the process of FIG. 9 ends. On the other hand, when the shooting method is polarized shooting (step S24 YES), the processor 20a specifies the orientation of the film from the position of the detected mark (step S25).

[0056] In the process shown in FIG. 9, when the orientation of the protective film 12 is specified in step S25, the processor 20a determines whether an image taken without the protective film 12 can be reproduced (step S26). Here, the above-described determination means 26 makes the determination. If it is determined that reproduction is possible (step S26 YES), the processor 20a performs the processes from step S27 to step S30. The process of step S28 is as described above. Also, the processes of step S27, step S29, and step S30 are the same as the processes of step S16 to step S18 in FIG. 6.

[0057] In step S26, when the processor 20a determines that reproduction is impossible (step S26 NO), the processor 20a issues a warning to the user (step S31), and the process of FIG. 9 ends. Here, the above-described warning means 27 issues a warning.

[0058] As described above, by the processor 20a executing the process shown in FIG. 9, the dermoscopy camera according to the present embodiment can suppress the influence of the protective film 12, which is the same as the dermoscopy camera 10. Further, when the dermoscopy camera according to the present embodiment determines that image adjustment (image reproduction) cannot be performed with sufficient accuracy, it warns the user to that effect without performing image adjustment. Thereby, it is possible to avoid an image in which the influence of the protective film 12 has not been sufficiently eliminated from being stored in the dermoscopy camera. Further, since it is determined whether image adjustment is possible or not with a live image before still image shooting and a warning is issued, it is possible to prevent unnecessary shooting. Further, by issuing a warning, the user can be made aware that the protective film 12 is not correctly oriented, so that the user can be prompted to reattach the protective film 12.

[0059] Note that in FIG. 9, an example in which only a warning is issued when it is determined in step S26 that reproduction is impossible is shown, but the processor 20a may perform the same process as in the case where reproduction is possible after issuing a warning. That is, after issuing a warning, shooting may be performed, image adjustment may be performed so as to approach the image shot without the protective film 12, and the adjusted image may be saved.

[0060] Further, in FIG. 9, an example in which a warning is issued based on a live image acquired before still image shooting is shown, but the warning may be issued based on an image acquired by still image shooting.

[0061] <Third Embodiment> Unlike the dermoscopy camera described above, the dermoscopy camera according to this embodiment does not perform image adjustment. The dermoscopy camera according to this embodiment is configured to determine whether the protective film 12 is correctly attached in polarized light imaging, and to display a warning to the user if it is not correctly attached. Note that the configuration according to this embodiment has a minor impact of the protective film 12 if it is not polarized light imaging, and does not significantly affect image diagnosis. Also, if the protective film 12 is correctly attached, even in polarized light imaging, the impact of the protective film 12 is minor and does not significantly affect image diagnosis, which is particularly effective when it can be considered in this way.

[0062] The dermoscopy camera according to this embodiment is different from the dermoscopy camera 10 in that it includes a processor 20b shown in FIG. 10 instead of the processor 20 shown in FIG. 5. Other points are the same as those of the dermoscopy camera 10. Hereinafter, the same reference numerals are used for the same components as those of the dermoscopy camera 10 included in the dermoscopy camera according to this embodiment.

[0063] As shown in FIG. 10, the processor 20b omits the selection means 23, the acquisition means 24, and the adjustment means 25. This is because image adjustment is not performed in the dermoscopy camera according to this embodiment. Also, the processor 20b is the same as the processor 20 in that it includes the detection means 21 and the identification means 22. Further, the processor 20b includes a determination means 28 and a warning means 29.

[0064] The determination means 28 of the processor 20b determines whether the protective film 12 is correctly attached based on the orientation of the protective film 12 identified by the identification means 22. The determination means 28 determines that it is correctly attached, for example, when the error between the identified orientation and the correct orientation is within a predetermined allowable range.

[0065] The warning means 29 of the processor 20b issues a warning based on the result of the determination by the determination means 28. For example, the warning means 29 issues a warning when it is determined that the orientation of the protective film 12 is not within the allowable range. The warning is displayed as a message, for example, on a display (not shown) of the dermoscopy camera. The content of the message is not particularly limited, but may be the same as in the second embodiment. For example, it may indicate that the influence of the protective film 12 may appear in the image and that it is desirable to adjust the orientation of the protective film 12 correctly. By displaying such a message, the user can recognize that the orientation of the protective film 12 is incorrect, so that the user can be given an opportunity to reattach the protective film 12. As a result, an image from which the influence of the protective film 12 has been eliminated can be acquired again.

[0066] The process shown in FIG. 11 starts when the dermoscopy camera according to the present embodiment shifts to the shooting mode. When shifting to the shooting mode and live shooting is started with the dermoscopy camera (step S41), the processor 20b determines whether the shooting method is polarized shooting (step S42).

[0067] When it is determined that the shooting method is not polarized shooting (step S42 NO), the processor 20b directly performs still image shooting to acquire an image without performing the processes from step S43 to step S47 described later (step S48), and then saves the acquired image (step S49), and ends the process of FIG. 11.

[0068] When it is determined that the shooting method is polarized shooting (step S42 YES), the processor 20b performs a process of detecting the mark M from the live image (step S43). The process of step S43 is the same as the process of step S12 in FIG. 6.

[0069] If the processor 20b does not detect the mark M (step S44 NO), the processor 20b directly performs still image shooting to acquire an image without performing the processes from step S45 to step S47 described below (step S48). Then, the acquired image is saved (step S49), and the process of FIG. 11 ends.

[0070] On the other hand, if the processor 20b detects the mark M (step S44 YES), the processor 20b identifies the orientation of the protective film 12 from the position of the mark M detected in step S43 (step S45). The process of step S45 is the same as the process of step S15 in FIG. 6.

[0071] After that, the processor 20b determines whether the orientation of the protective film 12 specified in step S45 is within the allowable range (step S46). Here, the above-described determination means 28 makes the determination. If it is determined that it is within the allowable range (step S46 YES), the processor 20b directly performs still image shooting to acquire an image (step S48). Then, the acquired image is saved (step S49), and the process of FIG. 11 ends. That is, in the case of polarized light photography, when the orientation of the protective film 12 is within the allowable range, the dermoscopy camera saves the photographed image.

[0072] In step S46, if the processor 20b determines that the orientation of the protective film 12 is outside the allowable range (step S46 NO), the processor 20b issues a warning (step S47) and ends the process of FIG. 11. That is, in the case of polarized light photography, when the orientation of the protective film 12 is outside the allowable range, the dermoscopy camera does not save the photographed image and issues a warning.

[0073] As described above, by the processor 20b executing the process shown in FIG. 11, the dermoscopy camera according to the present embodiment saves the captured image 1 when the influence of the protective film 12 is minor, and when the influence of the protective film 12 cannot be ignored, it does not save the captured image 1 and issues a warning. Thereby, it is possible to avoid the situation where an image in which the influence of the protective film 12 has not been sufficiently eliminated is saved in the dermoscopy camera. In addition, since the orientation of the film is judged from the live image and a warning is issued before still image shooting, it is possible to prevent unnecessary shooting. Further, by issuing a warning, the user can be made aware that the orientation of the protective film 12 is incorrect, so that the user can be prompted to reattach the protective film 12. Note that, in FIG. 11 as well, an example of issuing a warning based on the live image acquired before still image shooting is shown, but the warning may also be issued based on the image acquired during still image shooting.

[0074] The above-described embodiments are presented with specific examples for ease of understanding of the invention, and the present invention is not limited to the above-described embodiments, but should be understood to include various modifications and alternative forms of the above-described embodiments. For example, it will be understood that the above-described embodiments can be embodied by modifying the components without departing from the gist thereof. Also, it will be understood that various embodiments can be implemented by appropriately combining the plurality of components disclosed in the above-described embodiments. Furthermore, it will be understood by those skilled in the art that various embodiments can be implemented by deleting some of the components shown in the embodiments or adding some components to the components shown in the embodiments. That is, the above-described image processing apparatus, image processing method, and program can be variously modified and changed without departing from the scope of the claims.

[0075] FIG. 12 is a diagram illustrating a modified example of the protective film shown in FIG. 3. In the above-described embodiment, an example of attaching the protective film 12 shown in FIG. 3 to the lens 11 has been shown. However, instead of the protective film 12, the protective film 12a shown in FIG. 12 may be used. The protective film 12a is different from the protective film 12 in that two marks (M1, M2) serving as a guide for attaching the protective film 12a in the correct orientation when attaching the protective film 12a to the lens 11 are provided at two locations.

[0076] By using the protective film 12a provided with marks at two locations, it is possible to inspect whether the protective film 12a is attached to the center of the lens 11 by image processing. As a result, since it becomes possible to more accurately specify the orientation of the protective film 12a, it is possible to more accurately determine whether it is attached in the correct orientation. The configuration of the protective film 12a is particularly effective when the lens 11 is somewhat larger than the protective film 12a and the protective film 12a is likely to be attached while being displaced from the center of the lens 11.

[0077] For example, as shown in FIG. 12, when trying to specify the orientation of the protective film 12a in a state where the protective film 12a is not attached to the center of the lens 11, there is a risk of misidentifying that the protective film 12a is attached in the orientation corresponding to the angle θ1, although it is actually attached in the orientation corresponding to the angle θ. By specifying the orientation after confirming that the protective film 12a is attached to the center of the lens 11 using the above-described two marks, such misidentification about the orientation can be avoided.

[0078] FIG. 13 is a diagram illustrating the configuration of the image processing system 100 according to an embodiment. In the above-described embodiment, an example in which image processing is performed within a dermoscopy camera has been shown. However, the image processing may be performed by an image processing apparatus different from the dermoscopy camera. As shown in FIG. 13, the captured image 1 acquired by the dermoscopy camera 10 may be transmitted to the image processing apparatus 30, and the processor of the image processing apparatus 30 may execute the image processing illustrated in FIGS. 6, 9, and 11. That is, the image processing apparatus that executes the above-described image processing may be a computer including a processor, and may or may not include the dermoscopy camera 10.

[0079] Further, in the above-described embodiment, a dermoscopy camera has been exemplified as the imaging device. However, the imaging device that acquires an image to be subjected to image processing is not limited to a dermoscopy camera. Any device that performs polarized light imaging by attaching a protective film 12 to the lens surface may be used.

[0080] Also, in the above-described embodiment, an example in which a mark is provided on the protective film attached to the lens surface has been shown. However, if it is possible to set in the imaging device whether or not a protective film is attached, a protective film without a mark may be attached to the lens. When the imaging device determines that a protective film is attached according to the setting, the imaging device may perform the same processing as when it determines that a protective film is attached by detecting a mark. For example, the imaging device may adjust the acquired non-polarized light captured image so as to reproduce an image captured by the imaging device without a protective film performing non-polarized light imaging.

[0081] Also, when the EXIF information of the non-polarized light image includes the setting information of the imaging device, specifically, information on whether or not imaging was performed with a protective film attached, an image processing apparatus different from the imaging device may determine whether or not a protective film was attached during imaging based on the EXIF information of the non-polarized light image. If a protective film was attached, the non-polarized light image may be adjusted so as to reproduce an image captured by performing imaging without a protective film.

Description of Reference Numerals

[0082] 1: Photographed image, 10: Dermoscopy camera, 11: Lens, 12, 12a: Protective film, 20, 20a, 20b: Processor, 21: Detection means, 22: Identification means, 23: Selection means, 24: Acquisition means, 25: Adjustment means, 26, 28: Judgment means, 27, 29: Warning means, 30: Image processing device

Claims

1. Detection means for detecting a predetermined mark attached to a protective film attached to a lens of the imaging device from an image acquired by the imaging device; When the image in which the mark is detected is a polarized light imaging image acquired by polarized light imaging, based on the position where the mark is detected in the polarized light imaging image, specifying means for specifying the orientation of the protective film with respect to the lens; Adjusting means for adjusting the polarized light imaging image according to the specified orientation so as to reproduce an image taken by the imaging device without the protective film; An image processing apparatus.

2. Adjusting means for adjusting a non-polarized light imaging image acquired by the imaging device performing non-polarized light imaging with a protective film attached to the lens of the imaging device so as to reproduce an image taken by the imaging device performing the non-polarized light imaging without the protective film; An image processing apparatus.

3. In the image processing apparatus according to claim 1, further comprising: A plurality of conversion matrices for converting an image acquired by the imaging device with the protective film by the polarized light imaging into an image acquired by the imaging device without the protective film by the polarized light imaging, each corresponding to a different orientation of the protective film with respect to the lens, and selection means for selecting one conversion matrix based on the specified orientation; The adjusting means adjusts the polarized light imaging image according to the specified orientation based on the one conversion matrix selected by the selection means; An image processing apparatus.

4. In the image processing apparatus according to claim 1 or claim 3, further comprising: Determination means for determining whether an image taken by the imaging device without the protective film can be reproduced from the polarized light imaging image based on the specified orientation; Warning means for issuing a warning based on the result of the determination; Image processing apparatus.

5. In the image processing apparatus according to claim 1 or claim 3, when the image in which the mark is detected is a non-polarized captured image that is not the polarized captured image, the adjustment means adjusts the non-polarized captured image regardless of the orientation of the protective film with respect to the lens so as to reproduce an image captured by the imaging device without the protective film. Image processing apparatus.

6. In the image processing apparatus according to claim 2, further, acquisition means for acquiring a conversion matrix for converting the non-polarized captured image acquired by the imaging device with the protective film by non-polarized imaging into an image acquired by the imaging device without the protective film by non-polarized imaging; the adjustment means adjusts the non-polarized captured image based on the conversion matrix acquired by the acquisition means. Image processing apparatus.

7. In the image processing apparatus according to claim 5, further, acquisition means for acquiring a conversion matrix for converting the non-polarized captured image acquired by the imaging device with the protective film by non-polarized imaging into an image acquired by the imaging device without the protective film by non-polarized imaging; the adjustment means adjusts the non-polarized captured image based on the conversion matrix acquired by the acquisition means. Image processing apparatus.

8. An image processing method performed by an image processing apparatus that processes an image acquired by an imaging device, detecting a predetermined mark attached to a protective film attached to a lens of the imaging device from an image acquired by the imaging device; when the image in which the mark is detected is a polarized captured image acquired by polarized imaging, specifying an orientation of the protective film with respect to the lens based on a position where the mark is detected in the polarized captured image; Adjust the polarized captured image according to the specified orientation so as to reproduce the image captured by the imaging device without the protection film. Image processing method. **Claim 9** On a computer of an image processing device that processes an image acquired by an imaging device, Detect a predetermined mark attached to a protection film attached to a lens of the imaging device from the image acquired by the imaging device, When the image in which the mark is detected is a polarized captured image acquired by polarized imaging, identify the orientation of the protection film with respect to the lens based on the position where the mark is detected in the polarized captured image, Execute a process of adjusting the polarized captured image according to the identified orientation so as to reproduce the image captured by the imaging device without the protection film. Program. **Claim 10** An image processing method performed by an image processing device that processes an image acquired by an imaging device, Adjust an unpolarized captured image acquired by the imaging device performing unpolarized imaging with a protection film attached to the lens of the imaging device so as to reproduce an image captured by the imaging device performing the unpolarized imaging without the protection film. Image processing method. **Claim 11** On a computer of an image processing device that processes an image acquired by an imaging device, Execute a process of adjusting an unpolarized captured image acquired by the imaging device performing unpolarized imaging with a protection film attached to the lens of the imaging device so as to reproduce an image captured by the imaging device performing the unpolarized imaging without the protection film. Program.

Images