Image processing device, program, and image processing method

Abstract

To make it possible to perform image selection more efficiently.SOLUTION: An image processing device includes a processing unit that acquires first information indicating the order of query images Dq set in the order of sizes of target parts in multiple images obtained by photographing each of multiple target parts selected from multiple target parts included in a subject, acquires second information indicating the order of sizes of multiple cut-out images cut out from an entire image obtained by photographing the subject, narrows down the multiple cut-out images based on the first information and the second information, and identifies candidate cut-out images CC that correspond to the query images Dq.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The disclosure of this specification relates to an image processing device, a program, and an image processing method. [Background technology]

[0002] Patent Document 1 describes a technique for quickly extracting an image containing an arbitrary object from a large number of images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2013 / 175608 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, metadata is generated from a query image, and images to be searched are narrowed down based on the metadata. However, it is not necessarily easy to generate metadata useful for narrowing down from individual query images, and there is a risk that subsequent image selection cannot be performed efficiently. An object of one aspect of the present invention is to perform image selection more efficiently. [Means for solving the problem]

[0005] An image processing device according to one embodiment of the present invention includes a processing unit that acquires first information indicating the order of query images set in order of size of a plurality of target areas selected from a plurality of target areas included in a subject in a plurality of images captured of the target areas, acquires second information indicating the order of size of a plurality of cut-out images cut out from an entire image captured of the subject, narrows down the plurality of cut-out images based on the first information and the second information, and identifies candidate cut-out images corresponding to the query image. [Effects of the Invention]

[0006] According to the above aspect, image selection can be performed more efficiently. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating a configuration of a system according to an embodiment of the present invention. [Figure 2] 10 is an example of a flowchart of a pre-processing. [Figure 3] FIG. 10 is a diagram showing an example of a wide-field image. [Figure 4] 10 is an example of a flowchart of an automatic matching process. [Figure 5] 10 is an example of a flowchart illustrating a part of a candidate specification process. [Figure 6] 6 is a diagram illustrating an example of a method for specifying a candidate image in the processing shown in FIG. 5. FIG. [Figure 7] 10 is an example of a flowchart showing the remaining part of the candidate identification process. [Figure 8] 8A and 8B are diagrams illustrating an example of a method for specifying a candidate image in the processing shown in FIG. 7. [Figure 9] 10 is another example of a flowchart illustrating the remainder of the candidate identification process. [Figure 10] 10 is yet another example of a flowchart illustrating the remainder of the candidate identification process. [Figure 11] 10 is an example of a flowchart of a matching support process. DETAILED DESCRIPTION OF THE INVENTION

[0008] System 1 shown in FIG. 1 is a system that manages dermoscopy images D used to assist in the diagnosis of skin tumors, in association with wide-field images W of a subject. More specifically, system 1 manages dermoscopy images D, which are images of a target area, in association with images (cut-out images C) of each target area cut out from wide-field images W. A user of system 1 is, for example, a doctor who diagnoses skin tumors. System 1 includes an information processing device 100 and peripheral devices (display device 10, input device 20, dermoscope-equipped imaging device 30, imaging device 40) connected to information processing device 100.

[0009] The display device 10 is an example of a display unit that displays information to a user, such as a liquid crystal display or an organic electroluminescence (EL) display. The input device 20 is an example of an input unit that accepts user operations. The input device 20 is, for example, a keyboard, a mouse, or a touch panel. A dermoscope is a magnifying glass that can magnify and non-invasively observe a target area suspected of being a malignant skin tumor. The dermoscope-equipped imaging device 30 is a first imaging device that captures an image of the target area and generates a dermoscopy image D of the target area. The imaging device 40 is a second imaging device that captures a wide area of ​​the subject, which is the subject, and generates a wide-field image W, which is a clinical image including multiple target areas. The subject is the patient's skin. The wide-field image W is an image of a wide area of ​​the skin (e.g., the entire back) and is an example of a whole image. The target area is an abnormality that has occurred on the subject, such as a mole or wart on the skin. Dermoscopy image D is an example of a first image, which is a close-up image of the target area taken using a polarizing film or gel to reduce reflection on the skin surface.

[0010] The information processing device 100 is an example of an image processing device and is a general-purpose or dedicated computer including a CPU 101, a GPU 102, and a storage device 103. The CPU 101 and the GPU 102 are processors, i.e., processing units, that execute programs stored in the storage device 103 to function as an acquisition unit and an extraction unit, respectively, and perform various processes described below. It is sufficient for the information processing device 100 to include one or more processors. A single processor may perform the processes, or multiple processors may perform the processes collaboratively. The information processing device 100 does not necessarily need to be directly connected to the dermoscope-equipped imaging device 30 and the imaging device 40. The information processing device 100 may be a server connected to these devices via a network and may perform various processes in response to requests from another terminal operated by a user. The storage device 103 stores programs executed by processors such as the CPU 101 and the GPU 102, as well as data used when executing the programs (wide-field image W, cropped image C, dermoscopy image D), etc. The storage device 103 includes, for example, semiconductor memory that operates as a main storage device such as RAM (Random Access Memory) or ROM (Read Only Memory), and storage that operates as an auxiliary storage device such as SSD (Solid State Drive) or HDD (Hard Disk Drive).

[0011] When a doctor uses system 1 to perform dermoscopy, that is, when observing and evaluating skin using dermoscopy technology, the procedure is, for example, as follows: First, the doctor acquires a wide-field image W using the imaging device 40. That is, the imaging device 40 captures an image of a predetermined area of ​​the patient (for example, the entire back), and a wide-field image W is generated. Then, the doctor selects a particularly suspicious area or an area to be observed in detail (hereinafter referred to as a selected area) from multiple target areas captured in the wide-field image W, and acquires a dermoscopy image D of the selected area using the imaging device 30 with a dermoscope. That is, the selected area is photographed using the imaging device 30 with a dermoscope, and a dermoscopy image D is generated. Finally, the doctor makes a diagnosis based on the dermoscopy image D.

[0012] It is desirable that the generated dermoscopy image D be managed in the system 1 in association with the target area in the wide-field image W. Managing the images in association provides various benefits. For example, it becomes possible to grasp the overall image of the subject (such as the location and distribution of malignant skin tumors) before creating a diagnosis or treatment plan. It also makes it easier to access the dermoscopy image D, which contains more detailed information, from the wide-field image W. Furthermore, by periodically capturing the wide-field image W and associating it with the dermoscopy image D, it becomes possible to detect newly appearing malignant skin tumors and observe the progression of malignant skin tumors over time.

[0013] However, when a large number of target areas are present in a wide-field-of-view image W and many dermoscopy images D are acquired, the number of patterns for matching the target areas in the dermoscopy image D with the wide-field-of-view image W becomes enormous. Therefore, manual matching imposes a heavy burden on the operator. Furthermore, when matching is performed automatically using image features, etc., it is necessary to compare the matching degree for all patterns to determine the optimal matching, which requires a huge amount of calculation and a long processing time. For these reasons, a technology for narrowing down the target areas corresponding to a dermoscopy image D is desired.

[0014] A method for narrowing down and then associating target areas corresponding to a dermoscopy image D, performed by the system 1, will be described below. First, with reference to FIGS. 2 and 3, a pre-processing performed by the system 1 before narrowing down will be described. As shown in FIG. 2, the pre-processing may include, for example, six steps. First, the system 1 performs wide-field imaging (step S1). Here, in accordance with the doctor's operation, the imaging device 40 captures, for example, an image of the patient's entire back, and generates a wide-field image W of the entire back as shown in FIG. 3. As shown in FIG. 3, the wide-field image W contains multiple target areas L, such as moles and warts. The wide-field image W generated by the imaging device 40 is input to the information processing device 100. Next, the system 1 performs dermoscopy imaging (step S2). Here, in accordance with the doctor's operation, the dermoscope-equipped imaging device 30 captures, for example, target areas L that the doctor has determined to be particularly suspicious (selected target areas) from among the multiple target areas L captured in the wide-field image W, and generates multiple dermoscopy images D. It is also possible to generate a dermoscopy image D by capturing images of all target areas captured in the wide-field image W. The multiple dermoscopy images D generated by the dermoscope-equipped photographing device 30 are input to the information processing device 100.

[0015] Next, the system 1 generates a group of cutout images C (step S3). The group of cutout images is an example of a second image group. Here, the information processing device 100 detects an area corresponding to the target portion L from the wide-field image W by image processing, and cuts out each of the detected areas corresponding to the target portion L to generate a plurality of cutout images C. The detection of the area corresponding to the target portion may be performed, for example, using an object detection model that has previously performed machine learning on the target portion, or may be performed using BLOB detection. Here, each image (cutout image C) in the group of cutout images is a rectangular image cut out to match the size of the target portion L, as shown in FIG. 3, for example. However, it is sufficient that the size of the target portion L included in the cutout image C can be identified. The cutout image C may be a fixed-size image that includes the target portion L regardless of the size of the target portion L, and may include size information indicating the size of the target portion L as meta information, for example. The wide-field image W is an image of a roughly planar subject (back) photographed from a certain distance. Therefore, the cropped image C cropped from the wide-field image W can be considered to be an image with a roughly constant magnification regardless of the cropping position. Therefore, the relative relationship of the size of the target portion L in the cropped image C maintains the relative relationship of the size of the actual target portion L. Once the multiple cropped images C are generated, the system 1 aligns the multiple cropped images C (step S4). Here, the information processing device 100 aligns the multiple cropped images C based on the size of each image (cropped image C) in the cropped image group, and acquires a cropped image sequence CS (second image sequence) in which the multiple cropped images C are aligned in order of the size of the target portion L (e.g., largest to smallest).

[0016] Furthermore, the system 1 detects the target area L in the dermoscopy image D (step S5). Here, the information processing device 100 sequentially reads out the multiple dermoscopy images D constituting the dermoscopy image group, detects an area corresponding to the target area L (selected target area) from the dermoscopy image D through image processing, and generates size information indicating the size of the target area. The dermoscopy image group is an example of a first image group. The detection of the area corresponding to the target area may be performed, for example, using a segmentation model (e.g., semantic segmentation) that has previously learned about the target area through machine learning. Alternatively, as in step S3, the detection may be performed using an object detection model, blob detection, or the like. If the dermoscope-equipped photographing device 30 is a contact type that photographs while in contact with the skin, the magnification of the dermoscopy image is approximately constant. In this case, the relative relationship of the size of the target area L in the dermoscopy image maintains the relative relationship of the size of the actual target area L, and therefore, the size information indicating the size of the target area may be generated based on the size of the detected area. On the other hand, if the focal length of the dermatoscope-equipped photographing device 30 is variable, the relative size relationship of the target area L in the dermoscopy image may not necessarily maintain the relative size relationship of the actual target area L. In this case, information about the focal length at the time of photographing can be read from the EXIF ​​file of the dermoscopy image, the size of the target area L in the dermoscopy image can be normalized based on the read information, and size information indicating the size of the target area can be generated based on the normalized size of the target area L. Once the size information for each dermoscopy image is generated, the system 1 arranges the multiple dermoscopy images D (step S6). Here, the information processing device 100 acquires a dermoscopy image sequence DS (first image sequence) in which the multiple dermoscopy images D are arranged in order of size of the target area L (here, in descending order, like the sequence of cropped images) based on the size information generated in step S5. By the pre-processing shown in FIG. 2, the dermoscopy images D and the cropped images C cropped from the wide-field image W are arranged in order of size, respectively.

[0017] A method for associating a dermoscopy image D with a cut-out image C performed by the information processing device 100 will be described below. First, with reference to FIGS. 4 to 8, a process (automatic matching process) in which the information processing device 100 automatically matches and associates a dermoscopy image D with a cut-out image C will be described. In the automatic matching process, as shown in FIG. 4, the information processing device 100 selects a plurality of dermoscopy images D one by one in order as a query image Dq, and associates a cut-out image C with the selected query image Dq. More specifically, first, the information processing device 100 selects a query image Dq (step S10). The query image Dq is an image selected from the plurality of dermoscopy images D to be associated with the cut-out image C. The method for selecting the query image Dq is not particularly limited. The information processing device 100 may select any image as the query image Dq.

[0018] Once the query image Dq is selected, the information processing device 100 then performs a candidate identification process (step S100). The candidate identification process is a specific example of an image processing method performed by the information processing device 100, and is a process of narrowing down the multiple cropped images C generated in step S3 to identify candidates (candidate images) for the cropped image C corresponding to the query image Dq. Details of the candidate identification process will be described later. Once candidate images have been identified by the candidate identification process, the information processing device 100 extracts features of the query image Dq (step S20) and further extracts features of each candidate image (cropped image C) (step S30). The feature extraction in steps S20 and S30 uses a feature extractor that has been trained in advance to extract features similar to those from the dermoscopy image D and the target area L contained in the dermoscopy image D and the cropped image C of the same target area L. The feature extractor may be trained using training data consisting of pairs of dermoscopy images D and cropped images C of the same target area.

[0019] Thereafter, the information processing device 100 determines a corresponding image, which is a cut-out image C corresponding to the query image Dq (step S40). Here, the information processing device 100 identifies a feature closest to the feature of the query image Dq identified in step S20 from the feature amounts extracted in step S30, and determines a candidate image having the identified feature amount as the corresponding image. Thereafter, the information processing device 100 determines whether or not matching has been completed for all dermoscopy images D (step S50), and repeats the above-described process while changing the query image until matching has been completed for all dermoscopy images D. In this way, corresponding images are automatically determined for all dermoscopy images D.

[0020] The candidate identification process will be described in more detail. As shown in FIG. 5, in the candidate identification process, the information processing device 100 first acquires information on the order of query images (step S101). Here, the information processing device 100 acquires information indicating the order of the query image Dq in the dermoscopy image sequence DS (e.g., the (N+1)th from the beginning) as first information. The first information is information indicating the order of the query images Dq set in the order of the sizes of target areas in a plurality of images (dermoscopy images D) obtained by capturing each of a plurality of target areas selected from a plurality of target areas included in a subject. Next, the information processing device 100 determines whether or not there is a matched image set between the dermoscopy image sequence DS and the cut-out image sequence CS (step S102). If there is no matched image set, the information processing device 100 counts the number of images before the query image Dq in the dermoscopy image sequence DS (the number of images with a larger size) based on the first information acquired in step S101 (step S103).

[0021] When the target areas are sorted in descending order of size, the N dermoscopy images D (including the Nth dermoscopy image Dx from the beginning) located before the query image Dq in the dermoscopy image sequence DS shown in FIG. 6, i.e., larger in size than the query image Dq, contain target areas larger than those in the query image Dq. Therefore, the N cutout images C corresponding to the N dermoscopy images D before the query image Dq also contain target areas larger than those in the cutout image C corresponding to the query image Dq, and the cutout image sequence CS contains N or more cutout images C containing target areas larger than those in the cutout image C corresponding to the query image Dq. Therefore, the first N cutout images C (up to the Nth cutout image Cx from the beginning) of the cutout image sequence CS shown in FIG. 6 are not the cutout images C corresponding to the query image Dq. Using this fact, the information processing device 100 excludes a predetermined number of images from the beginning from the cutout image sequence CS based on the number of images counted in step S103 (step S104). The predetermined number is the number of images counted in step S103. However, in consideration of cases where the difference in size is small, the predetermined number may be a number smaller than the number of images counted in step S103.

[0022] The information processing device 100 further counts the number of images in the dermoscopy image sequence DS after the query image Dq based on the information on the order of the images acquired in step S101 (step S105). If the images are arranged in descending order of size, the M dermoscopy images D (including the M-th dermoscopy image Dy from the end) located after the query image Dq in the dermoscopy image sequence DS shown in FIG. 6 contain smaller target areas than those in the query image Dq. Therefore, the M cutout images C corresponding to the M dermoscopy images D after the query image Dq also contain at least smaller target areas than those in the cutout image C corresponding to the query image Dq, and the cutout image sequence CS contains M or more cutout images C containing smaller target areas than those in the cutout image C corresponding to the query image Dq. Therefore, it is unlikely that the last M cutout images C in the cutout image sequence CS shown in FIG. 6 (up to the M-th cutout image Cy from the end) are the cutout images C corresponding to the query image Dq. Using this fact, the information processing device 100 cuts out a predetermined number of images from the end of the dermoscopy image sequence DS based on the number of images counted in step S105 and excludes them from the sequence (step S106). The predetermined number is the number of images counted in step S105. However, taking into consideration cases where the difference in size is small, the predetermined number may be a number smaller than the number of images counted in step S105. By performing the processes of steps S104 and S106, it is possible to exclude from the candidates cut-out images C at least as many as the number of dermoscopy images constituting the dermoscopy image sequence DS (more specifically, a number one smaller than the number of dermoscopy images).

[0023] In steps S103 to S106, the information processing device 100 acquires second information indicating the order of sizes of the multiple cropped images C cropped from the entire image of the subject, and further excludes some images from the cropped image sequence based on the first information and the second information. The information processing device 100 then determines the remaining cropped images C from the cropped image sequence CS that were not excluded in the processes of steps S104 and S106 as candidate images CC (step S107), and terminates the candidate identification process for the first query image Dq. As shown in FIGS. 5 and 6, the information processing device 100 narrows down the multiple cropped images C corresponding to the multiple target portions cropped from the wide-field image W based on the first information of the initial query image Dq and the second information indicating the order of sizes of the cropped image sequence, and identifies candidates for the cropped image C corresponding to the query image Dq. This allows the information processing device 100 to reduce the number of cropped images C to be compared with the query image Dq in the automatic matching process, thereby reducing the amount of calculation and determining the corresponding cropped image C in a short time.

[0024] In the candidate identification process performed on the second and subsequent query images Dq, the information processing device 100 acquires first information in step S101 and then determines in step S102 that there is a matched image set. Thereafter, as shown in Fig. 7, the information processing device 100 acquires correspondence information on the matched image set, i.e., information indicating the correspondence between a dermoscopy image D in the dermoscopy image group (first image group) and a clipped image C in the clipped image group (second image group) for which the correspondence is known (step S111). The information processing device 100 determines whether there is a matched dermoscopy image D before the query image Dq based on the correspondence information acquired in step S111 (step S112). If there is a matched dermoscopy image D before the query image Dq, the information processing device 100 identifies the matched dermoscopy image D that is before the query image Dq in the dermoscopy image sequence DS and is closest to the query image Dq (step S113), and also identifies a matched cut-out image C that corresponds to the identified matched dermoscopy image D (step S114). In the example shown in Fig. 8, there is only one matched dermoscopy image before the query image Dq, so the information processing device 100 identifies that one matched dermoscopy image Db and also identifies a matched cut-out image Cb that corresponds to the identified matched dermoscopy image Db.

[0025] If the target areas are sorted in descending order of size, the matched dermoscopy image Db and the matched cropped image Cb in the dermoscopy image sequence DS shown in FIG. 8 depict target areas larger than those in the query image Dq. Furthermore, a cropped image C positioned in front of the matched cropped image Cb in the cropped image sequence CS depicts an even larger target area. Therefore, none of the images from the top of the cropped image sequence CS shown in FIG. 8 to the matched cropped image Cb are cropped images C corresponding to the query image Dq. Using this fact, the information processing device 100 excludes the matched cropped image Cb identified in step S114 and the cropped images C located before it from the cropped image sequence CS (step S115). That is, the information processing device 100 excludes some images from the cropped image sequence CS based on the first information, the second information, and the correspondence information. The information processing device 100 further counts the number of images located between the query image Dq and the matched dermoscopy image Db in the dermoscopy image sequence DS based on the first information acquired in step S101 and the correspondence information acquired in step S111 (step S116).

[0026] If the target areas are sorted in descending order of size, the N dermoscopy images D located between the query image Dq and the matched dermoscopy image Db in the dermoscopy image sequence DS shown in FIG. 8 also depict target areas larger than those in the query image Dq. Therefore, it is clear that the N cut-out images C corresponding to the N dermoscopy images D located between the query image Dq and the matched dermoscopy image Db also depict target areas larger than those in the cut-out image C corresponding to the query image Dq. Therefore, the N cut-out images C immediately following the matched cut-out image Cb shown in FIG. 8 are not cut-out images C corresponding to the query image Dq. Using this fact, the information processing device 100 excludes a predetermined number of images immediately following the matched cut-out image Cb from candidates for the cut-out image C corresponding to the query image Dq, based on the number of images counted in step S116 (step S117). That is, the information processing device 100 excludes some images from the cut-out image sequence CS based on the first information, the second information, and the correspondence information. The predetermined number is the number of images counted in step S116. However, in consideration of cases where the difference in size is small, the predetermined number may be a number smaller than the number of images counted in step S116.

[0027] The information processing device 100 then determines whether or not there is a matched dermoscopy image D after the query image Dq based on the match information acquired in step S111 (step S118). If there is a matched dermoscopy image D after the query image Dq, the information processing device 100 identifies the matched dermoscopy image D that is after the query image Dq in the dermoscopy image sequence DS and that is closest to the query image Dq (step S119), and also identifies the matched cropped image C that corresponds to the identified matched dermoscopy image D (step S120). In the example shown in FIG. 8, there are two matched dermoscopy images (dermoscopy image Ds1 and dermoscopy image Ds2) after the query image Dq, but the information processing device 100 identifies the matched dermoscopy image Ds1 that is closest to the query image Dq and also identifies the matched cropped image Cs1 that corresponds to the identified matched dermoscopy image Ds1.

[0028] If the target areas are sorted in descending order of size, the matched dermoscopy image Ds1 and the matched cropped image Cs1 in the dermoscopy image sequence DS shown in FIG. 8 depict target areas that are smaller than those in the query image Dq. Furthermore, a cropped image C located after the matched cropped image Cs1 in the cropped image sequence CS depicts an even smaller target area. Therefore, the images from the end of the cropped image sequence CS shown in FIG. 8 to the matched cropped image Cs1 are not cropped images C that correspond to the query image Dq. Using this fact, the information processing device 100 excludes the matched cropped image Cs1 identified in step S120 and any cropped images C that are further back from the cropped image sequence CS (step S121). That is, the information processing device 100 excludes some images from the cropped image sequence CS based on the first information, the second information, and the correspondence information. The information processing device 100 further counts the number of images located between the query image Dq and the matched dermoscopy image Ds1 in the dermoscopy image sequence DS based on the first information acquired in step S101 and the correspondence information acquired in step S111 (step S122).

[0029] If the target areas are sorted in descending order of size, the M dermoscopy images D located between the query image Dq and the matched dermoscopy image Ds1 in the dermoscopy image sequence DS shown in FIG. 8 also contain target areas smaller than those in the query image Dq. Therefore, it is clear that the M cropped images C corresponding to the M dermoscopy images D located between the query image Dq and the matched dermoscopy image Ds1 also contain target areas smaller than those in the cropped image C corresponding to the query image Dq. Therefore, the M cropped images C immediately preceding the matched cropped image Cs1 shown in FIG. 8 are not the cropped images C corresponding to the query image Dq. Using this fact, the information processing device 100 excludes a predetermined number of images immediately preceding the matched cropped image Cs1 from candidates for the cropped image C corresponding to the query image Dq (step S123). That is, the information processing device 100 excludes some images from the cropped image sequence CS based on the first information, the second information, and the correspondence information. The predetermined number is the number of images counted in step S122. However, in consideration of cases where the difference in size is small, the predetermined number may be a number smaller than the number of images counted in step S122.

[0030] The information processing device 100 then determines, as candidate images CC, the clipped images C remaining in the sequence of clipped images CS that were not excluded in the processes of steps S115, S117, S121, and S123 (step S124), and terminates the candidate identification process for the second and subsequent query images Dq. As shown in FIGS. 7 and 8, the information processing device 100 narrows down the multiple clipped images C corresponding to multiple target portions clipped from the wide-field image W based on the first information, second information, and correspondence information of the second and subsequent query images Dq, and identifies candidates for the clipped image C corresponding to the query image Dq. As a result, by using the first information, second information, and correspondence information, the information processing device 100 can more efficiently reduce the number of clipped images C to be compared with the query image Dq in the automatic matching process, thereby reducing the amount of calculation and determining the corresponding clipped image C in a short time. In particular, since the correspondence information can be used to simultaneously exclude images that are before (after) the matched cut-out image C that is closest to the query image, the more correspondence is established, the more efficiently the cut-out image C can be excluded.Furthermore, by using the number of images between the matched image and the query image to exclude cut-out images, the cut-out image C can be excluded even more efficiently.

[0031] The above describes an example in which the order of the cut-out images C that can be excluded from the sequence of cut-out images CS is identified based on the first information, the second information, and the correspondence information, thereby narrowing down the cut-out images C. However, the information processing device 100 may further narrow down the cut-out images C by using the size of the target portion of the cut-out image C that corresponds to the query image Dq, which is estimated based on the correspondence information. For example, in the candidate identification process, if there is a matched image set, the information processing device 100 may perform the process shown in FIG. 9 instead of the process shown in FIG. 7.

[0032] Specifically, the information processing device 100 performs a process of excluding a portion of the cut-out image C based on the first information, the second information, and the correspondence information, and then acquires size information indicating the size of the target area included in the cut-out image C corresponding to the query image Dq (step S125). Here, the information processing device 100 estimates the size of the target area included in the cut-out image C corresponding to the query image Dq based on the correspondence information, and acquires size information indicating the estimated size. If the size of the target area included in the matched dermoscopy image D indicated by the correspondence information is Sd, the size of the target area included in the matched cut-out image C indicated by the correspondence information is Sc, and the size of the target area included in the query image Dq is Sq, the information processing device 100 may calculate the size S of the target area included in the cut-out image C corresponding to the query image Dq using, for example, the formula S=(Sc / Sd)×Sq.

[0033] When the size information is acquired, the information processing device 100 narrows down the cut-out images C based on the size information (step S126). Here, the information processing device 100 determines candidate images CC by narrowing down the cut-out images C to images having target areas of sizes within a predetermined range centered on the size indicated by the size information. This allows the cut-out images C to be narrowed down efficiently even when, for example, the number of dermoscopy images D is small compared to the number of cut-out images C.

[0034] 9 illustrates an example in which the cutout images C are narrowed down to a certain extent based on the first information, the second information, and the correspondence information, and then further narrowed down using the size of the target portion of the cutout image C corresponding to the query image Dq estimated from the correspondence information. However, if correspondence information is obtained, the estimated size of the target portion of the cutout image C corresponding to the query image Dq may be used alone for narrowing down the cutout images. For example, in the candidate identification process, if there is a matched image set, the information processing device 100 may perform the process shown in FIG. 10 instead of the process shown in FIG. 9. Specifically, upon acquiring the correspondence information (step S111), the information processing device 100 may acquire size information indicating the size of the target portion included in the cutout image C corresponding to the query image Dq (step S125) without performing the processes from step S112 to step S124, and narrow down the cutout images C based on the acquired size information (step S126). According to the process shown in FIG. 10, it is possible to expect substantially the same effect as the process shown in FIG. 9, particularly when the number of dermoscopy images D is small compared to the number of cutout images C.

[0035] Next, a process (matching support process) performed by the information processing device 100 to support manual matching when manually matching and associating a dermoscopy image D and a cut-out image C will be described with reference to FIG. 11 . Similar to the automatic matching process, the matching support process is performed after the pre-processing shown in FIG. 2 . The information processing device 100 first displays a list of the cut-out image C and the dermoscopy image D on the display device 10 (steps S210 and S220). After that, the information processing device 100 accepts a user's selection operation of a query image Dq and selects a query image Dq from the dermoscopy images D listed in step S220 (step S230). Once the query image Dq is selected, the information processing device 100 performs a candidate identification process to extract candidates for the cut-out image C corresponding to the query image Dq (step S300), and displays the extracted candidate images CC in a list on the display device 10 (step S240). The candidate identification process in step S300 is the same as the candidate identification process shown in FIG. 4 .

[0036] The user finds and selects a cut-out image C corresponding to the query image Dq from the cut-out images C narrowed down to a certain extent by the candidate identification process of step S300. When the user performs an operation to select a specific image from the candidate images CC displayed in the list, the information processing device 100 accepts the user's selection operation and determines the image selected from the candidate images CC (cut-out images C) displayed in the list in step S240 as the corresponding image (step S260). Thereafter, the information processing device 100 determines whether or not association has been completed for all dermoscopy images D (step S270), and repeats the processes of steps S230 to S270 until association has been completed for all dermoscopy images D. As a result, corresponding images are determined for all dermoscopy images D.

[0037] The above-described embodiments are illustrative examples provided to facilitate understanding of the invention. The present invention is not limited to the above-described embodiments, and should be understood to encompass various modifications and alternative forms of the above-described embodiments. For example, it will be understood that the above-described embodiments can be embodied by modifying the components without departing from the spirit of the invention. It will also be understood that various embodiments can be implemented by appropriately combining multiple components disclosed in the above-described embodiments. Furthermore, it will be understood by those skilled in the art that various embodiments can be implemented by deleting some components from all of the components shown in the embodiments, or by adding some components to the components shown in the embodiments.

[0038] In the above-described embodiment, a skin image is used as an example. However, the application field may be any field in which a detailed image captured using a different imaging method or device from that of the entire image is efficiently associated with the entire image. The target image is not limited to a skin image, and may be applied to, for example, a dental field, or may be applied to an industrial field other than the medical field, such as cracks in buildings. In addition, in the above-described embodiment, size information is obtained from the correspondence between a pair of dermoscopy images D and cut-out images C identified from the correspondence information, and the cut-out images C corresponding to the query image Dq are narrowed down based on the size information. However, two or more pieces of size information may be obtained from the correspondence between two or more pairs of dermoscopy images D and cut-out images C, and the cut-out images C corresponding to the query image Dq may be narrowed down based on the two or more pieces of size information.

[0039] In the above-described embodiment, in the process of determining a cutout image C corresponding to a query image Dq, which is performed after extracting candidate cutout images C, an example has been described in which the feature amounts of the query image Dq are compared with the feature amounts of each cutout image C, and the cutout image C having feature amounts closest to the feature amounts of the query image Dq is determined as the cutout image C corresponding to the query image Dq. That is, an example has been described in which the query images Dq are read one by one, and the corresponding cutout images C are determined one by one. However, the method of determining the corresponding cutout images C is not limited to this example. For example, if there are L query images Dq, each pattern of correspondences consisting of L sets of query images Dq and cutout images C may be evaluated, and L cutout images C corresponding to each query image Dq may be determined from the correspondences consisting of L sets of query images Dq and cutout images C that form the best pattern. This allows the matching process to be optimized globally rather than individually, thereby expecting more accurate matching results. For the evaluation of each pattern for overall optimization, for example, the sum of the differences in the features between the query image Dq and the cropped image C in the L set may be used, and the pattern with the smallest sum of the differences in the features may be determined as the best pattern. Furthermore, in addition to the differences in the features, the differences in the order within the sequence of images sorted by size may also be used to evaluate each pattern, and the differences in the features and the differences in the order may be alpha blended as shown in equation (1) to achieve a desired balance. In this case, too, the pattern with the smallest sum may be determined as the best pattern.

number

[0040] Here, α is a value between 0 and 1. i is the feature of the i-th set of dermoscopy images D. fc i is the feature value of the i-th set of cutout images C. S(fd i , fc i ) is the difference in the feature amount between the i-th set of dermoscopy images D and the i-th cut-out image C. iis the order (for example, the order from the beginning) of the i-th set of dermoscopy images D in the dermoscopy image sequence DS. i is the order (for example, the order from the beginning) of the i-th cutout image C in the cutout image sequence CS. i , oc i ) is the difference in rank between the i-th dermoscopy image D and the i-th extracted image C. [Explanation of symbols]

[0041] C, Cb, Cs1, Cs2, Cx, Cy: cropped images, CC: candidate images, CS: cropped image sequence, D, Db, Ds1, Ds2, Dx, Dy: dermoscopy images, DS: dermoscopy image sequence, Dq: query image, L: target area, W: wide-field image

Claims

1. acquiring first information indicating an order of query images set in order of size of a plurality of target portions selected from a plurality of target portions included in a subject in a plurality of images obtained by capturing the plurality of target portions; acquiring second information indicating an order of sizes of a plurality of cut-out images cut out from an entire image of a photographed subject; Narrowing down the plurality of cutout images based on the first information and the second information, and identifying a cutout image candidate corresponding to the query image. An image processing device comprising a processing unit.

2. 2. The image processing device according to claim 1, The processing unit acquiring correspondence information indicating correspondence between images in a first image group consisting of the plurality of images and images in a second image group consisting of the plurality of cropped images, the correspondence between the images being known; The plurality of cutout images are narrowed down based on the first information, the second information, and the correspondence information to identify cutout image candidates corresponding to the query image. Image processing device.

3. 3. The image processing device according to claim 2, The processing unit acquiring size information indicating a size of a target portion included in a cut-out image corresponding to the query image estimated based on the correspondence information; The plurality of cut-out images are narrowed down based on the first information, the second information, the correspondence information, and the size information, and a cut-out image candidate corresponding to the query image is identified. Image processing device.

4. 2. The image processing device according to claim 1, The processing unit based on the number of images located before the query image in a first image sequence in which the plurality of images are arranged in order of size of the target portion, excluding a predetermined number of images from the beginning of a second image sequence in which the plurality of cut-out images are arranged in order of size of the target portion from candidates for cut-out images corresponding to the query image; A predetermined number of images from the end of the second image sequence are excluded from candidates for cut-out images corresponding to the query image based on the number of images located after the query image in the first image sequence. Image processing device.

5. 3. The image processing device according to claim 2, The processing unit Excluding a matched cutout image, which is a cutout image that constitutes the correspondence indicated by the correspondence information, from cutout image candidates that correspond to the query image; When a matched image that constitutes the correspondence indicated by the correspondence information is located ahead of the query image in a first image sequence in which the plurality of images are arranged in order of size of the target portion, an image that is ahead of the matched cut-out image in a second image sequence in which the plurality of cut-out images are arranged in order of size of the target portion is excluded from candidates for cut-out images that correspond to the query image; When the matched image is located behind the query image in the first image sequence, images behind the matched cut-out image in the second image sequence are excluded from cut-out image candidates corresponding to the query image. Image processing device.

6. 6. The image processing device according to claim 5, The processing unit When the matched image is located before the query image in the first image sequence, a predetermined number of images immediately following the matched cut-out image in the second image sequence are excluded from candidates for cut-out images corresponding to the query image, based on the number of images located between the query image and the matched image in the first image sequence. When the matched image is located after the query image in the first image sequence, a predetermined number of images immediately before the matched cut-out image in the second image sequence are excluded from candidates for cut-out images corresponding to the query image, based on the number of images located between the query image and the matched image in the first image sequence. Image processing device.

7. 2. The image processing device according to claim 1, The target area is a lesion. Image processing device.

8. acquiring correspondence information indicating correspondence between a plurality of images obtained by photographing a plurality of target parts selected from a plurality of target parts included in a subject and a plurality of clipped images corresponding to the plurality of target parts clipped from an entire image obtained by photographing the subject, the clipped images having a known correspondence relationship; and acquiring size information indicating a size of a target portion included in a cut-out image corresponding to a query image selected from the plurality of images, the size information being estimated based on the correspondence information; The plurality of cutout images are narrowed down based on the size information, and a cutout image candidate corresponding to the query image is identified. Equipped with a processing unit 1. An image processing device comprising:

9. On the computer, acquiring first information indicating an order of query images set in order of size of a plurality of target portions selected from a plurality of target portions included in a subject in a plurality of images obtained by capturing the plurality of target portions; acquiring second information indicating an order of sizes of a plurality of cut-out images cut out from an entire image of a photographed subject; Narrowing down the plurality of cutout images based on the first information and the second information, and identifying a cutout image candidate corresponding to the query image. A program characterized by executing a process.

10. The computer acquiring first information indicating an order of query images set in order of size of a plurality of target portions selected from a plurality of target portions included in a subject in a plurality of images obtained by capturing the plurality of target portions; acquiring second information indicating an order of sizes of a plurality of cut-out images cut out from an entire image of a photographed subject; Narrowing down the plurality of cutout images based on the first information and the second information, and identifying a cutout image candidate corresponding to the query image. An image processing method comprising:

Images