Image search device, method and program

Abstract

According to the present invention, a processor: receives, as input, opinion information, representing at least one opinion for which a search is desired, about a query base image that forms the basis of a query image and includes a portion for which a search is desired, thereby deriving a query image having the opinion appended; derives at least one appended opinion feature amount representing an image feature of the appended opinion; and derives a query normal feature amount representing an image feature of a normal region included in the portion of the query base image. The degrees of similarity between the query image and a plurality of reference images including opinions are derived on the basis of a comparison of the appended opinion feature amount and the query normal feature amount with reference to an image database in which the plurality of reference images are registered, and the reference images similar to the query image are extracted as similar images from the image database on the basis of the degrees of similarity.

Description

[Technical field]

[0001] The present disclosure relates to an image search device, method, and program. [Background technology]

[0002] In recent years, diagnosis is being made by using medical images acquired by medical devices such as CT (Computed Tomography) devices and MRI (Magnetic Resonance Imaging) devices. When making a diagnosis using an image, past medical images similar to cases of findings contained in the medical image to be diagnosed are often referred to. A large number of past medical images are stored in an image database called PACS (Picture Archiving and Communication Systems) installed in a hospital or the like. For this reason, a method for searching past medical images similar to a target medical image has been proposed. For example, JP 2003-025723 A proposes a method of extracting color, texture, shape, etc. as features from a query image that is the source of the search, and searching for a reference image having features similar to the extracted features. In addition, JP 2018-165926 A proposes a method of deriving features of a query image using a convolutional neural network, and searching for a reference image having features similar to the derived features. Summary of the Invention [Problem to be solved by the invention]

[0003] A query image is necessary to search for an image. However, if a query image containing the desired findings is not available, the search itself cannot be performed. In addition, in clinical practice, there is a demand for comparative interpretation of different diseases present in the same location. For example, assume a case where a cystic lesion is found at a specific anatomical location of the brain, and image diagnosis is performed using an MRI image of the head suspected of having a brain tumor or a brain abscess. In this case, there is a demand for searching a database for MRI images containing a brain tumor and a brain abscess at a specific anatomical location, respectively, and for making a final diagnosis on the image to be diagnosed after confirming the unique features of each disease.

[0004] In such a case, a user such as a doctor who is searching for images must search through the images stored in the database one by one to find the query image. However, this is not realistic because it requires a lot of work.

[0005] The present disclosure has been made in consideration of the above circumstances, and has an object to enable a user to easily search for a desired image. [Means for solving the problem]

[0006] The image search device according to the present disclosure includes at least one processor, The processor deriving a query image with a finding added thereto by receiving an input of finding information representing at least one finding desired to be searched for a query base image that is a source of the query image and includes a region desired to be searched; deriving at least one additional finding feature representing an image feature for the additional finding; Deriving at least one query normal feature representing an image feature for a normal region included in the site in the query base image; refer to an image database in which a plurality of reference images including findings are registered, the plurality of reference images being associated with at least one reference finding feature representing an image feature for the finding included in each of the reference images and at least one reference normal feature representing an image feature for an image when the finding included in each of the reference images is a normal region, and derive a similarity between the query image and each of the plurality of reference images based on a comparison of the additional finding feature and the query normal feature with the reference finding feature and the reference normal feature; Based on the similarity, reference images similar to the query image are extracted from the image database as similar images.

[0007] In the image search device according to the present disclosure, the finding information may include an area according to the type of finding.

[0008] In addition, in the image search device according to the present disclosure, the processor derives an additional finding feature by encoding the added finding; The query normal feature may be derived by encoding the query base image.

[0009] Furthermore, in the image search device according to the present disclosure, the query base image may be a standard image in which the part desired to be searched is made up of only normal areas.

[0010] In addition, the image search device according to the present disclosure includes a storage device that stores at least one finding feature vector representing representative image features for a finding and a normal feature vector representing representative image features for a normal region; the processor derives an additional finding feature vector representing image features for the added finding, quantizes the additional finding feature vector by replacing it with a finding feature vector among the finding feature vectors that has a minimum difference from the additional finding feature vector, and derives an additional finding feature amount; The method may derive a query normal feature vector that represents image features for a normal region included in a part in the query base image, and quantize the query normal feature vector by replacing it with a normal feature vector among the normal feature vectors that has a minimum difference from the query normal feature vector, thereby deriving a query normal feature.

[0011] In the image search device according to the present disclosure, the query base image is an image including a finding, The query normal feature may represent image features of an image when the findings included in the query base image are considered to be normal regions.

[0012] In addition, the image search device according to the present disclosure includes a storage device that stores at least one finding feature vector representing a representative image feature of a finding, and a normal feature vector representing a representative image feature of an image when the included finding is assumed to be a normal region; the processor derives an additional finding feature vector representing image features for the added finding, quantizes the additional finding feature vector by replacing it with a finding feature vector among the finding feature vectors that has a minimum difference from the additional finding feature vector, and derives an additional finding feature amount; A query normal feature vector representing image features of the query base image in the case where a region including the finding in the query base image is assumed to be a normal region may be derived, and the query normal feature vector may be quantized by replacing the query normal feature vector with a normal feature vector among the normal feature vectors that has a minimum difference from the query normal feature vector, thereby deriving a query normal feature.

[0013] In addition, in the image search device according to the present disclosure, the processor may derive the additional finding feature by using an encoding learning model that has been trained to derive the additional finding feature when the added finding is input.

[0014] In addition, in the image search device according to the present disclosure, the processor may derive the query normal features using an encoding learning model that has been trained to derive the query normal features when a query base image is input.

[0015] In the image search device according to the present disclosure, the reference finding feature amount and the reference normal feature amount may be derived by encoding the reference image.

[0016] Furthermore, in the image search device according to the present disclosure, a combination of the reference finding feature amount and the reference normal feature amount may represent an image feature of the reference image.

[0017] In addition, in the image search device according to the present disclosure, the reference finding feature amount is derived by quantizing a feature vector representing an image feature regarding the abnormality of the finding included in the reference image by replacing the feature vector with a first feature vector having a minimum difference from the image feature regarding the abnormality of the finding, among at least one first feature vector representing a representative image feature regarding the abnormality of the finding; The reference normal feature may be derived by quantizing a feature vector representing image features of the reference image when the finding is considered to be in a normal region, by replacing the feature vector with a second feature vector that has a minimum difference from the image features of the reference image when the finding is considered to be in a normal region, among at least one second feature vector representing representative image features of the image when the finding is considered to be in a normal region.

[0018] In addition, in the image search device according to the present disclosure, the reference finding feature and the reference normal feature may be derived using an encoding learning model that has been trained to derive the reference finding feature and the reference normal feature when a reference image is input.

[0019] The image search method according to the present disclosure includes: receiving input of finding information indicating at least one finding desired to be searched for a query base image that is a source of a query image and includes a region desired to be searched; deriving a query image to which the finding is added; deriving at least one additional finding feature representing an image feature for the additional finding; Deriving at least one query normal feature representing an image feature for a normal region included in the site in the query base image; refer to an image database in which a plurality of reference images including findings are registered, the plurality of reference images being associated with at least one reference finding feature representing an image feature for the finding included in each of the reference images and at least one reference normal feature representing an image feature for an image when the finding included in each of the reference images is a normal region, and derive a similarity between the query image and each of the plurality of reference images based on a comparison of the additional finding feature and the query normal feature with the reference finding feature and the reference normal feature; Based on the similarity, reference images similar to the query image are extracted from the image database as similar images.

[0020] The image search method according to the present disclosure may be provided as a program for causing a computer to execute the method. Effect of the Invention

[0021] According to the present disclosure, a user can easily search for a desired image. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a medical information system to which an image search device according to an embodiment of the present disclosure is applied; [Diagram 2] FIG. 1 is a diagram showing a schematic configuration of an image processing system including an image search device according to an embodiment of the present invention. [Diagram 3] A functional configuration diagram of an image processing system including an image search device according to the present embodiment. [Figure 4] Diagram showing the search screen [Diagram 5] A diagram showing a search screen from which a query image is derived. [Figure 6] Conceptual diagram of processing performed by the image encoding unit and the image decoding unit [Figure 7] Conceptual diagram of processing performed by the image encoding unit and the image decoding unit [Figure 8] FIG. 1 is a diagram for explaining replacement with a first feature vector; [Figure 9] A diagram showing an example of training data used for learning [Figure 10] A diagram showing an example of training data used for learning [Figure 11] A diagram for explaining how features are derived from a query image Q0 when searching for similar images. [Figure 12] Diagram showing search result list [Figure 13] A diagram showing a search screen with search results [Figure 14] A flowchart showing the learning process performed in this embodiment. [Figure 15] A flowchart showing an image search process performed in this embodiment. [Figure 16] FIG. 1 shows a query-based image with findings. [Figure 17] FIG. 17 illustrates a query image derived by adding a finding to a query base image including a finding as shown in FIG. 16. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. First, the configuration of a medical information system to which an image search device according to this embodiment is applied will be described. Fig. 1 is a diagram showing a schematic configuration of a medical information system. In the medical information system shown in Fig. 1, a computer 1 including an image search device according to this embodiment, an imaging device 2, and an image storage server 3 are connected in a communicable state via a network 4.

[0024] The computer 1 includes the image search device according to this embodiment, and has the image search program of this embodiment installed. The computer 1 may be a workstation or personal computer operated directly by a doctor who performs a diagnosis, or may be a server computer connected to them via a network. The image search program is stored in a storage device of a server computer connected to the network, or in a network storage, in a state accessible from the outside, and is downloaded and installed in the computer 1 used by the doctor upon request. Alternatively, the program is distributed by recording on a recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory), and is installed in the computer 1 from the recording medium.

[0025] The imaging device 2 is a device that captures an image of a diagnostic target part of a subject to generate a three-dimensional image representing the part, and specifically includes a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, etc. The three-dimensional image composed of a plurality of slice images generated by the imaging device 2 is transmitted to and stored in the image storage server 3. In this embodiment, the diagnostic target part of the subject, that is, the patient, is the brain, and the imaging device 2 is an MRI device that generates an MRI image of the subject's head including the brain as a three-dimensional image.

[0026] The image storage server 3 is a computer that stores and manages various data, and is equipped with a large-capacity external storage device and database management software. The image storage server 3 communicates with other devices via a wired or wireless network 4, and transmits and receives image data and the like. Specifically, various data including image data of three-dimensional images generated by the imaging device 2 is acquired via the network, and stored and managed in a recording medium such as a large-capacity external storage device. The storage format of the image data and the communication between the devices via the network 4 are based on protocols such as DICOM (Digital Imaging and Communication in Medicine). The image storage server 3 also stores teacher data, which will be described later.

[0027] In this embodiment, an image database DB is stored in the image storage server 3. In the image database DB, a plurality of images including various diseases of various parts of the human body are registered as reference images. For example, images of the brain including brain tumors, cerebral hemorrhage, cerebral infarction, etc., and images of the lungs including lung cancer, etc. are registered as reference images. The image database DB will be described later. In this embodiment, the reference image is also a three-dimensional image consisting of a plurality of slice images.

[0028] Next, an image search device according to this embodiment will be described. FIG. 2 describes the hardware configuration of an image processing system including the image search device according to this embodiment. As shown in FIG. 2, the image search device 10 according to this embodiment includes a CPU (Central Processing Unit) 11, a non-volatile storage 13, and a memory 16 as a temporary storage area. The image search device 10 also includes a display 14 such as a liquid crystal display, an input device 15 such as a keyboard and a mouse, and a network I / F (InterFace) 17 connected to a network 4. The CPU 11, the storage 13, the display 14, the input device 15, the memory 16, and the network I / F 17 are connected to a bus 18. The CPU 11 is an example of a processor in this disclosure.

[0029] Storage 13 is realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. Storage 13 as a storage medium stores image search program 12. CPU 11 reads image search program 12 from storage 13, expands it in memory 16, and executes the expanded image search program 12.

[0030] Next, the functional configuration of the image retrieval device according to this embodiment will be described. Fig. 3 is a functional configuration diagram of an image processing system including the image retrieval device according to this embodiment. As shown in Fig. 3, the image retrieval device 10 according to this embodiment includes an information acquisition unit 20, a query image derivation unit 21, an image encoding unit 22, an image decoding unit 23, a learning unit 24, a similarity derivation unit 25, an extraction unit 26, and a display control unit 27. The image encoding unit 22 includes a first feature derivation unit 22A, a second feature derivation unit 22B, and a third feature derivation unit 22C. The image decoding unit 23 includes a segmentation unit 23A, a first reconstruction unit 23B, and a second reconstruction unit 23C.

[0031] When the CPU 11 executes the image search program 12, the CPU 11 functions as an information acquisition unit 20, a query image derivation unit 21, a first feature amount derivation unit 22A, a second feature amount derivation unit 22B, a third feature amount derivation unit 22C, a similarity derivation unit 25, an extraction unit 26, and a display control unit 27. When the CPU 11 executes a learning program (not shown), the CPU 11 functions as a first feature amount derivation unit 22A, a second feature amount derivation unit 22B, a third feature amount derivation unit 22C, a segmentation unit 23A, a first reconstruction unit 23B, a second reconstruction unit 23C, and a learning unit 24.

[0032] In response to an instruction from the operator via the input device 15, the information acquiring unit 20 acquires a query base image for deriving a query image to be searched, which will be described later, from the image storage server 3. The query base image includes a region desired to be searched. In response to an instruction from the operator via the input device 15, the information acquiring unit 20 also acquires from the image storage server 3 a plurality of teacher data for the learning unit 24 to learn an encoding learning model in the image encoding unit 22 and a decoding learning model in the image decoding unit 23, as will be described later.

[0033] In addition, in the case where the query base image and the training data have already been stored in the storage 13, the information acquisition unit 20 may acquire the query base image and the training data from the storage 13.

[0034] The query image derivation unit 21 derives a query image. When deriving a query image, first, the display control unit 27 displays a search screen on the display 14. Fig. 4 is a diagram showing the search screen. As shown in Fig. 4, the search screen 40 includes a first display area 41 serving as a working area for deriving and displaying a query image, a second display area 42 for selecting a finding, a third display area 43 for displaying the search results, and a search execution button 44.

[0035] A query base image B0 including a region desired to be searched is displayed in the first display area 41. As shown in FIG. 4, the query base image B0 is, for example, a standard image in which the region desired to be searched is composed only of normal regions. In this embodiment, since the region desired to be searched is the brain, the query base image B0 is a standard brain image. Examples of standard images include a standard human body atlas image, a representative real image of a healthy subject, an average image obtained by averaging real images of multiple healthy subjects, and an artificially generated image. Note that the query base image B0 may originally include findings, but the query base image B0 including findings will be described later.

[0036] The query base image B0 is acquired by the information acquisition unit 20 from the image storage server 3 or the storage 13 when the user gives an instruction using the input device 15 on an image acquisition screen (not shown). The query base image B0 is a three-dimensional image composed of a plurality of slice images such as an MRI image. Therefore, the user can use the mouse wheel of the input device 15 to sequentially switch and display the slice images displayed in the first display area 41.

[0037] In the second display area 42, a list of findings that can be added to the query base image B0 is displayed. In this embodiment, the list of findings includes three types of findings, edema 42A, contrast effect 42B, and necrosis 42C, and areas corresponding to these three types of findings can be added to the query base image B0. In FIG. 4, for the sake of explanation, white is assigned to edema 42A, diagonal hatching is assigned to contrast effect 42B, and black is assigned to necrosis 42C, but it is preferable to assign colors. Also, any pattern may be assigned.

[0038] The user first selects a slice image to which a finding is to be added. The slice image to which a finding is to be added is also referred to as a query base image B0. Next, the user selects one of the findings from the list of findings displayed in the second display area 42 with the mouse cursor. Then, the user can add the finding to the query base image B0 by drawing an area corresponding to the type of the selected finding at a desired position on the query base image B0. Also, the finding can be added by drawing an area corresponding to the type of finding by superimposing different findings.

[0039] Fig. 5 is a diagram showing a search screen including a query base image B0 to which a finding has been added. In Fig. 5, a finding 45 is added to the left region (right brain region) of the query base image B0, in order from the outside, in which regions corresponding to three types of findings, edema, contrast effect, and necrosis, are superimposed. The query base image B0 to which the finding 45 has been added is used for search, and therefore becomes a query image Q0. Note that in this embodiment, the finding 45 is added to the query base image B0 to derive the query image Q0, and the query image Q0 is an image in which the query base image B0 and an image including only the region of the finding 45 (hereinafter referred to as an added finding label image) F0 are superimposed.

[0040] Here, the derived query image Q0 is input to the image encoding unit 22 when performing a similar image search as described later, and the query base image B0 and the additional finding label image F0 are encoded separately. In the following description, the query image Q0 derived by the query image derivation unit 21 and the query base image B0 and the additional finding label image F0 constituting the query image Q0 are collectively referred to as a target query image.

[0041] When performing a similar image search, not only the query image Q0 derived by the query image derivation unit 21 but also an image originally including a finding, which is acquired by imaging a subject having an abnormality, is input as a reference image to the image encoding unit 22 and encoded. Meanwhile, in this embodiment, there are cases where the reference image does not include a finding.

[0042] The target query image is encoded by the second feature derivation unit 22B and the third feature derivation unit 22C constituting the image encoding unit 22. The reference image is encoded by the first feature derivation unit 22A and the second feature derivation unit 22B constituting the image encoding unit 22. First, the encoding of the target query image will be described.

[0043] The second feature derivation unit 22B that constitutes the image encoding unit 22 derives at least one query normal feature that represents image features for a normal region included in a part in the query base image B0 as a second feature by encoding a query base image B0 of the target query images.

[0044] In this embodiment, when a finding is included in a query base image B0 of the target query images, the second feature derivation unit 22B constituting the image encoding unit 22 encodes the query base image B0 to derive, as a second feature, at least one query normality feature that represents an image feature of an image in the case where the finding included in the query base image B0 is a normal area.

[0045] The third feature derivation unit 22C constituting the image encoding unit 22 derives at least one additional finding feature representing an image feature regarding the abnormality of the finding added to the query base image B0 as a third feature by encoding the additional finding label image F0 of the target query image.

[0046] In this embodiment, when the additional finding label image F0 is not included in the query image Q0 of the target query images, the additional finding feature derived as the third feature by the third feature derivation unit 22C becomes information representing that no finding is included.

[0047] For this purpose, the second feature derivation unit 22B and the third feature derivation unit 22C have an encoder and a latent model as a coding learning model that are trained to derive the second feature and the third feature, respectively, when a target query image is input. The encoder and the latent model as a coding learning model will be described later.

[0048] Next, the encoding of the reference image will be described. The first feature derivation unit 22A constituting the image encoding unit 22 derives at least one reference finding feature representing an image feature regarding the abnormality of a finding included in a target image as a first feature by encoding the reference image. Note that in this embodiment, there are cases where the reference image does not include a finding. In such a case, the reference finding feature derived by the first feature derivation unit 22A as the first feature is information representing that no finding is included.

[0049] The second feature derivation unit 22B constituting the image encoding unit 22 derives, as a second feature, at least one reference normal feature representing an image feature for an image in the case where the findings included in the reference image are normal regions, by encoding the reference image. In addition, when the reference image does not include findings, the second feature derivation unit 22B constituting the image encoding unit 22 derives, as a second feature, at least one reference normal feature representing an image feature for a normal region included in a site in the reference image.

[0050] For this purpose, the first feature derivation unit 22A and the second feature derivation unit 22B have an encoder and a latent model as a coding learning model that have been trained to derive the first feature and the second feature, respectively, when a reference image is input. In this embodiment, the first feature derivation unit 22A and the second feature derivation unit 22B have a common coding learning model. The encoder and the latent model as a coding learning model will be described later.

[0051] Here, the additional finding feature derived as the third feature from the target query image and the reference finding feature derived as the first feature from the reference image both represent image features regarding the abnormality of findings included in the image. Therefore, by comparing the additional finding feature with the reference finding feature, it is possible to derive the similarity of image features regarding the abnormality of findings included in the images between the target query image and the reference image.

[0052] On the other hand, the query normal feature derived as the second feature from the target query image and the reference normal feature derived as the second feature from the reference image both represent image features for an image in which the findings included in the image are in a normal region. Therefore, by comparing the query normal feature and the reference normal feature, it is possible to derive the similarity of image features for an image in which the findings included in the image are in a normal region between the target query image and the reference image.

[0053] Therefore, by comparing the additional finding feature derived as the third feature from the target query image and the query normal feature derived as the second feature with the reference finding feature derived as the first feature from the reference image and the reference normal feature derived as the second feature, it is possible to derive a feature that combines image features regarding the abnormality of the finding included in the image and image features regarding the case where the finding included in the image is in a normal region, that is, the similarity of the images when a desired finding is included in a desired normal region, between the query image and the reference image.

[0054] On the other hand, the teacher data used in learning the encoding learning model and the decoding learning model consists of a teacher image and a teacher label image. The teacher label image represents an area of ​​the teacher image according to the type of finding. In the following description, the query base image B0, the reference image, and the teacher image as the image information are collectively referred to as the target image. Also, the additional finding label image F0, the teacher label image, and the finding label image V0 as the label image information are referred to as the target label image.

[0055] In this embodiment, the target image includes a brain, and the findings are regions determined according to the type of brain disease, such as brain tumor, cerebral infarction, cerebral hemorrhage, etc. The target label image represents a region in the target image determined according to the type of brain disease.

[0056] Here, the first feature amount represents an image feature regarding the abnormality of findings contained in the target image, i.e., an image feature that enables discrimination of areas determined according to the type of brain disease, such as brain tumor, cerebral infarction, or cerebral hemorrhage, as findings from normal tissue.

[0057] The second feature amount represents the image feature of the image when the findings in the target image are normal regions. Therefore, the second feature amount represents the image feature of the findings in the target image, i.e., the disease region, which is interpolated by the image feature of the region when the disease does not exist, particularly the image feature of normal brain tissue. Therefore, the second image feature represents the image feature of the image when all the brain tissue in the target image is normal.

[0058] The third feature amount represents image features that should be possessed by a region in the target image, which is indicated by the target labeled image and is determined according to the type of brain disease.

[0059] The combination of the first feature amount and the second feature amount may represent the image feature of the target image, particularly the image feature of the brain including the region determined according to the type of disease. In this case, the first feature amount represents the image feature regarding the abnormality of the findings included in the target image, but represents the image feature representing the difference from the image feature when the findings included in the target image are normal regions. In this embodiment, since the findings are brain diseases, the first feature amount represents the image feature representing the difference from the image feature of the image in the state where the brain in the target image is all normal tissue. This makes it possible to obtain the image feature regarding the abnormality of the region determined according to the type of disease and the image feature of the image in the state where the brain is all normal tissue, separately from the brain image including the abnormal region as a finding.

[0060] In addition, in this embodiment, the third feature derived from the target label image by learning the encoding learning model and the decoding learning model described later is an approximation of the first feature derived from the target image corresponding to the target label image. For this reason, the combination of the third feature and the second feature represents the image feature of the target image, particularly the image feature of the brain including the area determined according to the type of disease, like the combination of the first feature and the second feature. In this case, the combination of the third feature and the second feature is an image feature of the image in a state where all the brain tissues in the target image are normal, to which an image feature representing an abnormal area that should be included in the image as a finding is added. Therefore, by combining the third feature and the second feature, it is possible to synthesize and obtain the image feature of the image including the desired finding for the desired normal brain tissue.

[0061] The segmentation unit 23A of the image decoding unit 23 derives a finding label image V0 according to the type of abnormality of the finding in the target image, based on the first feature derived by the first feature derivation unit 22A that receives the target image as input.

[0062] The first reconstruction unit 23B of the image decoding unit 23 derives a first reconstructed image by reconstructing image features of an image in which the findings in the target image are assumed to be a normal area, based on the second feature derived by the second feature derivation unit 22B that uses the target image as input.

[0063] The second reconstruction unit 23C of the image decoding unit 23 derives a second reconstructed image by reconstructing the image features of the target image based on the first feature derived by the first feature derivation unit 22A and the second feature derived by the second feature derivation unit 22B, which use the target image as input. Note that the image features of the reconstructed target image are image features including the background other than the brain included in the target image.

[0064] For this purpose, the segmentation unit 23A, the first reconstruction unit 23B and the second reconstruction unit 23C have a decoder as a decoding learning model that is trained to derive a finding label image V0 according to the type of abnormality of the finding when the first feature and the second feature are input, and to derive a first reconstructed image and a second reconstructed image.

[0065] 6 and 7 are conceptual diagrams of the processes performed by the image encoding unit 22 and the image decoding unit 23. As shown in Fig. 6 and Fig. 7, the image encoding unit 22 has an encoder 31 and a latent model 31A, which are a first encoding learning model, and a label encoder 33 and a latent model 33A, which are a second encoding learning model.

[0066] Here, the encoder 31 and the latent model 31A, which are the first encoding learning model, receive image information, i.e., the input of a target image, and function as the first feature derivation unit 22A and the second feature derivation unit 22B according to this embodiment. On the other hand, the label encoder 33, which is the second encoding learning model, and Latent Model 33A accepts an input of a label image, that is, a target label image, and functions as a third feature derivation unit 22C according to this embodiment.

[0067] The image decoding unit 23 also has decoders 32A to 32C which are decoding learning models. The decoders 32A to 32C function as the segmentation unit 23A, the first reconstruction unit 23B, and the second reconstruction unit 23C, respectively. Note that the decoder 32A shown in Fig. 7 is the same as the decoder 32A shown in Fig. 6, but for the sake of explanation, the input and output directions are shown in Fig. 7 in the opposite direction to those in Fig. 6.

[0068] The encoder 31 and the latent model 31A as the first encoding learning model, and the decoders 32A to 32C as the decoding learning model are constructed by performing machine learning using a combination of a teacher image of a brain including findings as a subject and a teacher label image corresponding to an area in the teacher image determined according to the type of brain disease as teacher data. The encoder 31 and the decoders 32A to 32C are, for example, a convolutional neural network (CNN (Convolutional Neural Network)), which is one of multilayer neural networks in which multiple processing layers are hierarchically connected. The latent model 31A is trained using a method of VQ-VAE (Vector Quantised-Variational AutoEncoder).

[0069] VQ-VAE is a method proposed in "Neural Discrete Representation Learning, Aaron van den Oord et al., Advances in Neural Information Processing Systems 30 (NIPS), 6306-6315, 2017," which receives latent variables that represent the features of input data encoded by a feature extractor (i.e., an encoder), quantizes the received latent variables, passes the quantized latent variables to a feature decoder (i.e., a decoder), and learns the process of quantizing the latent variables based on whether the original input data was correctly reconstructed. The learning process will be described later.

[0070] In addition, the latent model 31A can be trained using any method, such as an AutoEncoder, a Variational AutoEncoder (VAE), Generative Adversarial Networks (GAN), and a Bidirectional GAN ​​(BiGAN), instead of VQ-VAE.

[0071] The convolutional neural network constituting the encoder 31 is composed of a plurality of processing layers. Each processing layer is a convolution processing layer, and performs convolution processing using various kernels while downsampling the image input from the previous processing layer. The kernel has a predetermined pixel size (e.g., 3×3), and a weight is set for each element. Specifically, a weight is set like a differential filter that emphasizes the edges of the image input from the previous processing layer. Each processing layer applies the kernel to the entire input image or the feature amount output from the previous processing layer while shifting the pixel of interest of the kernel, and outputs it as a feature map. In addition, the resolution of the feature map becomes smaller in the later processing layers of the encoder 31. As a result, the encoder 31 encodes the features of the input target image (assumed to be G0) by compressing (i.e., dimensional compression) so that the resolution of the feature map becomes smaller, and outputs two latent variables, that is, a first latent variable z1 and a second latent variable z2. The first latent variable z1 represents an image feature regarding the abnormality of the findings in the target image G0, and the second latent variable z2 represents an image feature regarding the image when the findings in the target image G0 are assumed to be normal regions.

[0072] The first and second latent variables z1 and z2 are each composed of n×n D-dimensional vectors. In FIG. 6, for example, n=4, and the first and second latent variables z1 and z2 can be expressed as an n×n map in which each position is composed of a D-dimensional vector. The first latent variable z1 and the second latent variable z2 may have different vector dimensions and vector numbers. Here, the first latent variable z1 corresponds to a feature vector representing image features regarding abnormality of findings. The second latent variable z2 corresponds to a feature vector representing image features of an image in the case where findings included in the target image G0 are normal regions.

[0073] Here, in this embodiment, in the latent model 31A, K D-dimensional first feature vectors e1k representing representative image features regarding the abnormality of findings are prepared in advance for the first latent variable z1. In addition, in the latent model 31A, K D-dimensional second feature vectors e2k representing representative image features regarding an image in the case where the findings are in a normal region are prepared in advance for the second latent variable z2. Note that the first feature vector e1k and the second feature vector e2k are stored in the storage 13. In addition, the number of the prepared first feature vectors e1k and the number of the prepared second feature vectors e2k may be different. The prepared second feature vector e2k is an example of a normal feature vector.

[0074] The image encoding unit 22 replaces each of the n×n D-dimensional vectors included in the first latent variable z1 with the first feature vector e1k in the latent model 31A. At this time, the n×n D-dimensional vectors included in the first latent variable z1 are replaced with the first feature vector e1k that has the smallest difference in the D-dimensional vector space. FIG. 8 is a diagram for explaining the replacement with the first feature vector. In FIG. 8, the vectors of the latent variables are shown in two dimensions for ease of explanation. In FIG. 8, it is assumed that four first feature vectors e11 to e14 are prepared. As shown in FIG. 8, the vector z1-1 of one latent variable included in the first latent variable z1 has the smallest difference with the first feature vector e12 in the vector space. Therefore, the vector z1-1 is replaced with the first feature vector e12. Also, for the second latent variable z2, similarly to the first latent variable z1, each of the n×n D-dimensional vectors is replaced with one of the second feature vectors e2k.

[0075] In this way, by replacing each of the n×n D-dimensional vectors included in the first latent variable z1 with the first feature vector e1k, the first latent variable z1 is represented by a combination of up to K latent variables having n×n predetermined values. Therefore, the first latent variable zd1 is quantized and distributed in the D-dimensional latent space.

[0076] Moreover, by replacing each of the n×n D-dimensional vectors included in the second latent variable z2 with the second feature vector e2k, the second latent variable z2 is represented by a combination of up to K latent variables having n×n predetermined values. Therefore, the second latent variable zd2 is quantized and distributed in the D-dimensional latent space.

[0077] The quantized first and second latent variables are designated by reference symbols zd1 and zd2. The quantized first and second latent variables zd1 and zd2 can also be represented as an n×n map in which each position is a D-dimensional vector. The quantized first and second latent variables zd1 and zd2 correspond to the first feature amount and the second feature amount, respectively.

[0078] The convolutional neural network that constitutes the decoders 32A to 32C is made up of a number of processing layers. Each processing layer is a convolution processing layer, and when the first and second latent variables zd1, zd2 are input as the first and second feature amounts, the processing layer performs convolution processing using various kernels while upsampling the feature amounts input from the previous processing layer. Each processing layer applies the kernel to the entire feature map made up of the feature amounts output from the previous processing layer while shifting the pixel of interest of the kernel. Also, the later the processing layers of the decoders 32A to 32C, the greater the resolution of the feature map. As will be described later, pictureWhen the image retrieval device retrieves a similar image, no processing is performed in the decoders 32A to 32C. However, since this is necessary for the learning processing described later, the processing performed in the decoders 32A to 32C will be described here using the first and second latent variables zd1 and zd2 derived from the target image G0 by the image encoding unit 22.

[0079] In this embodiment, a first latent variable zd1 is input to the decoder 32A. The decoder 32A derives a finding label image V0 according to the type of abnormality of the finding in the target image G0 input to the encoder 31, based on the first latent variable zd1.

[0080] The decoder 32B receives a second latent variable zd2. The decoder 32B derives a first reconstructed image V1 by reconstructing image features of an image in which the findings included in the target image G0 input to the encoder 31 are assumed to be normal regions, based on the second latent variable zd2. Therefore, even if the findings are included in the target image G0, the findings are not included in the first reconstructed image V1, and as a result, the brain included in the first reconstructed image V1 is composed of only normal tissue.

[0081] The second latent variable zd2 is input to the decoder 32C. In addition, a finding label image V0 having a size according to the resolution of each processing layer is input collaterally to each processing layer of the decoder 32C. Specifically, a feature map of the finding label image V0 having a size according to the resolution of each processing layer is input collaterally. The feature map input collaterally may be derived by reducing the feature map output from the processing layer immediately before deriving the finding label image V0 in the decoder 32A to a size according to the resolution of each processing layer of the decoder 32C. Alternatively, a feature map having a size according to the resolution of each processing layer derived by the decoder 32A in the process of deriving the finding label image V0 may be input to each processing layer of the decoder 32C. In the following description, it is assumed that the feature map output from the processing layer immediately before deriving the finding label image V0 is input collaterally to each processing layer of the decoder 32C by reducing the feature map to a size according to the resolution of each processing layer of the decoder 32C.

[0082] Here, the finding label image V0 and the feature map are derived based on the first latent variable zd1. Therefore, the decoder 32C derives the second reconstructed image V2 by reconstructing the image features of the input target image G0 based on the first and second latent variables zd1 and zd2. As a result, the second reconstructed image V2 is obtained by adding the image features of the abnormality of the region determined according to the type of disease based on the first latent variable zd1 to the image features of the brain consisting of only normal tissue included in the first reconstructed image V1 based on the second latent variable zd2. Therefore, the second reconstructed image V2 is obtained by reconstructing the image features of the input target image G0.

[0083] On the other hand, the label encoder 33 and the latent model 33A as the second encoding learning model are constructed by machine learning a neural network using a finding label image or the like including only the finding region as training data. The training data for training the label encoder 33 and the latent model 33A will be described later. The label encoder 33 is also composed of a convolutional neural network like the encoder 31. Also, like the latent model 31A, the latent model 33A is trained using the VQ-VAE method.

[0084] In this embodiment, when searching for a similar image, an additional finding label image F0 constituting a query image Q0 is input to the label encoder 33. The label encoder 33 encodes the features of the input additional finding label image F0 by compressing (i.e., dimensional compression) so that the resolution of the feature map is reduced, and outputs a third latent variable z3. The third latent variable z3 represents an image feature regarding the abnormality of the added finding.

[0085] The third latent variable z3 is composed of n×n D-dimensional vectors, similar to the first and second latent variables z1 and z2. In FIG. 7, for example, n=4, and the third latent variable z3 can be expressed as an n×n map in which each position is composed of a D-dimensional vector. Note that the number of vector dimensions and the number of vectors must match between the third latent variable z3 and the first latent variable z1. On the other hand, the third latent variable z3 may differ from the second latent variable z2 in the number of vector dimensions and the number of vectors. Here, the third latent variable z3 corresponds to an added finding feature vector representing the image features of the added finding.

[0086] In the present embodiment, K D-dimensional third feature vectors e3k representing representative image features regarding the abnormality of findings are prepared in advance for the third latent variable z3 in the latent model 33A. The third feature vector e3k is also stored in the storage 13. The prepared third feature vector e3k may be the same as or different from the first feature vector e1k. The number of prepared third feature vectors e3k may be different from the number of first and second feature vectors e1k and e2k. The prepared third feature vector e3k is an example of a finding feature vector.

[0087] The image encoding unit 22 replaces each of the n×n D-dimensional vectors included in the third latent variable z3 with the third feature vector e3k in the latent model 33A. At this time, the n×n D-dimensional vectors included in the third latent variable z3 are replaced with the third feature vector e3k that has the smallest difference in the D-dimensional vector space, similar to the first and second latent variables. In this way, by replacing each of the n×n D-dimensional vectors included in the third latent variable z3 with the third feature vector e3k, the third latent variable z3 is represented by a combination of up to K latent variables having n×n predetermined values. Therefore, the third latent variable zd3 is quantized and distributed in the D-dimensional latent space.

[0088] The quantized third latent variable zd3 is denoted by reference symbol zd3. The quantized third latent variable zd3 can also be expressed as an n×n map in which each position is composed of a D-dimensional vector. The quantized third latent variable zd3 derived for the additional finding label image F0 is an example of an additional finding feature.

[0089] The learning unit 24 performs learning of the encoder 31 and the latent model 31A of the image encoding unit 22, the decoders 32A to 32C of the image decoding unit 23, and the label encoder 33 and the latent model 33A of the image encoding unit 22. FIG. 9 is a diagram showing an example of teacher data used for learning the encoder 31 and the latent model 31A of the image encoding unit 22, and the decoders 32A to 32C of the image decoding unit 23. As shown in FIG. 9, the teacher data 35 includes a teacher image 36 of the brain including findings 37 such as a tumor, infarction, or hemorrhage, and a teacher label image 38 according to the type of abnormality of the findings in the teacher image 36. The teacher data used for learning the label encoder 33 and the latent model 33A will be described later.

[0090] The learning unit 24 inputs a teacher image 36 to the encoder 31, and causes the encoder 31 to output a first latent variable z1 and a second latent variable z2 for the teacher image 36. In the following description, the first latent variable and the second latent variable for the teacher image 36 will also be referred to as z1 and z2.

[0091] Next, the learning unit 24 obtains the quantized first latent variable zd1 and the second latent variable zd2 by replacing the vectors of latent variables included in the first latent variable z1 and the second latent variable z2 with the first and second feature vectors in the latent model 31A, respectively. In the following description, zd1 and zd2 are also used as reference symbols for the quantized first latent variable and the second latent variable for the teacher image 36. The quantized first latent variable zd1 and the second latent variable zd2 for the teacher image 36 correspond to the first learning feature and the second learning feature, respectively.

[0092] Then, the learning unit 24 inputs the first latent variable zd1 to the decoder 32A to derive a learning finding label image VT0 according to the type of abnormality of the finding 37 included in the teacher image 36. The learning unit 24 also inputs the second latent variable zd2 to the decoder 32B to derive a first learning reconstructed image VT1 in which image features of an image in which the finding 37 included in the teacher image 36 is assumed to be a normal region are reconstructed. Furthermore, the learning unit 24 inputs the second latent variable zd2 to the decoder 32C, and collaterally inputs a learning finding label image VT0 of a size according to the resolution of each processing layer, specifically, a feature map of the learning finding label image VT0, to each processing layer of the decoder 32C to derive a second learning reconstructed image VT2 in which image features of the teacher image 36 are reconstructed. In deriving the second training reconstructed image VT2, the feature map output from the processing layer immediately prior to deriving the training finding labeled image VT0 may be collaterally input to each processing layer of the decoder 32C by reducing the size of the feature map according to the resolution of each processing layer of the decoder 32C.

[0093] FIG. 10 is a diagram showing an example of teacher data used for training the label encoder 33 and the latent model 33A. As shown in FIG. 10, the teacher data 39 includes a teacher label image 38 included in the teacher data 35 and a training finding label image VT0 output from the decoder 32A. Here, when the training finding label image VT0 is used for training the label encoder 33 and the latent model 33A, it is referred to as a teacher label image. For this reason, the learning unit 24 stores a copy of the training finding label image VT0 in the memory 16 to be added to the teacher data 39 as a teacher finding label image. In FIG. 7, the broken line indicates that the training finding label image VT0 is copied to become a teacher finding label image. Note that the teacher label image is also referred to as a reference symbol VT0, which is the same as the training finding label image. Furthermore, in training the label encoder 33 and the latent model 33A, the first latent variable zd1 input to the decoder 32A when the teacher finding label image VT0 is derived is used as a first teacher feature. Therefore, the learning unit 24 copies the quantized first latent variable zd1 and stores it as the first teacher feature in the memory 16. In Fig. 7, the copying of the first latent variable zd1 to become the first teacher feature is indicated by a dashed line. Note that the first teacher feature will also be referred to as zd1.

[0094] The teacher data 39 used for training the label encoder 33 and the latent model 33A may include only one of the teacher label image 38 included in the teacher data 35 and the teacher finding label image VT0 derived from the teacher image 36. However, since the more images used for training the label encoder 33 and the latent model 33A, the more effectively the training can proceed, it is preferable that the teacher data 39 includes both the teacher label image 38 included in the teacher data 35 and the teacher finding label image VT0 derived from the teacher image 36.

[0095] The learning unit 24 inputs the teacher label image 38 to the label encoder 33, and causes it to output a third latent variable z3 for the teacher label image 38. Also, the teacher finding label image VT0 is input to the label encoder 33, and causes it to output a third latent variable z3 for the teacher finding label image VT0. In the following description, z3 will also be used as a reference symbol for the third latent variable for the teacher data 39. Also, since the learning process performed when the teacher finding label image VT0 is input to the label encoder 33 is the same as when the teacher label image 38 is input to the label encoder 33, the process performed when the teacher label image 38 is input to the label encoder 33 will be described here.

[0096] Next, the learning unit 24 replaces the vector of latent variables included in the third latent variable z3 with the third feature vector in the latent model 33A, thereby obtaining a quantized third latent variable zd3. In the following description, the third latent variable used for learning will also be referred to as zd3. The quantized third latent variable zd3 derived for the teacher data 39 is set as a third learning feature.

[0097] The learning unit 24 derives the difference between the first latent variable zd1, which is the first learning feature, and a predetermined probability distribution of the first feature as the first loss L1. Here, the predetermined probability distribution of the first feature is a probability distribution that the first latent variable zd1 should follow. When the VQ-VAE method is used, a codeword loss and a commitment loss are derived as the first loss L1. The codeword loss is a value that a codeword, which is a representative local feature in the probability distribution of the first feature, should take. The commitment loss is the distance between the first latent variable zd1 and the codeword closest to the first latent variable zd1. The encoder 31 and the latent model 31A are trained so that the first latent variable zd1 according to the predetermined probability distribution of the first feature is acquired by the first loss L1.

[0098] Moreover, the learning unit 24 derives the difference between the second latent variable zd2, which is the second learning feature, and the probability distribution of the predetermined second feature as the second loss L2. Here, the predetermined probability distribution of the second feature is the probability distribution that the second latent variable zd2 should follow. When the VQ-VAE method is used, the codeword loss and the commitment loss are derived as the second loss L2, similar to the first loss L1. The codeword loss regarding the second latent variable zd2 is the value that the codeword, which is a representative local feature in the probability distribution of the second feature, should take. The commitment loss regarding the second latent variable zd2 is the distance between the second latent variable zd2 and the codeword closest to the second latent variable zd2. The encoder 31 and the latent model 31A are trained so that the second latent variable zd2 according to the predetermined probability distribution of the second feature is acquired by the second loss L2.

[0099] In addition, the learning unit 24 derives the difference as a semantic segmentation between a teacher label image 38 corresponding to the type of abnormality of a finding 37 contained in the teacher image 36 and the learning finding label image VT0 as a third loss L3.

[0100] The "difference as semantic segmentation" is an index determined based on the overlap between an area corresponding to the type of abnormality represented by the teacher label image 38 and an area corresponding to the type of abnormality represented by the learning finding label image VT0. Specifically, the value of the number of elements common to the teacher label image 38 and the learning finding label image VT0 × 2 relative to the sum of the number of elements in the teacher label image 38 and the number of elements in the learning finding label image VT0 can be used as the difference as semantic segmentation, i.e., the third loss L3.

[0101] Furthermore, the learning unit 24 derives the difference between the area outside the findings 37 included in the teacher image 36 and the first reconstructed learning image VT1 as the fourth loss L4. Specifically, the learning unit 24 derives the difference between the area obtained by removing the findings 37 from the teacher image 36 and the first reconstructed learning image VT1 as the fourth loss L4.

[0102] Furthermore, the learning unit 24 derives the difference between the teacher image 36 and the second reconstructed learning image VT2 as a fifth loss L5.

[0103] Furthermore, the learning unit 24 derives the second latent variable zd2, which is the second learning feature, so that the second latent variable zd2 includes image features in the case where the finding 37 in the teacher image 36 is a normal region, and does not include image features regarding the finding 37 in the teacher image 36. Specifically, by using a Wasserstein GAN (Proceedings of the 34th International Conference on Machine Learning, PMLR 70:214-223, 2017.) method, the difference between the distribution of the second latent variable zd2 derived from the teacher image 36 not including the finding 37 as an ideal distribution and the distribution of the second latent variable zd2 derived from the teacher image 36 including the finding 37 is derived as the sixth loss L6.

[0104] According to the Wasserstein GAN method, a neural network called a critic network is constructed as a convolutional neural network similar to the encoder 31 and the label encoder 33. The critic network receives as input a plurality of second latent variables zd2 derived from a plurality of teacher images 36 not including findings 37 and a plurality of second latent variables zd2 derived from a plurality of teacher images 36 including findings 37. The critic network then calculates the difference between the distribution of the second latent variables zd2 derived from the teacher images 36 not including findings 37 and the distribution of the second latent variables zd2 derived from the teacher images 36 including findings 37 as the Wasserstein distance. At this time, a constraint term may be added so that the critic network has a unit gradient norm at all points included in the distribution of the second latent variables zd2 derived from the teacher images 36 not including findings 37 and the distribution of the second latent variables zd2 derived from the teacher images 36 including findings 37.

[0105] Furthermore, the learning unit 24 derives the difference between a third latent variable zd3, which is a third learning feature, and a probability distribution of a predetermined third feature as a seventh loss L7.

[0106] Here, the predetermined probability distribution of the third feature is a probability distribution that the third latent variable zd3 should follow. When the VQ-VAE method is used, the codeword loss and the commitment loss are derived as the seventh loss L7, similar to the first loss L1. The codeword loss for the third latent variable zd3 is a value that a codeword, which is a representative local feature in the probability distribution of the third feature, should take. The commitment loss for the third latent variable zd3 is the distance between the third latent variable zd3 and the codeword closest to the third latent variable zd3. The label encoder 33 and the latent model 33A are trained so that the third latent variable zd3 according to the predetermined probability distribution of the third feature is obtained by the seventh loss L7.

[0107] In addition, the learning unit 24 derives the difference between the third latent variable zd3 and the first teacher feature zd1 as the eighth loss L8.

[0108] Here, the closer the first latent variable zd1 acquired by the encoder 31 and the latent model 31A follows the probability distribution of the predetermined first feature amount, the more the encoder 31 can output a preferable first latent variable z1 that can faithfully reproduce the abnormality of the finding 37 included in the teacher image 36. Also, the more preferable quantized first latent variable zd1 can be acquired by the latent model 31A.

[0109] Moreover, the more closely the second latent variable zd2 acquired by the encoder 31 and the latent model 31A follows the probability distribution of the predetermined second feature amount, the more the encoder 31 can output a preferable second latent variable z2 that can faithfully reproduce an image in which the findings 37 included in the teacher image 36 are assumed to be normal. Moreover, the more preferable quantized second latent variable zd2 can be acquired by the latent model 31A.

[0110] Moreover, since the learning finding label image VT0 output from the decoder 32A is derived based on the first latent variable zd1, it does not completely match the teacher label image 38. Moreover, the learning finding label image VT0 does not completely match the finding 37 included in the teacher image 36. However, the smaller the difference between the learning finding label image VT0 and the teacher label image 38 as semantic segmentation with respect to the teacher image 36, the more preferable the first latent variable z1 can be output from the encoder 31 when the target image G0 is input. That is, it is possible to output the first latent variable z1 that potentially includes information indicating where in the target image G0 the finding is and image features regarding the abnormality of the finding. Moreover, it is possible to obtain a more preferable quantized first latent variable zd1 by the latent model 31A. Therefore, the encoder 31 extracts the findings from the target image G0 while deriving the first latent variable zd1 that represents image features regarding the abnormality of the findings. Moreover, the decoder 32A can output a finding label image V0 corresponding to the type of abnormality of a finding for a region corresponding to the finding included in the target image.

[0111] In addition, since the first learning reconstructed image VT1 output from the decoder 32B is derived based on the second latent variable zd2, it does not completely match the image features of the image when the findings 37 included in the teacher image 36 are assumed to be normal. However, the smaller the difference between the first learning reconstructed image VT1 and the area in the teacher image 36 that is not the findings 37, the more preferable the second latent variable z2 can be output from the encoder 31 when the target image G0 is input. In addition, the more preferable quantized second latent variable zd2 can be obtained by the latent model 31A. In addition, the decoder 32B can output the first reconstructed image V1 that is closer to the image when the findings included in the target image G0 are assumed to be normal.

[0112] In addition, the second learning reconstructed image VT2 output from the decoder 32C is derived based on the first latent variable zd1 and the second latent variable zd2, and therefore does not completely match the teacher image 36. However, the smaller the difference between the second learning reconstructed image VT2 and the teacher image 36, the more preferable the first latent variable z1 and the second latent variable z2 can be output from the encoder 31 when the target image G0 is input. In addition, the more preferable quantized first latent variable zd1 and quantized second latent variable zd2 can be obtained by the latent model 31A. In addition, the decoder 32C can output the second reconstructed image V2 that is closer to the target image G0.

[0113] In addition, for the second latent variable zd2 acquired by the encoder 31 and the latent model 31A, the distribution (absence of findings) when derived from a teacher image 36 not including findings 37 and the distribution (presence of findings) when derived from a teacher image 36 including findings 37 are influenced by the difference in image features according to the presence or absence of findings 37. For this reason, the absence of findings distribution and the presence of findings distribution do not completely match. However, if the absence of findings distribution is considered to be a distribution to be, the smaller the difference between the absence of findings distribution and the presence of findings distribution, the less the second latent variable zd2 derived from the teacher image 36 including findings 37 includes image features of the findings 37 included in the teacher image 36, and includes only image features of the image when the findings 37 included in the teacher image 36 are in a normal region. This makes it possible to acquire a preferable second latent variable zd2. Therefore, it is possible to output a preferable second latent variable z2 that can faithfully reproduce an image when the findings 37 included in the teacher image 36 are in a normal region from the encoder 31.

[0114] Moreover, the closer the third latent variable zd3 acquired by the label encoder 33 and the latent model 33A follows the probability distribution of the predetermined third feature amount, the more the label encoder 33 and the latent model 33A can output a preferable third latent variable z3 that can faithfully reproduce the abnormality of the findings included in the teacher label image 38 or the teacher finding label image. Moreover, the more preferable quantized third latent variable zd3 can be acquired by the latent model 33A.

[0115] Furthermore, the smaller the difference between the third latent variable zd3 acquired by the label encoder 33 and the latent model 33A and the first teacher feature zd1, the more the third latent variable z3 derived from the label encoder 33 and the latent model 33A and representing the abnormality of the finding included in the teacher label image 38 or the teacher finding label image VT0 approximates the first latent variable zd1 derived from the encoder 31 and the latent model 31A and representing the image feature regarding the abnormality of the finding included in the target image G0. Therefore, it is possible to estimate and acquire the first latent variable zd1 representing the image feature of the finding 37 to be included in the teacher image 36 according to the abnormality of the finding included in the teacher label image 38 or the teacher finding label image VT0 from the label encoder 33 and the latent model 33A.

[0116] Therefore, the learning unit 24 learns the encoder 31, the latent model 31A, the decoders 32A-32C, the label encoder 33, and the latent model 33A based on at least one of the first to eighth losses L1-L8 derived as described above. In this embodiment, the learning unit 24 learns the encoder 31, the latent model 31A, the decoders 32A-32C, the label encoder 33, and the latent model 33A so that all of the losses L1-L8 satisfy a predetermined condition. That is, the number of processing layers, the number of pooling layers, the kernel coefficients in the processing layers, the kernel size, the weights of the connections between the layers, the first feature vector e1k, the second feature vector e2k, and the like constituting the encoder 31, the latent model 31A, and the decoders 32A-32C are updated so that the losses of the first to sixth losses L1-L6 are reduced, thereby learning the encoder 31, the latent model 31A, and the decoders 32A-32C. In addition, the learning unit 24 learns the label encoder 33 and the latent model 33A by updating the number of processing layers, the number of pooling layers, the kernel coefficients in the processing layers, the kernel sizes, the connection weights between each layer, the third feature vector e3k, etc. that constitute the label encoder 33 and the latent model 33A so that the seventh and eighth losses L7, L8 are small.

[0117] In this embodiment, the learning unit 24 learns the encoder 31, the latent model 31A, the decoders 32A-32C, the label encoder 33, and the latent model 33A so that the first loss L1 is equal to or less than a predetermined threshold Th1, the second loss L2 is equal to or less than a predetermined threshold Th2, the third loss L3 is equal to or less than a predetermined threshold Th3, the fourth loss L4 is equal to or less than a predetermined threshold Th4, the fifth loss L5 is equal to or less than a predetermined threshold Th5, the sixth loss L6 is equal to or less than a predetermined threshold Th6, the seventh loss L7 is equal to or less than a predetermined threshold Th7, and the eighth loss L8 is equal to or less than a predetermined threshold Th8. Instead of learning using a threshold, learning may be performed a predetermined number of times, or learning may be performed so that each of the losses L1-L8 is minimized or maximized.

[0118] In this way, the learning unit 24 learns the encoder 31, the latent model 31A, the decoders 32A to 32C, the label encoder 33, and the latent model 33A, so that the encoder 31 outputs a first latent variable z1 that more appropriately represents the image features of the abnormality of the findings of the brain included in the input target image G0. The encoder 31 also outputs a second latent variable z2 that more appropriately represents the image features of the brain included in the input target image G0 when the findings are in a normal region. The latent model 31A also acquires a quantized first latent variable zd1 that more appropriately represents the image features of the brain included in the input target image G0 when the findings are in a normal region. The latent model 31A also acquires a quantized second latent variable zd2 that more appropriately represents the image features of the brain included in the input target image G0 when the findings are in a normal region.

[0119] When the quantized first latent variable zd1 is input, the decoder 32A outputs a finding label image V0 that more accurately represents semantic segmentation according to the type of abnormality of the finding included in the target image G0. When the quantized second latent variable zd2 is input, the decoder 32B outputs a first reconstructed image V1 that reconstructs image features of the brain in the target image G0 when the finding is a hypothetical normal region. When the quantized second latent variable zd2 is input and the finding label image V0 is collaterally input to each processing layer, the decoder 32C adds image features of the abnormality of the region determined according to the type of disease based on the first latent variable zd1 to image features of the brain consisting of only normal tissue included in the first reconstructed image V1 based on the second latent variable zd2, and as a result, outputs a second reconstructed image V2 that reconstructs image features of the brain including the finding.

[0120] Furthermore, when the additional finding label image F0 is input, the label encoder 33 outputs a third latent variable z3 that more appropriately represents the image features of the abnormality of the brain findings included in the additional finding label image F0. Furthermore, the latent model 33A outputs a quantized third latent variable zd3 that more appropriately represents the image features representing the abnormality of the brain findings included in the additional finding label image F0. Furthermore, the third latent variable zd3 derived in this process becomes an estimated value of the first latent variable zd1 that represents the image features of the abnormality of the brain findings included in the image corresponding to the additional finding label image F0.

[0121] When the query image Q0 is derived and the search execution button 44 on the search screen 40 is selected by the user, the similarity derivation unit 25 derives the similarity between the query image Q0 and all reference images registered in the image database DB in order to search for a similar reference image similar to the query image Q0 (i.e., the target image G0) derived by the query image derivation unit 21 among the reference images registered in the image database DB stored in the image storage server 3. Here, a plurality of reference images for various brain cases are registered in the image database DB. In this embodiment, for the reference image, the image encoding unit 22 including the trained encoder 31 derives first and second quantized latent variables in advance, and the quantized first and second latent variables are associated with the reference image and registered in the image database DB. The first and second latent variables associated with the reference image and registered in the image database DB are referred to as first and second reference latent variables. The first reference latent variable is an example of a reference finding feature, and the second reference latent variable is an example of a reference normal feature.

[0122] The following describes the derivation of the similarity in the similarity derivation unit 25. In this embodiment, the query image Q0 includes a comment added by the user to the query base image B0. The similarity derivation unit 25 derives the similarity between the query image Q0 and the reference image.

[0123] FIG. 11 is a diagram for explaining derivation of a feature from a query image Q0 when searching for a similar image. As shown in FIG. 11, in this embodiment, a quantized third latent variable zd3 representing an image feature regarding the abnormality of a finding included in an additional finding label image F0 constituting the query image Q0 is derived as an additional finding feature by the label encoder 33 and latent model 33A of the image encoding unit 22. In addition, a quantized second latent variable zd2 representing an image feature regarding a query base image B0 constituting the query image Q0 is derived as a query normal feature by the encoder 31 and latent model 31A of the image encoding unit 22. The similarity derivation unit 25 derives a similarity based on a difference between the third latent variable zd3 derived for the query image Q0 and a first reference latent variable corresponding to the reference image, and a difference between the second latent variable zd2 derived for the query image Q0 and a second reference latent variable corresponding to the reference image.

[0124] Specifically, as shown in the following formula (1), the similarity derivation unit 25 calculates the Euclidean distance √{(Vq3(i,j)-Vr1(i,j)} 2 Derive the sum of the derived Euclidean distances Σ[√{(Vq3(i,j)-Vr1(i,j)} 2 In addition, the similarity derivation unit 25 derives the Euclidean distance √{(Vq2(i,j)-Vr2(i,j)} 2 Derive the sum of the derived Euclidean distances Σ[√{(Vq2(i,j)-Vr2(i,j)} 2 Then, the similarity derivation unit 25 derives the sum of the two summations as the similarity.

[0125] In equation (1), S0 represents similarity, Vq3(i,j) represents the vector at map position (i,j) in the third latent variable zd3, Vr1(i,j) represents the vector at map position (i,j) in the first reference latent variable, Vq2(i,j) represents the vector at map position (i,j) in the second latent variable zd2, and Vr2(i,j) represents the vector at map position (i,j) in the second reference latent variable. S0 = Σ[√{(Vq3(i,j)-Vr1(i,j)} 2 ]+Σ[√{(Vq2(i,j)-Vr2(i,j)} 2 ] (1)

[0126] Instead of the above formula (1), the similarity S0 may be derived by the following formula (1a): Here, concat(a, b) is an operation for concatenating vector a and vector b. S0 = Σ[√{(Vq32(i,j)-Vr12(i,j)} 2 ] (1a) however, Vq32(i,j) = concat(Vq3(i,j),Vq2(i,j)) Vr12(i,j) = concat(Vr1(i,j),Vr2(i,j))

[0127] The method of deriving the similarity S0 is not limited to the above method. Instead of the Euclidean distance, the Manhattan distance, the vector inner product, the cosine similarity, or the like may be used.

[0128] The extraction unit 26 extracts images similar to the query image Q0 from the image database DB based on the similarity S0 according to the input search criteria. Similar referenceBased on the similarity S0 between the query image Q0 and all reference images registered in the image database DB, the extraction unit 26 extracts reference images similar to the query image Q0 as similar reference images. Specifically, the extraction unit 26 sorts the reference images in descending order of similarity S0 to create a search result list. FIG. 12 is a diagram showing the search result list. As shown in FIG. 12, in the search result list 50, the reference images registered in the image database DB are sorted in descending order of similarity S0. Then, the extraction unit 26 extracts a predetermined number of reference images that are ranked at the top of the sorting order in the search result list 50 from the image database DB as similar reference images.

[0129] The display control unit 27 displays the extraction result by the extraction unit 26 on the display 14. Fig. 13 is a diagram showing a search screen including a search result that is the extraction result by the extraction unit 26. As shown in Fig. 13, four similar reference images R11 to R14 that are similar to the query image Q0 are displayed in a third display area 43 of the search screen 40.

[0130] Next, the process performed in this embodiment will be described. FIG. 14 is a flowchart showing the learning process performed in this embodiment. It is assumed that a plurality of teacher data are acquired from the image storage server 3 and stored in the storage 13. First, the learning unit 24 acquires one teacher data 35 including a teacher image 36 and a teacher label image 38 from the storage 13 (step ST1), and inputs the teacher image 36 included in the teacher data 35 to the encoder 31 of the image encoding unit 22. The encoder 31 derives a first latent variable z1 and a second latent variable z2 as a first learning feature and a second learning feature, respectively. In addition, the learning unit 24 uses the teacher label image 38 as teacher data 39, and inputs the teacher label image 38 to the label encoder 33 of the image encoding unit 22. The label encoder 33 derives a third latent variable z3 as a third learning feature (learning feature derivation; step ST2).

[0131] When the teacher observation label image VT0 is used as the teacher data 39, the derivation and quantization of the third learning feature from the teacher observation label image VT0 is performed after step ST4 described later.

[0132] Next, the learning unit 24 derives a quantized first latent variable zd1, a quantized second latent variable zd2, and a quantized third latent variable zd3 from the first latent variable z1, the second latent variable z2, and the third latent variable z3. Furthermore, the learning unit 24 stores a copy of the quantized first latent variable zd1 in the memory 16 as a first teacher feature zd1 (quantization; step ST3).

[0133] The learning unit 24 then inputs the quantized first latent variable zd1 to the decoder 32A of the image decoding unit 23. As a result, the decoder 32A derives a learning finding label image VT0 according to the type of abnormality of the finding 37 included in the teacher image 36. When the teacher finding label image VT0 is used as the teacher data 39, the learning unit 24 copies the learning finding label image VT0 and stores it in the memory 16 as the teacher finding label image VT0. The learning unit 24 also inputs the quantized second latent variable zd2 to the decoder 32B of the image decoding unit 23. As a result, the decoder 32B derives a first learning reconstructed image VT1 obtained by reconstructing an image in the case where the findings included in the teacher image 36 are normal regions. The learning unit 24 also inputs the second latent variable zd2 to the decoder 32C, and further inputs a learning finding label image VT0 having a size according to the resolution of each processing layer of the decoder 32C in a collateral manner to each processing layer of the decoder 32C. As a result, the decoder 32C derives a second learning reconstructed image VT2 by reconstructing the image features of the teacher image 36. ( Deriving learning images; step ST4).

[0134] Next, the learning unit 24 derives the first to eighth losses L1 to L8 as described above (step ST5).

[0135] Then, the learning unit 24 judges whether the first to eighth losses L1 to L8 satisfy a predetermined condition (condition judgment; step ST6). If step ST6 is negative, the learning unit 24 acquires new teacher data from the storage 13 (step ST7), returns to the process of step ST2, and repeats the processes of steps ST2 to ST6 using the new teacher data. If step ST6 is positive, the learning unit 24 ends the learning process. As a result, the encoder 31, latent model 31A, label encoder 33, and latent model 33A of the image encoding unit 22, and the decoders 32A to 32C of the image decoding unit 23 are constructed.

[0136] Next, the image search process performed in this embodiment will be described. Fig. 15 is a flowchart of the image search process performed in this embodiment. First, based on an instruction from a user, the information acquisition unit 20 acquires a query base image B0 for deriving a query image Q0 to be searched (step ST11), and the display control unit 27 displays a search screen 40 including the query base image B0 on ​​the display 14 (step ST12).

[0137] Next, the query image derivation unit 21 receives a drawing of the findings on the query base image B0, thereby deriving a query image Q0 to which the findings have been added (step ST13).

[0138] Then, when the search execution button 44 is selected to instruct search execution (step ST14; YES), the image encoding unit 22 derives the quantized third latent variable zd3 for the additional finding label image F0 constituting the query image Q0 and the quantized second latent variable zd2 for the query base image B0 constituting the query image Q0 as the additional finding feature and the query normal feature, respectively (feature derivation; step ST15). Then, the similarity derivation unit 25 derives the similarity between the query image Q0 and the reference image registered in the image database DB of the image storage server 3 based on the additional finding feature and the query normal feature (step ST16). Next, the extraction unit 26 extracts a predetermined number of reference images with the highest similarity according to the search conditions as similar reference images (step ST17). Furthermore, the display control unit 27 displays the similar reference images in the third display area 43 of the search screen 40 (search result display; step ST18), and ends the process.

[0139] In this manner, in the present embodiment, a query image Q0 to which a finding is added is derived by receiving an input of finding information representing at least one finding desired to be searched for the query base image B0. Also, at least one additional finding feature representing an image feature of the added finding and at least one query normal feature representing an image feature of a normal region included in the brain in the query base image B0 are derived, and similarities between the query image Q0 and each of a plurality of reference images stored in the image storage server 3 are derived based on the additional finding feature and the query normal feature, and reference images similar to the query image Q0 are extracted from the image storage server 3 as similar images based on the similarities.

[0140] Therefore, even if a user does not have a query image Q0 including a desired finding at hand, the user can derive a query image Q0 including the desired finding and search for images similar to the query image Q0. Therefore, the user does not need to search for the query image Q0 from images stored in a database, and as a result, the user can easily search for a desired image.

[0141] In the above embodiment, a standard image in which the region desired to be searched consists only of normal regions is used as the query base image B0, but the present invention is not limited to this. An image in which the region desired to be searched includes findings may be used as the query base image B0. FIG. 16 is a diagram showing an example of a query base image B0 including findings. As shown in FIG. 16, the query base image B0 includes findings 60 such as cerebral infarction in the left brain, for example.

[0142] Fig. 17 is a diagram showing a query image derived by adding findings to the query base image B0 including the findings shown in Fig. 16. As shown in Fig. 17, the query image Q1 includes the finding 60 included in the query base image B0 and the added finding 45. The query image Q1 is composed of the query base image B0 including the finding 60 and the added finding label image F0 including only the finding 45. At this time, the added finding label image F0 may be created by replacing information representing the finding 60 originally included in the query base image B0 with the finding label information derived from the decoder 32A and modifying this to more desirable label information according to the type of the desired finding by a user operation.

[0143] 17, when a similar image search is performed using a query image Q1 as shown in FIG. 17, a third latent variable zd3 representing an image feature regarding the abnormality of a finding included in an additional finding label image F0 constituting the query image Q1 is derived by the label encoder 33 and latent model 33A of the image encoding unit 22. In addition, a second latent variable zd2 representing an image feature regarding a query base image B0 including a finding 60 constituting the query image Q1 is derived by the encoder 31 and latent model 31A of the image encoding unit 22. Here, the second latent variable zd2 more appropriately represents the image feature of the query base image B0 when the finding 60 included in the query base image B0 is assumed to be a normal region.

[0144] Therefore, in this embodiment, even if the query base image B0 includes a finding, a query image Q1 including a different finding at a position different from the included finding can be derived to perform a similar image search.

[0145] In the above embodiment, the finding is added to only one location in the query base image B0, but the present invention is not limited to this. Findings may be added to multiple locations in the query base image B0. In this case, the added finding label image F0 includes the areas of the findings added to multiple locations.

[0146] In the above embodiment, the encoder 31 and the latent model 31A of the image encoding unit 22, the decoders 32A to 32C of the image decoding unit 23, and the label encoder 33 and the latent model 33A of the image encoding unit 22 are trained simultaneously, but the present invention is not limited to this. First, the encoder 31 and the latent model 31A of the image encoding unit 22, and the decoders 32A to 32C of the image decoding unit 23 may be trained, and then the label encoder 33 and the latent model 33A of the image encoding unit 22 may be trained. In this case, when the label encoder 33 and the latent model 33A are trained, the decoder 32A is not trained.

[0147] In the above embodiment, the image encoding unit 22 inputs the third latent variable z3 derived by the label encoder 33 into the latent model 33A to obtain the quantized third latent variable zd3 as an approximation of the quantized first latent variable zd1, but this is not limited to the above. The quantized first latent variable zd1 may be directly obtained by inputting the third latent variable z3 derived by the label encoder 33 into the latent model 31A without preparing the latent model 33A. In this case, instead of deriving the difference between the third latent variable zd3 and the first teacher feature zd1 as the eighth loss L8, the learning unit 24 may derive the difference between the third latent variable z3 before quantization and the first teacher feature zd1 as the eighth loss L8. In this case, the learning unit 24 may learn the encoding learning model and the decoding learning model based only on the first to sixth losses L1 to L6 and the eighth loss L8, without deriving the difference between the third latent variable zd3, which is the third learning feature, and a probability distribution of a predetermined third feature as the seventh loss L7.

[0148] In the above embodiment, the first to third latent variables zd1 to zd3 are derived by constructing the encoding learning model through learning, but the present invention is not limited to this. A previously created trained model may be used, or the first to third latent variables zd1 to zd3 may be derived using a feature calculator designed on a rule base. In particular, for the first latent variable and the third latent variable, the first feature derivation unit 22A and the label encoder 33 of the encoder 31 in the encoding learning model of this embodiment can be constructed relatively easily by using a data set used for learning a model that segments a region of a finding from a medical image including the finding.

[0149] In the above embodiment, an MRI image is used as the query base image B0, but the present invention is not limited to this. In addition to a CT image or a PET image, a radiation image obtained by photographing a subject with radiation, an ultrasound image, or the like can be used as the query base image B0.

[0150] In the above embodiment, the image of the brain is used as the target image, but the target image is not limited to the brain. Image Alternatively, the target image may be an image including any part of the human body, such as the lungs, heart, liver, kidneys, and limbs. In this case, the encoder 31, latent model 31A, decoders 32A to 32C, label encoder 33, and latent model 33A may be trained using teacher images and teacher label images including, as findings, diseases such as tumors, infarctions, cancers, and fractures that appear in the parts. In the above embodiment, an MRI image is used as the query base image B0, but the present invention is not limited to this. In addition to a CT image or a PET image, a radiation image obtained by photographing a subject with radiation, an ultrasound image, or the like can be used as the query base image B0.

[0151] In addition, a separate encoding learning model may be used for each of the first feature derivation unit 22A, the second feature derivation unit 22B and the third feature derivation unit 22C, and the first feature, the second feature and the third feature may be derived using the separate encoding learning model.

[0152] In the above embodiment, for example, the information acquisition unit 20, the query image derivation unit 21, the first feature derivation unit 22A, the second feature derivation unit 22B, the third feature derivation unit 22C, the segmentation unit 23A, the first reconstruction unit 23B, the second reconstruction unit 23C, the learning unit 24, the similarity derivation unit 25, the extraction unit 26, and the display control unit 27 may have a hardware structure of a processing unit that executes various processes. The above-mentioned various processors include, as described above, a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processes such as an ASIC (Application Specific Integrated Circuit), etc.

[0153] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0154] As an example of configuring multiple processing units with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip, as typified by System On Chip (SoC), etc. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0155] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. [Explanation of symbols]

[0156] 1. Computer 2. Imaging Device 3. Image storage server 4. Network 10 Image search device 11 CPU 12. Image Search Programs 13. Storage 14 Display 15 Input Devices 16 Memory 17 Network Interface 18 Bus 20 Information acquisition Department 21 Query image derivation part 22 Image Encoding Unit 22A First feature derivation unit 22B Second feature derivation unit 22C Third feature extraction unit 23 Image Decoding Unit 23A Segmentation section 23B First Reconstruction Section 23C Second Reconstruction Section 24 Learning Department 25 Similarity derivation part 26 Extraction part 27 Display control section 31 Encoder 31A Potential Model 32A~32C Decoder 33 Label Encoder 33A Potential Model 35 Teacher Data 36 Teacher Images 37 Findings 38 Teacher Label Images 39 Teacher Data 40 Search screen 41 First display area 42 Second display area 42A Edema 42B Contrast effect 42C Necrosis 43 Third display area 44 Search button 45 Findings 50 Search Results List 60 findings B0 Query-based image F0 Additional findings label image G0 Target image R11~R14 Similar reference images V0 Findings Label Image V1 First reconstructed image V2 Second reconstructed image VT0 Additional findings label image for learning (findings label image for teacher) VT1 First reconstructed image for training VT2 Second reconstructed image for training z1 First latent variable z2 Second latent variable z3 Third latent variable zd1 Quantized first latent variable (first teacher feature) zd2 quantized second latent variable zd3 quantized third latent variable

Claims

1. At least one processor; The processor, receiving input of at least one finding that is desired to be searched for a query base image that is a source of the query image and includes a region that is desired to be searched, thereby deriving a query image to which the finding is added; deriving at least one additional finding feature representing an image feature for the additional finding; deriving at least one query normal feature representing an image feature for a normal region included in the site in the query base image; referring to an image database in which a plurality of reference images including findings are registered, the plurality of reference images being associated with at least one reference finding feature amount representing an image feature for the findings included in each of the reference images and at least one reference normal feature amount representing an image feature for an image in the case where the findings included in each of the reference images are assumed to be normal regions, deriving a similarity between the query image and each of the plurality of reference images based on a comparison of the additional finding feature amount and the query normal feature amount with the reference finding feature amount and the reference normal feature amount; The image search device extracts the reference images similar to the query image from an image database as similar images based on the similarity.

2. The image search device according to claim 1 , wherein the finding information includes an area corresponding to a type of the finding.

3. The processor derives the additional finding feature by encoding the additional finding; The image search device according to claim 1 , wherein the query normal feature is derived by encoding the query base image.

4. The image search device according to claim 1 , wherein the query base image is a standard image in which the region desired to be searched is made up of only normal regions.

5. a storage for storing at least one finding feature vector representing representative image features for the finding and a normal feature vector representing representative image features for the normal region; the processor derives an additional finding feature vector representing an image feature for the added finding, quantizes the additional finding feature vector by replacing it with a finding feature vector among the finding feature vectors that has a minimum difference from the additional finding feature vector, and derives the additional finding feature amount; 5. The image search device according to claim 4, further comprising: a query normal feature vector representing image features for a normal region included in the part in the query base image; and a quantization step for replacing the query normal feature vector with a normal feature vector among the normal feature vectors that has a minimum difference from the query normal feature vector, thereby deriving the query normal feature.

6. the query base image is an image including a finding; The image search device according to claim 1 , wherein the query normal feature amount represents an image feature of an image in a case where the findings included in the query base image are considered to be normal regions.

7. A storage device for storing at least one finding feature vector representing representative image features for the finding, and a normal feature vector representing representative image features for an image in the case where the included finding is a normal region, the processor derives an additional finding feature vector representing an image feature for the added finding, quantizes the additional finding feature vector by replacing it with a finding feature vector among the finding feature vectors that has a minimum difference from the additional finding feature vector, and derives the additional finding feature amount; 7. The image search device according to claim 6, further comprising: a query normal feature vector representing image features of the query base image in a case where a region including the finding in the query base image is assumed to be a normal region; and a quantization step for replacing the query normal feature vector with a normal feature vector among the normal feature vectors that has a minimum difference from the query normal feature vector, thereby deriving the query normal feature vector.

8. The image search device according to claim 1 , wherein the processor derives the additional finding feature by using an encoding learning model that has been trained to derive the additional finding feature when the added finding is input.

9. The image search device according to claim 1 , wherein the processor derives the query normal feature using an encoding learning model that has been trained to derive the query normal feature when the query base image is input.

10. The image search device according to claim 1 , wherein the reference finding feature amount and the reference normal feature amount are derived by encoding the reference image.

11. The image search apparatus according to claim 10 , wherein a combination of the reference finding feature amount and the reference normal feature amount represents an image feature for the reference image.

12. the reference finding feature amount is derived by quantizing a feature vector representing an image feature regarding the abnormality of a finding included in the reference image by replacing the feature vector with a first feature vector having a minimum difference from the image feature regarding the abnormality of the finding, among at least one first feature vector representing a representative image feature regarding the abnormality of the finding; 12. The image search device according to claim 10, wherein the reference normal feature is derived by quantizing a feature vector representing an image feature for the reference image when the finding is considered to be a normal area, by a second feature vector having a minimum difference from the image feature for the reference image when the finding is considered to be a normal area, among at least one second feature vector representing representative image features for the image when the finding is considered to be a normal area.

13. 13. The image search device according to claim 10, wherein the reference finding feature and the reference normal feature are derived using an encoding learning model that is trained to derive the reference finding feature and the reference normal feature when the reference image is input.

14. receiving input of at least one finding that is desired to be searched for a query base image that is a source of the query image and includes a region that is desired to be searched, thereby deriving a query image to which the finding is added; deriving at least one additional finding feature representing an image feature for the additional finding; deriving at least one query normal feature representing an image feature for a normal region included in the site in the query base image; referring to an image database in which a plurality of reference images including findings are registered, the plurality of reference images being associated with at least one reference finding feature amount representing an image feature for the findings included in each of the reference images and at least one reference normal feature amount representing an image feature for an image in the case where the findings included in each of the reference images are assumed to be normal regions, deriving a similarity between the query image and each of the plurality of reference images based on a comparison of the additional finding feature amount and the query normal feature amount with the reference finding feature amount and the reference normal feature amount; The image retrieval method further comprises extracting the reference images similar to the query image from an image database as similar images based on the similarity.

15. A step of receiving an input of finding information representing at least one finding desired to be searched for a query base image that is a source of the query image and includes a region desired to be searched, thereby deriving a query image to which the finding is added; deriving at least one additional finding feature representing an image feature for the added finding; deriving at least one query normal feature representing an image feature for a normal region included in the part in the query base image; a step of referring to an image database in which a plurality of reference images including findings are registered, the plurality of reference images being associated with at least one reference finding feature representing an image feature for a finding included in each of the reference images and at least one reference normal feature representing an image feature for an image in a case where the finding included in each of the reference images is a normal region, and deriving a similarity between the query image and each of the plurality of reference images based on a comparison of the additional finding feature and the query normal feature with the reference finding feature and the reference normal feature; and extracting the reference images similar to the query image from an image database as similar images based on the similarity.

Images