Image retrieval method, electronic device, and storage medium

Abstract

Embodiments of the present application provide an image retrieval method, an electronic device and a storage medium. The method comprises: in response to the size of a first image being greater than a preset size threshold, scaling the first image to a second image according to the preset size threshold; extracting a plurality of third images from a sliding window of the first image, wherein each third image has an image overlap with at least one other third image; performing feature extraction on the second image and the plurality of third images respectively, and correspondingly obtaining a first global feature and a plurality of first local features; and retrieving one or more first target images in a pre-constructed image library according to the plurality of first local features and the first global feature; wherein the image library is constructed by: performing clustering and nesting processing on a plurality of first candidate images obtained, to obtain a plurality of representative images; and constructing the image library according to the plurality of representative images. The embodiments of the present application can improve the accuracy of the scale adaptive retrieval of images.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of medical imaging technology, and particularly to an image retrieval method, electronic device, and storage medium. Background Technology

[0002] In related technologies, pathologists can select regions of interest on whole-slice images of tissue pathology and then search for similar pathologies in a pathology image database for case comparison, diagnostic reference, teaching research, and other purposes. However, the selected region of interest cannot be controlled, which necessitates that image retrieval support scale-adaptive retrieval.

[0003] In the process of scale-adaptive retrieval, image segmentation may be involved. In existing technologies, segmented images can describe the spatial relationship between local tissues. However, there may be differences in staining intensity, tissue sparsity, lesion distribution, etc. between different segmented images, which can easily lead to unstable retrieval results. Therefore, scale-adaptive retrieval can generally rely on artificially designed shallow features such as color and texture, and map image blocks to binary representations through hash encoding to achieve scale-adaptive retrieval. However, the information that is truly diagnostic in pathological sections is often reflected in complex high-level semantic features such as cell nuclear morphology, glandular structure, tissue arrangement and lesion infiltration patterns. Binary encoding is difficult to represent the complex morphological information of pathological tissues, resulting in low accuracy of retrieval results. Summary of the Invention

[0004] This application provides an image retrieval method, an electronic device, and a storage medium, aiming to improve the accuracy of scale-adaptive image retrieval.

[0005] On one hand, embodiments of this application provide an image retrieval method, characterized in that the method includes: responding to a first image whose size is greater than a preset size threshold, scaling the first image according to the preset size threshold to obtain a second image; extracting multiple third images from the first image using a sliding window, wherein each third image overlaps with at least one other third image; extracting features from the multiple third images to obtain multiple first local features, and extracting features from the second image to obtain first global features; performing image retrieval in a pre-constructed image library based on the multiple first local features and the first global features to obtain one or more first target images; wherein the image library is constructed through the following steps: obtaining multiple first candidate images; performing clustering nesting processing on the multiple first candidate images to obtain multiple representative images; and constructing the image library based on the multiple representative images.

[0006] In one embodiment, the step of performing image retrieval in a pre-constructed image library based on multiple first local features and first global features to obtain one or more first target images includes: performing image retrieval in the image library based on multiple first local features and first global features to obtain multiple second candidate images; ranking the multiple second candidate images according to their relevance, and determining one or more second candidate images with the highest ranking as the first target images.

[0007] In one embodiment, the step of performing image retrieval in the image library based on multiple first local features and first global features to obtain multiple second candidate images includes: obtaining retrieval constraint information input by the user; and performing image retrieval in the image library based on the retrieval constraint information, multiple first local features, and first global features to obtain multiple second candidate images.

[0008] In one embodiment, the step of ranking the relevance of multiple second candidate images and determining one or more second candidate images with the highest ranking as the first target image includes: removing duplicate second candidate images from the retrieved second candidate images based on the globally unique identifier of each second candidate image; ranking the remaining second candidate images based on relevance and determining one or more second candidate images with the highest ranking as the first target image.

[0009] In one embodiment, the step of ranking the remaining second candidate images by relevance and determining one or more of the highest-ranked second candidate images as the first target image includes: grouping the remaining second candidate images to obtain multiple candidate image groups, wherein the second candidate images in the candidate image groups belong to the same whole-slice image; determining multiple target candidate images from each of the candidate image groups, wherein the multiple target candidate images are multiple second candidate images with the highest similarity ranking in each of the candidate image groups; ranking the multiple target candidate images by relevance and determining one or more of the highest-ranked target candidate images as the first target image.

[0010] In one embodiment, the step of performing clustering nesting processing on multiple first candidate images to obtain multiple representative images includes: extracting features from each first candidate image to obtain a second global feature for each first candidate image; Clustering is performed on each of the second global features to obtain multiple clusters; in response to the generation of clusters, for each newly generated cluster, clustering is performed according to each of the second global features clustered into the newly generated cluster, and a new cluster is generated in the newly generated cluster; in response to the completion of the last layer of clustering processing, for each cluster of the last layer, the second global feature that is closest to the spatial cluster center of the last layer cluster is determined as the target global feature; the first candidate image to which each target global feature belongs is determined as the representative image.

[0011] In one embodiment, the method further includes: in response to the first image having a size smaller than the preset size threshold, filling the first image according to the preset size threshold to obtain a fourth image; extracting features from the fourth image to obtain a third global feature; and performing image retrieval in the image library according to the third global feature to obtain one or more second target images.

[0012] In one embodiment, the method further includes: in response to the size of the first image being equal to the preset size threshold, performing feature extraction on the first image to obtain a fourth global feature; and performing image retrieval in the image library based on the fourth global feature to obtain one or more third target images.

[0013] On the other hand, embodiments of this application also provide an electronic device, including: at least one processor; at least one memory for storing at least one program; and when at least one of the programs is executed by at least one of the processors, implementing the image retrieval method as described above.

[0014] On the other hand, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for performing the image retrieval method as described above.

[0015] On the other hand, embodiments of this application also provide a computer program product, including a computer program or computer instructions, the computer program or computer instructions being stored in a computer-readable storage medium, a processor of an electronic device reading the computer program or computer instructions from the computer-readable storage medium, and the processor executing the computer program or computer instructions to cause the electronic device to perform the image retrieval method as described above.

[0016] This application provides an image retrieval method, electronic device, and storage medium. The method includes: responding to a first image whose size is greater than a preset size threshold, scaling the first image according to the preset size threshold to obtain a second image; extracting multiple third images from the first image using a sliding window, wherein each third image overlaps with at least one other third image; extracting features from the multiple third images to obtain multiple first local features, and extracting features from the second image to obtain first global features; performing image retrieval in a pre-constructed image library based on the multiple first local features and the first global features to obtain one or more first target images; wherein the image library is constructed through the following steps: obtaining multiple first candidate images; performing clustering nesting processing on the multiple first candidate images to obtain multiple representative images; and constructing the image library based on the multiple representative images. By using clustering nesting to classify the first candidate images, the first candidate images can be classified in a fine-grained manner, thereby enabling the image library to provide finer-grained retrieval capabilities, and thus making the retrieved target images more accurate. Based on this, since multiple first local features and first global features are used for retrieval, during the retrieval process, the first local features can provide the first global features with detailed information lost due to image scaling, and the first global features can provide the first local features with global detailed information. Since there is overlap between the third images, each first local feature contains correlation information with other third images. In this way, the information introduced during the retrieval process can enable the image library to more accurately detect the first target image most relevant to the first image, effectively reducing the situation where the retrieval is inaccurate due to the selected first image exceeding the preset size threshold, thereby improving the accuracy of scale-adaptive image retrieval. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating the steps of an image retrieval method provided in one embodiment of this application; Figure 2 yes Figure 1 A flowchart illustrating an embodiment of a sub-step of step 140; Figure 3 yes Figure 2 A flowchart illustrating an embodiment of a sub-step of step 220; Figure 4 yes Figure 2 A flowchart illustrating another sub-step embodiment of step 220; Figure 5 yes Figure 4 A flowchart illustrating an embodiment of a sub-step of step 420; Figure 6This is a schematic diagram of the image library construction process provided in the embodiments of this application; Figure 7 This is a schematic diagram illustrating the steps of an image retrieval method provided in another embodiment of this application; Figure 8 This is a schematic diagram of the steps of an image retrieval method provided in another embodiment of this application; Figure 9 This is a schematic diagram illustrating the implementation principle of the image retrieval method provided in the embodiments of this application; Figure 10 This is a schematic diagram of the search results provided in an embodiment of this application; Figure 11 This is a schematic diagram of the architecture of the image retrieval device provided in the embodiments of this application; Figure 12 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that although the flowchart shows a logical order, in some cases, the steps shown or described may be executed in a different order than that shown in the flowchart. In the description of the embodiments of this application, "multiple" (or more than) means two or more, "greater than," "less than," and "exceeding" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. If "first," "second," etc., are described, they are only used to distinguish technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated. Furthermore, in the description of the embodiments of this application, the various values ​​mentioned (such as first value, second value, etc.) can be flexibly represented as a single numeric code or an enumerated type value.

[0020] In related technologies, pathologists can select regions of interest on whole-section tissue pathology images and then search for similar pathologies in a pathology image database for case comparison, diagnostic reference, teaching research, etc. However, the selected region of interest cannot be controlled, which necessitates that image retrieval support scale-adaptive retrieval.

[0021] In the process of scale-adaptive retrieval, image segmentation may be involved. In existing technologies, segmented images can describe the spatial relationship between local tissues. However, there may be differences in staining intensity, tissue sparsity, lesion distribution, etc. between different segmented images, which can easily lead to unstable retrieval results. Therefore, scale-adaptive retrieval can generally rely on artificially designed shallow features such as color and texture, and map image blocks to binary representations through hash encoding to achieve scale-adaptive retrieval. However, the information that is truly diagnostic in pathological sections is often reflected in complex high-level semantic features such as cell nuclear morphology, glandular structure, tissue arrangement and lesion infiltration patterns. Binary encoding is difficult to represent the complex morphological information of pathological tissues, resulting in low accuracy of retrieval results.

[0022] To improve the accuracy of scale-adaptive image retrieval, embodiments of this application provide an image retrieval method, electronic device, computer-readable storage medium, and computer program product. The method includes: responding to a first image whose size is greater than a preset size threshold, scaling the first image according to the preset size threshold to obtain a second image; extracting multiple third images from the first image using a sliding window, wherein each third image overlaps with at least one other third image; extracting features from the multiple third images to obtain multiple first local features, and extracting features from the second image to obtain first global features; performing image retrieval in a pre-constructed image library based on the multiple first local features and the first global features to obtain one or more first target images; wherein the image library is constructed through the following steps: obtaining multiple first candidate images; performing clustering nesting on the multiple first candidate images to obtain multiple representative images; and constructing the image library based on the multiple representative images. By using clustering nesting to classify the first candidate images, fine-grained classification of the first candidate images is possible, thereby enabling the image library to provide finer-grained retrieval capabilities and making the retrieved target images more accurate. Based on this, since multiple first local features and first global features are used for retrieval, during the retrieval process, the first local features can provide the first global features with detailed information lost due to image scaling, and the first global features can provide the first local features with global detailed information. Since there is overlap between the third images, each first local feature contains correlation information with other third images. In this way, the information introduced during the retrieval process can enable the image library to more accurately detect the first target image most relevant to the first image, effectively reducing the situation where the retrieval is inaccurate due to the selected first image exceeding the preset size threshold, thereby improving the accuracy of scale-adaptive image retrieval.

[0023] See Figure 1 , Figure 1The steps of an image retrieval method provided in an embodiment of this application are illustrated. In one embodiment, the image retrieval method may include the following steps.

[0024] Step 110: In response to the fact that the size of the obtained first image is greater than a preset size threshold, the first image is scaled according to the preset size threshold to obtain the second image; Step 120: Extract multiple third images from the first image sliding window, wherein each third image overlaps with at least one other third image; Step 130: Extract features from multiple third images to obtain multiple first local features, and extract features from the second image to obtain first global features; Step 140: Perform image retrieval in a pre-built image library based on multiple first local features and first global features to obtain one or more first target images; wherein, the image library is constructed through the following steps: obtaining multiple first candidate images; performing clustering and nesting processing on the multiple first candidate images to obtain multiple representative images; and constructing the image library based on the multiple representative images.

[0025] In one embodiment, the first image refers to the image selected by the user on the full slice image to be retrieved.

[0026] In one embodiment, scaling the first image according to a preset size threshold to obtain the second image refers to scaling the first image to obtain a second image with a size equal to the preset size threshold.

[0027] In one embodiment, image overlap between each third image and at least one other third image means that each third image has partially identical content with at least one other third image.

[0028] For example, suppose the size of the first image is 10×10 and the sliding window size is 6×6. Then, the first image can be used to extract four third images by the sliding window. Assuming the extraction order is from left to right and from top to bottom, the right edge of the first third image and the left edge of the second third image overlap, the bottom edge of the first third image and the top edge of the third third image overlap, and the bottom right corner of the first third image and the top left corner of the fourth third image overlap.

[0029] In one embodiment, feature extraction is performed on multiple third images to obtain multiple first local features. This means that for each third image, feature extraction is performed on that third image to obtain its first local features. Here, the first local features refer to the global features of the third image. Furthermore, the first local features can be extracted using neural network models such as the CONCH model or the CHIEF model; no specific limitation is made here.

[0030] In one embodiment, the first global feature refers to the global features of the second image. The first global feature can map the feature information of the first image.

[0031] In one embodiment, image retrieval in a pre-built image library based on multiple first local features and first global features to obtain one or more first target images refers to the operation of retrieving images corresponding to features similar to the multiple first local features and first global features from the pre-built image library to obtain one or more first target images. The pre-built image library can be a Milvus vector database, a Qdrant database, a Weaviate database, etc., and is not specifically limited here.

[0032] It should be noted that, in the embodiments of this application, the image library can calculate the similarity between the input features and the features corresponding to the stored images, and return the retrieval results based on the calculated similarity.

[0033] In one embodiment, during the process of retrieving images from a pre-built image library based on multiple first local features and first global features to obtain one or more first target images, the multiple first local features and first global features can be fused to obtain target features. Then, based on the target features, the image library is searched, and one or more images with high similarity to the target features are identified as first target images.

[0034] In one embodiment, during the process of retrieving images in a pre-built image library based on multiple first local features and first global features to obtain one or more first target images, specifically, the image library can be searched based on each first local feature, and images with high similarity between one or more features and the first local features can be identified as first target images; then, the image library can be searched based on the first global features, and images with high similarity between one or more features and the first global features can be identified as first target images.

[0035] It is important to note that during the process of retrieving the first target image, the availability of the search results must be ensured. Therefore, the retrieved images need to be filtered for relevance in order to retain the usable first target images.

[0036] See Figure 2 In one embodiment, step 140 may include the following sub-steps.

[0037] Step 210: Perform image retrieval in the image database based on multiple first local features and first global features to obtain multiple second candidate images; Step 220: Rank the multiple second candidate images according to their relevance, and determine one or more second candidate images with the highest ranking as the first target image.

[0038] In one embodiment, performing image retrieval in an image library based on multiple first local features and first global features to obtain one or more first target images refers to the operation of retrieving images corresponding to features similar to the multiple first local features and first global features from a pre-constructed image library to obtain one or more second candidate images. The second candidate images refer to images initially selected from the image library; after candidate filtering, the remaining second candidate images are the first target images.

[0039] In one embodiment, during the process of retrieving images from an image library based on multiple first local features and first global features to obtain multiple second candidate images, the multiple first local features and first global features can be fused to obtain target features. Then, based on the target features, images with high similarity to the target features are retrieved from a pre-built image library, and one or more images are identified as second candidate images.

[0040] In one embodiment, during the process of obtaining one or more first target images by performing image retrieval in a pre-built image library based on multiple first local features and first global features, specifically, images with high similarity to the first local features can be identified as second candidate images by retrieving images in the pre-built image library based on each first local feature; then, images with high similarity to the first global features can be identified as second candidate images by retrieving images in the pre-built image library based on the first global features.

[0041] In one embodiment, ranking multiple second candidate images based on their relevance and determining one or more of the highest-ranked second candidate images as the first target image involves determining the relevance value between each second candidate image and the first image; then, ranking each second candidate image according to its relevance value, and determining one or more of the highest-ranked second candidate images as the first target image. The relevance value can be determined using methods such as Maximum Marginal Relevance (MMR) or feature similarity, and is not specifically limited here.

[0042] By ranking multiple second candidate images based on their relevance, we can ensure their relevance to the first image while also introducing constraints on the differences between the candidate results. This improves the diversity and usability of the target image during the determination process.

[0043] It is important to note that during the image library construction process, each image added to the library is also stored along with some information, such as staining type and organ type. At this time, the search range of image blocks can be constrained by inputting some information, which can make the search results more accurate.

[0044] See Figure 3 In one embodiment, step 220 may include the following sub-steps.

[0045] Step 310: Obtain the search constraint information input by the user; Step 320: Based on the retrieval constraint information, multiple first local features and first global features, perform image retrieval in the image database to obtain multiple second candidate images.

[0046] In one embodiment, retrieval constraint information refers to information used to indicate the scope of the search required by the user. The retrieval constraint information may include at least one of the following: the staining type to be searched, or the organ type to be searched.

[0047] In one embodiment, performing image retrieval in an image library based on retrieval constraint information, multiple first local features, and first global features to obtain multiple second candidate images means determining the matching images in the image library based on the retrieval constraint information, and then searching among the matching images based on multiple first local features and first global features, and determining images with one or more features that have a high similarity to the target features as second candidate images.

[0048] In one embodiment, in the process of retrieving images in an image library based on retrieval constraint information, multiple first local features, and first global features to obtain multiple second candidate images, specifically, images that meet the requirements can be determined in the image library based on the retrieval constraint information firstly, then feature fusion can be performed on multiple first local features and first global features to obtain target features, and then, images that meet the requirements can be retrieved based on the target features, and images with one or more features that have a high similarity to the target features can be determined as second candidate images.

[0049] In one embodiment, during the process of retrieving images from an image library based on retrieval constraint information, multiple first local features, and first global features to obtain multiple second candidate images, specifically, images that meet the criteria can be determined first based on the retrieval constraint information in the image library; then, images that have a high similarity between one or more features and the first local features are determined as second candidate images based on each first local feature; and finally, images that have a high similarity between one or more features and the first global features are determined as second candidate images based on the first global features.

[0050] By introducing search constraint information into the image retrieval process within the image library, the candidate pool can be narrowed and irrelevant matches reduced, making subsequent candidate images more relevant to the first image. Furthermore, the introduction of search constraint information increases user involvement in the image library retrieval process, enhancing the interactivity between the user and the image library.

[0051] It should be noted that duplicate second candidate images may be retrieved during the retrieval process. To ensure the diversity of the retrieval results, deduplication can be performed after the second candidate images are retrieved to ensure the diversity of the first target image.

[0052] See Figure 4 In one embodiment, step 220 may include the following sub-steps.

[0053] Step 410: Based on the globally unique identifier of each second candidate image, remove duplicate second candidate images from the retrieved second candidate images; Step 420: Sort the remaining second candidate images by relevance, and determine one or more second candidate images with the highest relevance as the first target image.

[0054] In one embodiment, a globally unique identifier refers to a unique identifier for an image in the image database.

[0055] In one embodiment, removing duplicate second candidate images from the retrieved second candidate images based on the globally unique identifier of each second candidate image means removing the duplicate second candidate image with the same globally unique identifier, leaving only one second candidate image.

[0056] For example, suppose there are multiple second candidate images retrieved. There are two second candidate images with globally unique identifier A and four second candidate images with globally unique identifier C. In this case, remove one second candidate image with globally unique identifier A and three second candidate images with globally unique identifier C, leaving only one second candidate image with globally unique identifier A and one second candidate image with globally unique identifier C.

[0057] In one embodiment, ranking the remaining second candidate images by relevance and determining one or more of the highest-ranked second candidate images as the first target image means that for each remaining second candidate image, the relevance value between the second candidate image and the first image is determined; then, based on the relevance value of each remaining second candidate image, each second candidate image is ranked, and one or more of the highest-ranked second candidate images are determined as the first target image.

[0058] By using the globally unique identifier of each second candidate image, duplicate second candidate images obtained from repeated searches are extracted. This avoids duplicate second candidate images being ranked high during subsequent relevance ranking, thus making the ranked second candidate images more diverse, resulting in a more diverse target image and providing more reference value for the first image.

[0059] It is important to note that after deduplication, the second candidate images obtained in the actual retrieval are highly similar to each other. These second candidate images may belong to different full slice images. If relevance ranking is performed directly, most of the final first target images may belong to the same full slice image. Such results are overly concentrated and will affect the representativeness of the retrieval results. Therefore, grouping can be used to avoid excessive concentration of results.

[0060] See Figure 5 In one embodiment, step 420 may include the following sub-steps.

[0061] Step 510: Group the remaining second candidate images to obtain multiple candidate image groups, wherein the second candidate images in the candidate image groups belong to the same full slice image; Step 520: Determine multiple target candidate images from each candidate image group, wherein the multiple target candidate images are the multiple second candidate images with the highest similarity ranking in each candidate image group; Step 530: Rank the multiple target candidate images according to their relevance, and determine one or more target candidate images with the highest relevance as the first target image.

[0062] In one embodiment, grouping the remaining second candidate images into multiple candidate image groups refers to obtaining the full-slice image attribution information of each remaining second candidate image, and then, based on the full-slice image attribution information of each second candidate image, dividing the second candidate images belonging to the same full-slice image into a candidate image group. Here, the full-slice image attribution information refers to information used to indicate the full-slice image when the second candidate image is extracted by the sliding window.

[0063] In one embodiment, determining multiple target candidate images from each candidate image group means, for each candidate image group, sorting each second candidate image in the candidate image group according to the similarity between each second candidate image and the first image, and determining the multiple second candidate images with the highest similarity as target candidate images.

[0064] In one embodiment, ranking multiple target candidate images by relevance and determining one or more target candidate images with the highest ranking as the first target image means that for each target candidate image, the relevance value between the second candidate image and the first image is determined; then, each target candidate image is ranked according to the relevance value of each target candidate image, and one or more target candidate images with the highest ranking are determined as the first target image.

[0065] The above embodiments mentioned an image library. The following describes the relevant content of building an image library.

[0066] In one embodiment, the first candidate image refers to an image extracted by sliding a window from a full-slice image used as an input sample. The size of the first candidate image is the same as a preset size threshold.

[0067] In one embodiment, nested clustering refers to clustering multiple images to obtain multiple clusters, and then performing further clustering operations within these clusters. The number of nested clustering levels varies and depends on the specific circumstances.

[0068] See Figure 6 In one embodiment, the process of “clustering multiple first candidate images to obtain multiple representative images” may include the following steps.

[0069] Step 610: Extract features from each first candidate image to obtain the second global features of each first candidate image; Step 620: Cluster each of the second global features to obtain multiple clusters; Step 630: In response to the generation of clusters, for each newly generated cluster, cluster according to each second global feature that was clustered into the newly generated cluster, and generate a new cluster in the newly generated cluster; Step 640: In response to the completion of the last layer of clustering, for each cluster in the last layer, the second global feature that is closest to the spatial cluster center of the last layer cluster is determined as the target global feature. Step 650: Determine the first candidate image to which the global features of each target belong as the representative image.

[0070] In one embodiment, the first candidate image can be used to extract second global features through neural network models such as the CONCH model and the CHIEF model.

[0071] In one embodiment, in response to the generation of clusters, for each newly generated cluster, clustering is performed based on the second global features of those clustered into the newly generated cluster. Generating new clusters within these newly generated clusters means that whenever a cluster is generated, image clustering is performed again on each newly generated cluster, generating new clusters under the newly generated clusters. The clustering can be K-means clustering, Bayesian GMM clustering, etc., and is not specifically limited here.

[0072] For example, assuming the clusters obtained in step 620 are A and B, then, in response to the generation of cluster A, clustering is performed based on the second global features that are clustered to cluster A, resulting in clusters A1 and A2 within cluster A; then, in response to the generation of cluster A1, clustering is performed based on the second global features that are clustered to cluster A1, resulting in clusters A11 and A12 within cluster A1. Similarly, A111 and A112 can be clustered within cluster A11, and clustering can be performed within cluster A12. A121, A122, and so on, until cluster A1 is nested to the lowest level; the same applies to cluster A2. In response to the generation of cluster A2, clustering is performed based on the second global features of those clustered into cluster A2, resulting in clusters A21 and A22 in cluster A2. Similarly, A211 and A212 can be clustered in cluster A21, and A221 and A222 can be clustered in cluster A22, and so on, until cluster A2 is nested to the lowest level.

[0073] Similarly, cluster B is also nested until cluster B is nested to the lowest level.

[0074] It's important to note that each nested cluster represents a different cluster type. Therefore, nested clustering can simultaneously meet multiple clustering needs. For example, based on the example above, the first-level clustering of A and B could be about cell morphology; the second-level clustering of A1, A2, B1, and B2 could be about cell density; the third-level clustering of A11, A12, A21, A22, B11, B12, B21, and B22 could be about staining degree, and so on. This allows for simultaneous clustering of different types such as cell morphology, cell density, and staining degree, ensuring accurate classification of each first candidate image. This results in finer-grained classification of representative images in the image library, providing finer-grained retrieval capabilities and ultimately leading to more accurate retrieved target images.

[0075] In one embodiment, determining the second global feature that is closest to the spatial cluster center of the last layer of clusters as the target global feature means determining the feature distance between each second global feature in the last layer of clusters and the spatial cluster center of the last layer of clusters, and then determining the second global feature corresponding to the minimum distance as the target global feature.

[0076] It should be noted that the above are the steps for building an image library. However, images in an image library can be added. In this case, the same steps can be used to add images to the image library.

[0077] In one embodiment, multiple second candidate images can be obtained first, then clustered and nested processing can be performed on the multiple second candidate images, and then the multiple images obtained from the clustering and nesting processing can be added to the image library.

[0078] In one embodiment, constructing an image library based on multiple representative images refers to the operation of structurally encapsulating the representative image and its related information for each representative image and writing it into the image library.

[0079] In one embodiment, the relevant information includes at least one of the following: a second global feature representing the image, used to represent information about the whole slice image to which the image belongs, the organ type corresponding to the content in the image, the staining type corresponding to the content in the image, metadata representing the image, and a globally unique identifier representing the image in the image library.

[0080] By using nested clustering to classify the first candidate image, the first candidate image can be classified in a fine-grained manner, thereby enabling the image library to provide more fine-grained retrieval capabilities and making the retrieved target image more accurate.

[0081] Based on this, since multiple first local features and first global features are used for retrieval, during the retrieval process, the first local features can provide the first global features with detailed information lost due to image scaling, and the first global features can provide the first local features with global detailed information. Since there is overlap between the third images, each first local feature contains the correlation information between it and other third images. In this way, the information introduced during the retrieval process can enable the image library to more accurately detect the first target image most related to the first image, effectively solving the problem of inaccurate retrieval due to the selected first image exceeding the preset size threshold.

[0082] It should be noted that in the above-mentioned embodiment where the size of the first image is greater than the preset size threshold, image retrieval can also be performed based on the same principle as the above embodiment when the selected first image is smaller than the preset size threshold.

[0083] See Figure 7 In one embodiment, the embodiment where the first image is smaller than a preset size threshold may include the following steps.

[0084] Step 710: In response to the fact that the size of the first image is smaller than a preset size threshold, the first image is filled according to the preset size threshold to obtain the fourth image; Step 720: Extract features from the fourth image to obtain the third global feature; Step 730: Perform image retrieval in the image library based on the third global feature to obtain one or more second target images.

[0085] In one embodiment, filling the first image with a preset size threshold to obtain a fourth image refers to filling the edges of the first image with useless image blocks to obtain a fourth image with a size equal to the preset size threshold.

[0086] In one embodiment, the third global feature refers to the global features of the fourth image. The third global feature can map the feature information of the first image. The third global feature can be extracted using neural network models such as the CONCH model or the CHIEF model; no specific method is limited here.

[0087] In one embodiment, retrieving one or more second target images from an image library based on a third global feature refers to the operation of retrieving images from a pre-built image library that correspond to features similar to the third global feature in order to retrieve one or more second target images.

[0088] It is important to note that during the process of retrieving a second target image, the availability of the search results must be ensured. Therefore, the retrieved images need to be filtered for relevance in order to retain usable second target images.

[0089] In one embodiment, during the process of retrieving images from an image library based on third global features to obtain one or more second target images, specifically, the image library can be searched based on third global features to obtain multiple third candidate images. Then, the multiple third candidate images are ranked by relevance, and one or more third candidate images with the highest ranking are determined as the second target images.

[0090] In one embodiment, the third candidate image refers to the image initially selected from the image library based on the third global feature; after the third candidate image is filtered, the remaining third candidate image is the second target image.

[0091] It is important to note that during the image library construction process, each image added to the library is also stored along with some information. At this time, the search range of image blocks can be constrained by inputting some information, which can make the search results more accurate.

[0092] In one embodiment, in the process of retrieving images from an image library based on third global features to obtain multiple third candidate images, the user-inputted retrieval constraint information can be obtained first, and then the image retrieval can be performed in the image library based on the retrieval constraint information and the third global features to obtain multiple third candidate images.

[0093] In one embodiment, performing image retrieval in the image library based on retrieval constraint information and third global features to obtain multiple third candidate images means determining the images that meet the criteria in the image library based on the retrieval constraint information, and then searching among the images that meet the criteria based on the third global features, and determining images with one or more features that have a high similarity to the target features as third candidate images.

[0094] In one embodiment, ranking multiple third candidate images based on their relevance and determining one or more of the highest-ranked third candidate images as the second target image involves determining the relevance value between each third candidate image and the first image; then, ranking each third candidate image according to its relevance value, and determining one or more of the highest-ranked third candidate images as the second target image. The relevance value can be determined using methods such as MMR (Mean Modulation Ratio) and feature similarity, which are not specifically limited here.

[0095] It should be noted that duplicate third candidate images may be retrieved during the retrieval process. To ensure the diversity of retrieval results, deduplication can be performed after the second candidate image is retrieved to ensure the diversity of the second target image.

[0096] In one embodiment, in the process of ranking multiple third candidate images by relevance and determining one or more third candidate images with the highest ranking as the second target image, the process can be as follows: First, based on the globally unique identifier of each third candidate image, duplicate third candidate images in the retrieved third candidate images can be removed. Then, the remaining third candidate images can be ranked by relevance, and one or more third candidate images with the highest ranking can be determined as the second target image.

[0097] In one embodiment, removing duplicate third candidate images from the retrieved third candidate images based on the globally unique identifier of each third candidate image means removing the duplicate third candidate image with the same globally unique identifier, leaving only one third candidate image.

[0098] In one embodiment, ranking the remaining third candidate images by relevance and determining one or more of the highest-ranked third candidate images as the second target image means that for each remaining third candidate image, the correlation value between the third candidate image and the first image is determined; then, based on the correlation value of each remaining third candidate image, each third candidate image is ranked, and one or more of the highest-ranked third candidate images are determined as the second target image.

[0099] It is important to note that after deduplication, the actual third candidate images retrieved are highly similar, and these third candidate images may belong to different full slice images. If relevance ranking is performed directly, most of the final second target images may belong to the same full slice image. Such results are overly concentrated and will affect the representativeness of the search results. Therefore, grouping can be used to avoid excessive concentration of results.

[0100] In one embodiment, in the process of ranking the remaining third candidate images by relevance and determining one or more of the highest-ranked third candidate images as the second target images, the remaining third candidate images can be first grouped to obtain multiple first target candidate image groups, wherein the third candidate images in the first target candidate image groups belong to the same full slice image; then, the multiple third candidate images with the highest similarity ranking in each first target candidate image group are ranked by relevance, and one or more of the highest-ranked target candidate images are determined as the second target images.

[0101] In one embodiment, grouping the remaining third candidate images to obtain multiple first target candidate image groups refers to determining the full slice image affiliation of each remaining third candidate image, and then dividing the third candidate images belonging to the same full slice image into a first target candidate image group.

[0102] In one embodiment, the relevance ranking of multiple third candidate images with the highest similarity in each first target candidate image group is performed, and one or more target candidate images with the highest ranking are determined as second target images. This means that for each third candidate image with the highest similarity ranking, the relevance value between the third candidate image and the first image is determined; then, according to the relevance value of each third candidate image, each third candidate image is ranked, and one or more third candidate images with the highest ranking are determined as second target images.

[0103] By using nested clustering to classify the first candidate image, fine-grained classification is achieved, enabling the image library to provide more granular retrieval capabilities and thus more accurate target images are retrieved. Furthermore, utilizing a third global feature for retrieval allows the image library to more accurately detect the first target image most relevant to the first image, effectively reducing inaccuracies caused by the selected first image being smaller than a preset size threshold, thereby improving the accuracy of scale-adaptive image retrieval.

[0104] It should be noted that in the above-mentioned embodiment where the size of the first image is greater than the preset size threshold, image retrieval can also be performed based on the same principle as the above embodiment when the selected first image is equal to the preset size threshold.

[0105] See Figure 8 In one embodiment, the embodiment where the first image is equal to a preset size threshold may include the following steps.

[0106] Step 810: In response to the first image's size being equal to a preset size threshold, feature extraction is performed on the first image to obtain the fourth global feature; Step 820: Perform image retrieval in the image library based on the fourth global feature to obtain one or more third target images.

[0107] In one embodiment, the fourth global feature refers to the global features of the first image, that is, the first global feature mentioned above.

[0108] In one embodiment, retrieving one or more third target images from an image library based on a fourth global feature refers to the operation of retrieving images from a pre-built image library that correspond to features similar to the fourth global feature in order to retrieve one or more third target images.

[0109] It is important to note that during the retrieval of third-party target images, the usability of the retrieval results must be ensured. Therefore, the retrieved images need to be filtered for relevance in order to retain usable third-party target images.

[0110] In one embodiment, during the process of retrieving images from an image library based on the fourth global feature to obtain one or more third target images, specifically, the image library can be searched based on the fourth global feature to obtain multiple fourth candidate images. Then, the multiple fourth candidate images are ranked by relevance, and one or more fourth candidate images with the highest ranking are determined as the third target images.

[0111] In one embodiment, the fourth candidate image refers to the image initially selected from the image library based on the fourth global feature; after the fourth candidate image is filtered, the remaining fourth candidate image is the third target image.

[0112] It is important to note that during the image library construction process, each image added to the library is also stored along with some information. At this time, the search range of image blocks can be constrained by inputting some information, which can make the search results more accurate.

[0113] In one embodiment, in the process of retrieving images from the image library based on the fourth global feature to obtain multiple fourth candidate images, the user-inputted retrieval constraint information can be obtained first, and then the image retrieval can be performed in the image library based on the retrieval constraint information and the fourth global feature to obtain multiple fourth candidate images.

[0114] In one embodiment, performing image retrieval in the image library based on retrieval constraint information and fourth global features to obtain multiple fourth candidate images means determining the images that meet the criteria in the image library based on the retrieval constraint information, and then searching among the images that meet the criteria based on the fourth global features, and determining images with one or more features that have a high similarity to the target features as fourth candidate images.

[0115] In one embodiment, ranking multiple fourth candidate images based on their relevance and determining one or more of the highest-ranked fourth candidate images as the third target image involves determining the relevance value between each fourth candidate image and the first image; then, ranking each fourth candidate image according to its relevance value, and determining one or more of the highest-ranked fourth candidate images as the third target image. The relevance value can be determined using methods such as MMR (Mean Modulation Ratio) and feature similarity, which are not specifically limited here.

[0116] It should be noted that during the retrieval of the fourth candidate image, duplicate fourth candidate images may be retrieved. In order to ensure the diversity of retrieval results, after retrieving the second candidate image, deduplication can be performed to ensure the diversity of the third target image.

[0117] In one embodiment, in the process of ranking multiple fourth candidate images by relevance and determining one or more fourth candidate images with the highest ranking as the third target image, the process can be as follows: First, based on the globally unique identifier of each fourth candidate image, duplicate fourth candidate images in the retrieved fourth candidate images can be removed. Then, the remaining fourth candidate images can be ranked by relevance, and one or more fourth candidate images with the highest ranking can be determined as the third target image.

[0118] In one embodiment, removing duplicate fourth candidate images from the retrieved fourth candidate images based on the globally unique identifier of each fourth candidate image means removing the duplicate fourth candidate image with the same globally unique identifier, leaving only one fourth candidate image.

[0119] In one embodiment, ranking the remaining fourth candidate images by relevance and determining one or more of the highest-ranked fourth candidate images as the third target image means that for each remaining fourth candidate image, the relevance value between the fourth candidate image and the first image is determined; then, based on the relevance value of each remaining fourth candidate image, each fourth candidate image is ranked, and one or more of the highest-ranked fourth candidate images are determined as the third target image.

[0120] It is important to note that after deduplication, the actual fourth candidate images retrieved are highly similar to each other. These fourth candidate images may belong to different full slice images. If relevance ranking is performed directly, most of the final third target images may belong to the same full slice image. Such results are overly concentrated and will affect the representativeness of the search results. Therefore, grouping can be used to avoid excessive concentration of results.

[0121] In one embodiment, in the process of ranking the remaining fourth candidate images by relevance and determining one or more fourth candidate images with the highest ranking as the third target image, the remaining fourth candidate images can be first grouped to obtain multiple second target candidate image groups, wherein the fourth candidate images in the second target candidate image groups belong to the same full slice image; then, the multiple fourth candidate images with the highest similarity ranking in each second target candidate image group are ranked by relevance, and one or more target candidate images with the highest ranking are determined as the third target image.

[0122] In one embodiment, grouping the remaining fourth candidate images to obtain multiple second target candidate image groups refers to determining the full slice image affiliation of each remaining fourth candidate image, and then dividing the fourth candidate images belonging to the same full slice image into a second target candidate image group.

[0123] In one embodiment, the relevance ranking of multiple fourth candidate images with the highest similarity in each second target candidate image group is performed, and one or more target candidate images with the highest ranking are determined as the third target image. This means that for each fourth candidate image with the highest similarity ranking, the relevance value between the fourth candidate image and the first image is determined; then, according to the relevance value of each fourth candidate image, each fourth candidate image is ranked, and one or more fourth candidate images with the highest ranking are determined as the third target image.

[0124] By using nested clustering to classify the first candidate image, fine-grained classification is achieved, enabling the image library to provide more granular retrieval capabilities and thus more accurate target images are retrieved. Furthermore, utilizing a fourth global feature for retrieval allows the image library to more accurately detect the first target image most relevant to the first image, effectively reducing inaccuracies caused by the selected first image being equal to a preset size threshold, thereby improving the accuracy of scale-adaptive image retrieval.

[0125] Based on the above embodiments, some specific embodiments are given below.

[0126] See Figure 9 First, an image library needs to be built.

[0127] Step 1: Read the whole tissue pathology slide image file, obtain its thumbnail, perform tissue segmentation based on the thumbnail to identify the effective tissue area, and remove the background area and invalid area.

[0128] Step 2: At the target magnification (default 20x scope, image patch size is 1024*1024), traverse the effective tissue region in a sliding window manner to generate candidate image patch coordinates, and filter according to the overlap ratio between the candidate region and the tissue region, filtering out candidate image patches (i.e., the first candidate image) containing ≤30% foreground tissue region, to ensure that the candidate image patch has effective tissue information.

[0129] Step 3: Representative selection of candidate image patches. In this specific embodiment, a two-layer nested clustering is used, specifically including: First, mapping the coordinates of candidate image patches to the thumbnail coordinate system according to the resolution mapping relationship between the full slice image and the thumbnail, and cropping the corresponding candidate region image on the thumbnail, uniformly scaling it to a fixed size of 64*64, and then extracting the color histogram features (i.e., the second global features) as the appearance feature representation of the candidate image patch; Next, performing a first-layer K-means clustering on all valid candidate image patches based on the appearance features, dividing the candidate image patches into several appearance clusters according to the preset appearance cluster number, so that candidate image patches with similar appearance features are grouped into the same cluster. Then, for each appearance cluster, performing a second-layer K-means clustering based on the spatial coordinates of the candidate image patches, further dividing the candidate image patches in the appearance cluster into several spatial sub-clusters according to the spatial distribution, and selecting the candidate image patch closest to the center of each spatial sub-cluster as the representative sample of the sub-cluster, thereby achieving spatial uniform sampling within each appearance cluster. Subsequently, the set of representative image patches selected from all appearance clusters is summarized, reducing the number of candidate image patches from thousands to tens of thousands to tens to hundreds. All of the above operations are performed only at the thumbnail level, without accessing high-resolution images, thereby achieving efficient extraction of representative image patches from whole pathological sections.

[0130] Step 4: Parallel reading of representative image blocks (i.e., representative images) extracted in Step 3 from the full-slice high-resolution image level, inputting each image block into the CONCH pathological image basic model for feature extraction and encoding, resulting in a 512-dimensional feature vector.

[0131] Step 5: Generate a globally unique identifier for each representative image block record, and encapsulate the image block coordinates, the whole slice information, organ type, staining type, feature vector, and metadata in a structured manner. Write these into the Milvus vector database in batches to complete the indexing and storage processing of the whole slice pathological images.

[0132] A search can only be performed after the image library is completed.

[0133] Step 6: Initiate a search request based on the region of interest (i.e., the first image) selected by the user on the whole-slice pathological image. Read the target region (i.e., the first image) from the whole-slice image according to the rectangular coordinates specified by the user, and perform resolution alignment according to the target magnification to ensure that the representation scale of the query region is consistent with that of the image patch dataset in the database.

[0134] Step 7: Perform adaptive feature extraction on the read region of interest (ROI), specifically including: determining whether the ROI size is smaller than the preset image patch size; if the condition is met, fill the ROI with white pixels to the preset size threshold, and input the filled ROI (i.e., the fourth image) into the feature extraction model to obtain a single-path query vector (i.e., the third global feature); if the ROI size is the same as the preset size threshold, directly input the ROI into the feature extraction model for encoding to obtain a single-path query vector (i.e., the fourth global feature); if the ROI patch size is larger than the preset size threshold, use a combination of global and local encoding: scale the entire ROI to the preset size threshold, and obtain... The obtained image (i.e., the second image) is input into the feature extraction model to obtain a global query vector (i.e., the first global feature) that reflects the global semantic information of the region of interest. A sliding window with overlap is performed on the region of interest with a preset step size to obtain multiple local sub-regions (i.e., the third image) that are consistent with the preset size threshold. Each sub-region is input into the feature extraction model for encoding to obtain several local query vectors (i.e., the first local features) that reflect the local details of interest. Among them, the right edge, the bottom edge, and the bottom right corner of the region of interest are supplemented to ensure that no edge region is missed. Finally, the global query vector and all local query vectors are merged to form a multi-way query vector set.

[0135] Step 8: Input the single-path or multi-path query vector set into the vector database to perform batch nearest neighbor retrieval, and support users to pre-filter based on metadata conditions such as staining type and organ type to narrow down the candidate range and reduce irrelevant matches. After the retrieval is completed, deduplication is performed according to the unique identifier of the image block, and the results are merged to form a unified candidate set.

[0136] Step 9: Group the candidate results according to the full slice to which the candidate image patch belongs, and retain only the candidates with the highest similarity in each group to avoid excessive concentration of single full slice results, which would affect the representativeness of the search results.

[0137] Step 10: See Figure 10 MMR is used to reorder candidate results, which introduces the difference constraint between candidate results while ensuring relevance to the query region, so as to improve the diversity and usability of the final returned results.

[0138] Specifically, let the query vector be... The candidate image patch set is The selected result set is The relevance weight parameter is The target returned a number of results. The definition of the MMR score for the candidate image patch can be found in formula (1).

[0139] (1) in, The function representing the similarity between vectors. This represents the feature vector corresponding to the query region. This represents the feature vector corresponding to the i-th candidate image patch. This represents the feature vector corresponding to the j-th candidate image patch. The first term in the formula measures the relevance between the candidate image patch and the query region, while the second term measures the maximum similarity between the candidate image patch and the selected results, indicating the redundancy of the candidate results. When the set of selected results... When the value is empty, the second redundancy penalty is recorded as 0.

[0140] The re-sorting process is implemented using a greedy iterative approach: in each round, the candidate image patch with the highest MMR score is selected from the remaining candidate set and added to the selected result set, and this process is repeated until the number of selected results reaches a certain threshold. The final output is the sorted result. This can be achieved by adjusting the parameters. A controllable balance between correlation and diversity: when When the value approaches 1, the search results are more biased towards relevance; when... When the value approaches 0, the search results tend to be more diverse.

[0141] In the specific embodiments described above, firstly, during the whole-slice pathological image database entry stage, the color histogram appearance features of each candidate image block are extracted from the thumbnail, and a first-level appearance clustering is performed to group candidate image blocks with similar colors and shapes into the same cluster. Subsequently, within each appearance cluster, a second-level spatial clustering is performed based on spatial coordinates, selecting the candidate closest to the center of each spatial sub-cluster as the representative sample. This method, through two-level clustering of appearance and space, ensures that the selected image blocks have both appearance diversity and spatial uniformity, while compressing the number of candidates from tens of thousands to tens to hundreds, and does not require access to high-resolution images throughout the process, significantly reducing computational, input, and output overhead.

[0142] Secondly, during the pathological slide retrieval stage, when the region of interest (ROI) is larger than the preset image patch size, a dual-path strategy combining global and local encoding is employed: the global path encodes the entire ROI after scaling to preserve macroscopic semantic information, while the local path encodes sub-regions one by one after dense cropping using overlapping sliding windows to preserve microscopic details, and supplements the cropping of edge regions to avoid information loss. Finally, the dual-path encoding results are merged into a multi-path query vector set. This method enables a single query to simultaneously represent the global structure and local texture of the ROI, enhancing its adaptability to different sizes and pathological morphologies.

[0143] Furthermore, in the candidate post-processing stage of vector retrieval, candidates are grouped according to the whole slice to which they belong, and only a few candidates with the highest similarity are retained in each group to prevent a single whole slice from monopolizing the results. Then, the maximum marginal relevance algorithm is used to greedily reorder the candidates. By weighted balancing of relevance score and redundancy score, the difference constraint is introduced while ensuring the relevance of the retrieval, so as to ensure that the final returned results have diversity in pathological morphology and improve the clinical usability of the retrieval results.

[0144] Furthermore, in the process from building the image database to realizing retrieval, the database entry stage achieves integrated processing of tissue segmentation, representative sampling, feature encoding, and structured indexing, while the retrieval stage achieves a complete process of adaptive query encoding, vector nearest neighbor retrieval, metadata filtering, group deduplication, and diversity reordering. The two stages are efficiently connected through a unified vector database and feature extraction model, forming a practical end-to-end pathological slide retrieval system.

[0145] Using the above method, the number of candidate image blocks is compressed to tens to hundreds before being stored in the database, achieving a compression ratio of over 90%. Furthermore, the sampling process is completed entirely at the thumbnail level, eliminating the need to read high-resolution images, significantly reducing the computational load for feature extraction and the storage requirements for data storage. Testing shows that under standard operating conditions, the system can complete data storage and feature processing for standard-sized digital slices within 30 seconds. Retrieving a single image from a dataset of millions of images takes less than 3 seconds. Moreover, despite the significant compression of the candidate image set within the entire slice image...

[0146] Furthermore, by using grouping and filtering based on whole-slice images to ensure the diversity of results sources, and by using MMR reordering to achieve a balance between relevance and diversity, the final search results can not only match the pathological features of the query area, but also cover a variety of different pathological manifestations in terms of morphology, providing doctors with richer reference information for diagnostic decisions.

[0147] In addition to the embodiments described above, this application also provides an image retrieval device. See also Figure 11 , Figure 11The structure of an image retrieval device 900 is shown. In one embodiment, the image retrieval device 900 may include the following modules.

[0148] The image scaling acquisition module 910 can be used to scale the first image according to the preset size threshold in response to the size of the acquired first image being greater than a preset size threshold, so as to obtain a second image. The image extraction module 920 can be used to extract multiple third images from a sliding window of a first image, wherein each third image overlaps with at least one other third image; The feature extraction module 930 can be used to extract features from multiple third images to obtain multiple first local features, and to extract features from the second image to obtain first global features; The retrieval module 940 can be used to perform image retrieval in a pre-built image library based on multiple first local features and first global features to obtain one or more first target images; wherein the image library is constructed through the following steps: obtaining multiple first candidate images; performing clustering and nesting processing on the multiple first candidate images to obtain multiple representative images; and constructing the image library based on the multiple representative images.

[0149] In one embodiment, during the process of retrieving one or more first target images from a pre-built image library based on multiple first local features and first global features, the retrieval module 940 may further be used to: Image retrieval is performed in the image database based on multiple first local features and first global features to obtain multiple second candidate images; The relevance of multiple second candidate images is ranked, and one or more of the highest-ranked second candidate images are determined as the first target image.

[0150] In one embodiment, during the process of retrieving images from an image database based on multiple first local features and first global features to obtain multiple second candidate images, the retrieval module 940 can also be used to: Obtain the search constraint information input by the user; Based on the retrieval constraint information, multiple first local features, and first global features, image retrieval is performed in the image database to obtain multiple second candidate images.

[0151] In one embodiment, during the process of ranking multiple second candidate images by relevance and determining one or more of the highest-ranked second candidate images as the first target image, the retrieval module 940 can also be used to: Based on the globally unique identifier of each second candidate image, duplicate second candidate images in the retrieved second candidate images are removed; The remaining second candidate images are ranked by relevance, and one or more of the highest-ranked second candidate images are determined as the first target image.

[0152] In one embodiment, during the process of ranking the remaining second candidate images by relevance and determining one or more of the highest-ranked second candidate images as the first target image, the retrieval module 940 can also be used to: The remaining second candidate images are grouped to obtain multiple candidate image groups, wherein the second candidate images in the candidate image groups belong to the same whole slice image; Multiple target candidate images are determined from each candidate image group, wherein the multiple target candidate images are the multiple second candidate images with the highest similarity ranking in each candidate image group; Multiple target candidate images are ranked by relevance, and one or more target candidate images with the highest relevance are determined as the first target image.

[0153] In one embodiment, the image retrieval device 900 further includes: The filling module can be used to fill the first image according to the preset size threshold in response to the first image being smaller than the size threshold, so as to obtain the fourth image; The feature extraction module 930 can also be used to extract features from the fourth image to obtain the third global feature; The retrieval module 940 can also be used to perform image retrieval in the image library based on the third global feature to obtain one or more second target images.

[0154] In one embodiment, the feature extraction module 930 can also be used to extract features from the first image in response to the size of the first image being equal to a preset size threshold, to obtain a fourth global feature; The retrieval module 940 can also be used to perform image retrieval in the image library based on the fourth global feature to obtain one or more third target images.

[0155] The image retrieval device provided in the above embodiments is based on the same principle as the image retrieval method provided in the above embodiments, and will not be described again here.

[0156] In addition to the embodiments described above, one embodiment of this application also provides an electronic device. See also Figure 12 , Figure 12 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Figure 12 As shown, the electronic device includes a memory 1100 and a processor 1200. The number of memories 1100 and processors 1200 can be one or more. Figure 12 Taking a memory 1100 and a processor 1200 as an example; Figure 12The memory 1100 and processor 1200 can be connected via a bus or other means. Figure 12 Taking the example of a connection between China and Israel via a bus.

[0157] The memory 1100, as a computer-readable storage medium, can be used to store one or more software programs, computer-executable programs, and modules, such as the program, instructions, or modules corresponding to the image retrieval method provided in any embodiment of this application. The processor 1200 implements the image retrieval method provided in any embodiment of this application by executing one or more computer programs, instructions, and modules stored in the memory 1100.

[0158] The memory 1100 may primarily include a program storage area and a data storage area, wherein the program storage area may store the operating system and computer programs required for at least one function. Furthermore, the memory 1100 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 1100 may further include memory remotely located relative to the processor 1200, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0159] In addition to the embodiments described above, one embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions for performing the image retrieval method as described in any of the preceding embodiments.

[0160] Furthermore, one embodiment of this application also provides a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the electronic device to perform the image retrieval method as described in any of the preceding embodiments.

[0161] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0162] The above is a detailed description of the preferred embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An image retrieval method, characterized in that, The method includes: In response to the fact that the size of the obtained first image is greater than a preset size threshold, the first image is scaled according to the preset size threshold to obtain a second image; Multiple third images are extracted from the first image through a sliding window, wherein each of the third images overlaps with at least one other third image; Feature extraction is performed on multiple third images to obtain multiple first local features, and feature extraction is performed on the second image to obtain first global features; Image retrieval is performed in a pre-built image library based on multiple first local features and first global features to obtain one or more first target images; wherein, the image library is constructed through the following steps: obtaining multiple first candidate images; performing clustering nesting processing on the multiple first candidate images to obtain multiple representative images; and constructing the image library based on the multiple representative images.

2. The method according to claim 1, characterized in that, The step of retrieving images from a pre-built image library based on multiple first local features and first global features to obtain one or more first target images includes: Based on multiple first local features and first global features, image retrieval is performed in the image library to obtain multiple second candidate images; The relevance of the multiple second candidate images is ranked, and one or more of the second candidate images with the highest ranking are determined as the first target image.

3. The method according to claim 2, characterized in that, The step involves retrieving images from the image database based on multiple first local features and first global features to obtain multiple second candidate images, including: Obtain the search constraint information input by the user; Based on the retrieval constraint information, multiple first local features, and first global features, image retrieval is performed in the image database to obtain multiple second candidate images.

4. The method according to claim 2 or 3, characterized in that, The step of ranking the relevance of multiple second candidate images and determining one or more of the highest-ranked second candidate images as the first target image includes: Based on the globally unique identifier of each second candidate image, duplicate second candidate images in the retrieved second candidate images are removed; The remaining second candidate images are sorted by relevance, and one or more of the second candidate images with the highest relevance are determined as the first target image.

5. The method according to claim 4, characterized in that, The step of ranking the remaining second candidate images by relevance and determining one or more of the highest-ranked second candidate images as the first target image includes: The remaining second candidate images are grouped to obtain multiple candidate image groups, wherein the second candidate images in the candidate image groups belong to the same full slice image; Multiple target candidate images are determined from each of the candidate image groups, wherein the multiple target candidate images are multiple second candidate images with the highest similarity ranking in each of the candidate image groups; The multiple target candidate images are ranked by relevance, and one or more of the target candidate images with the highest ranking are determined as the first target image.

6. The method according to claim 1, characterized in that, The clustering and nesting process of multiple first candidate images yields multiple representative images, including: Feature extraction is performed on each of the first candidate images to obtain the second global features of each of the first candidate images; Clustering is performed on each of the second global features to obtain multiple clusters; In response to the generation of clusters, for each newly generated cluster, clustering is performed based on each of the second global features that are clustered into the newly generated cluster, and a new cluster is generated in the newly generated cluster; In response to the completion of the last layer of clustering, for each cluster in the last layer, the second global feature that is closest to the spatial cluster center of the last layer is determined as the target global feature. The first candidate image to which each of the target global features belongs is determined as the representative image.

7. The method according to claim 1, characterized in that, The method further includes: In response to the fact that the size of the first image is smaller than the preset size threshold, the first image is filled according to the preset size threshold to obtain the fourth image; Feature extraction is performed on the fourth image to obtain the third global feature; Based on the third global feature, image retrieval is performed in the image library to obtain one or more second target images.

8. The method according to claim 1, characterized in that, The method further includes: In response to the first image's size being equal to the preset size threshold, feature extraction is performed on the first image to obtain a fourth global feature; Based on the fourth global feature, image retrieval is performed in the image library to obtain one or more third target images.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; The image retrieval method according to any one of claims 1 to 8 is implemented when at least one of the programs is executed by at least one of the processors.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are used to execute the image retrieval method according to any one of claims 1 to 8.

Images