A customized microscopy case classification retrieval system based on multimodal learning

Abstract

The present invention provides a customized microscopic case hierarchical retrieval system based on multimodal learning, which uses the CLIP model to process the retrieval requirements input in the form of images, and after fusing with the retrieval requirements input in the form of text, extracts its feature representation information, and then compares it with the feature representation of the image in the historical data after CLIP encoding to obtain the historical image closest to the fusion retrieval requirements; the present invention improves the existing feature matching method in view of the fact that medical images have many fine-grained differences, introduces a hierarchical retrieval design for the feature representation of historical data, and improves the accuracy of customized retrieval. The present invention combines the patient's microscopic images and the specific needs of the physician's diagnosis, retrieves and presents historical related case images and diagnostic information, assists the physician in making judgments, and improves the accuracy of diagnosis; the present invention will be particularly helpful for the diagnosis of controversial cases, rare cases, etc. in microscopy, and provide a reliable basis and reference for physicians' diagnosis and treatment.

Description

Technical Field

[0001] The present invention relates to the technical field of information and communication technology specifically used for handling or processing medical or health data, and in particular to a customized microscopy case hierarchical retrieval system based on multimodal learning. Background Art

[0002] In terms of grading microscopic cases, it is mainly solved by referring to image examples in relevant books and communicating with other physicians in the department. The images in relevant books are limited and generally typical, and it is difficult to cover the situations that may occur in real cases; internal communication efficiency is low, high-quality diagnosis depends on the physician's qualifications, and there is often a lack of actual images as a reference, and the information that physicians can obtain is limited.

[0003] In recent years, with the development of informatization in hospitals and other medical institutions, the number of medical institutions that digitize and manage patient medical records has gradually increased. In the actual use of electronic medical records, it is sometimes necessary to retrieve cases with similar examination images from the stored past history data as a reference for the diagnosis of the current case. However, how to retrieve from a large number of electronic medical records has become a difficult problem.

[0004] Therefore, a hierarchical retrieval system for microscopic examination cases is needed. Summary of the invention

[0005] The present invention aims to solve the problem of microscopic case retrieval and provide a customized microscopic case hierarchical retrieval system based on multimodal learning. It combines the patient's microscopic images and the specific needs of the physician in diagnosis to retrieve and present historical related case images and diagnostic information to assist the physician in making judgments. The system will be particularly helpful for the diagnosis of controversial cases, rare cases, etc. in microscopy, and provide a reliable basis and reference for physicians' diagnosis and treatment.

[0006] The present invention provides a customized microscopy case hierarchical retrieval system based on multimodal learning, comprising an image acquisition module, a detection and diagnosis module, a customized retrieval module and a database module that can interact with the detection and diagnosis module and the customized retrieval module, which are connected in sequence. The customized retrieval module can also interact with the detection and diagnosis module.

[0007] The image acquisition module performs imaging and image enhancement processing on the microscopic case and uniquely numbers the image in combination with the microscopic case information to obtain the microscopic case image and output it to the detection and diagnosis module;

[0008] The detection and diagnosis module receives data input from the image acquisition module, the external device and the customized retrieval module, and after obtaining the diagnosis result, packages the diagnosis result and the microscopic case image and transmits it to the database module for storage; the detection and diagnosis module also includes a system use interface, through which the user can observe the microscopic case image acquired by the image acquisition module and input the diagnosis result; the user can also input the retrieval requirements for historical microscopic case images and diagnosis results in the customized retrieval module through the system use interface in combination with the current microscopic case image, retrieve historical related cases and give a diagnosis based on the retrieval results to generate the diagnosis result;

[0009] The customized retrieval module analyzes the microscopic case images and text retrieval requirements passed in from the detection and diagnosis module, finds historical microscopic case images that meet the user's requirements from the historical data of the database module, and returns the historical microscopic case images and corresponding diagnosis results to the user;

[0010] The customized retrieval module uses a pre-trained model based on image-text matching to process retrieval requirements input in the form of images, fuses the retrieval requirements input in the form of images with the retrieval requirements input in the form of text to obtain fused retrieval requirements, extracts feature representation information of the fused retrieval requirements from the fused retrieval requirements, and then compares it with the feature representation of the historical microscopy case images encoded in the database module to obtain the historical microscopy case images that are closest to the fused retrieval requirements and outputs them to the system user interface;

[0011] The database module saves the input microscopic case images and diagnosis results. After receiving new data from the detection and diagnosis module, the database module saves and accepts the retrieval and call of data by the customized retrieval module.

[0012] The customized microscopic case graded retrieval system based on multimodal learning described in the present invention is, as a preferred embodiment, that the microscopic case image is a smear image under a digital microscope, and the image enhancement processing includes denoising and sharpening.

[0013] The customized microscopic case graded retrieval system based on multimodal learning described in the present invention, as a preferred embodiment, the retrieval method of the customized microscopic case graded retrieval system comprises the following steps:

[0014] S1, the image acquisition module scans the microscopic smear to be observed, performs image enhancement processing, and uniquely numbers the image in combination with the microscopic case information to obtain the microscopic case image and uploads it to the doctor for preview through the detection and diagnosis module;

[0015] S2, the doctor makes a diagnosis and determines whether it is necessary to search for similar cases. If yes, proceed to step S3, if not, proceed to step 4;

[0016] S3. The doctor customizes the search requirements in the system user interface of the detection and diagnosis module, customizes the image and text parsing requirements of the search module, searches the historical microscopic case images in the database module and returns the historical microscopic case images and diagnosis results that meet the requirements;

[0017] S4. The doctor issues a diagnosis result, and the search of the customized microscopic case grading retrieval system is completed.

[0018] The customized microscopic case graded retrieval system based on multimodal learning described in the present invention, as a preferred embodiment, step S3 includes the following steps:

[0019] S31. The user inputs text retrieval requirements and microscopic image retrieval requirements in the customized retrieval module through the detection and diagnosis module. The customized retrieval module fuses the text retrieval requirements and the microscopic image retrieval requirements through the template embedding method to obtain a fused retrieval requirement. The fused retrieval requirement is a unified representation of the retrieval requirement with text as the carrier; the customized retrieval module extracts the image features of historical data of different scales of the image to be retrieved in the database module;

[0020] S32. The customized retrieval module calculates the similarity between the historical data image features of different scales in the database module and gradually selects the images with the highest similarity to the semantic information features of the retrieval requirements after fusion according to the retrieval requirements after fusion, obtains the historical images closest to the retrieval requirements after fusion and outputs them to the system user interface.

[0021] The customized microscopic case graded retrieval system based on multimodal learning described in the present invention, as a preferred embodiment, step S31 includes the following steps:

[0022] S311, the user enters text retrieval requirements and microscopic image retrieval requirements in the customized retrieval module through the detection and diagnosis module, and the image encoder Retrieve requirements from microscopic images Extract features from ;

[0023] S312, image-text feature conversion unit Predictive features Vector representation in text representation space , by segmenting and linearly mapping the text retrieval requirements, we can obtain the vector representation in the text representation space ;

[0024] S313, vector representation With vector representation Fusion by template to get text , text encoder From the text Extract features from the fusion to obtain the retrieval requirement feature representation ;

[0025] S314, Image Encoder Extract the historical microscopic case images to be retrieved in the database module The last 3 layers output and , and obtain different scale feature representations of the historical microscopic case images to be retrieved.

[0026] The customized microscopic case classification retrieval system based on multimodal learning described in the present invention is preferably an image encoder. , Image-text feature conversion unit and text encoder All are based on CLIP components;

[0027] The template is pre-set by the user; when there is more than one search template, a template is randomly selected.

[0028] The customized microscopic case classification retrieval system based on multimodal learning described in the present invention is preferably an image encoder. , Image-text feature conversion unit , Text Encoder All of them are models with ResNet50 structure.

[0029] The customized microscopic case graded retrieval system based on multimodal learning described in the present invention, as a preferred embodiment, step S32 includes the following steps:

[0030] S321, Retrieval requirement feature representation after calculation fusion And the historical microscopic case images to be retrieved The last layer of features The similarity of is the total number of all historical microscopic case images to be retrieved; according to the predefined first similarity threshold The number of screened The first screening of historical data;

[0031] S322. Calculate the penultimate layer of features of the first screening historical data Representation of retrieval requirements after fusion similarity, according to a predefined second similarity threshold The number of screened The second screening of historical data;

[0032] S323. Calculate the third-to-last layer features of the second screening of historical data Representation of retrieval requirements after fusion similarity, according to a predefined third similarity threshold The images with the highest similarity to the semantic information features required for fusion retrieval are screened and output to the system user interface.

[0033] The customized microscopy case graded retrieval system based on multimodal learning described in the present invention is, as a preferred method, in steps S321, S322, and S323, when no historical data with a similarity exceeding a preset threshold is obtained, the information "no similar images in the historical data" is returned and displayed in the system user interface, prompting the user to modify the retrieval requirements or modify the retrieval similarity threshold setting.

[0034] The customized microscopic case classification retrieval system based on multimodal learning described in the present invention is preferably implemented in step S321 by using cosine similarity. Calculate the similarity, where and They are the normalized texts And the historical microscopic case image data to be retrieved Feature representation output by the CLIP encoder;

[0035] In steps S322 and S323, In the spatial dimension, each position The vectors at the positions are respectively obtained through the linear mapping layer to obtain the dimension reduction mapping vectors of the specific regions of the corresponding image in the output space of the image encoder. , through the spatial dimension Take the average of the above , and then use cosine similarity

[0036] , Perform similarity calculation;

[0037] In step S323, the 1 or 0 image indexes with the highest similarity in the historical data are returned, and the historical images and other information of the case images in the index positioning database are returned to the detection and diagnosis module;

[0038] , and The values ​​are all between 0.9 and 1.

[0039] The present invention has the following advantages:

[0040] (1) The present invention uses a CLIP model pre-trained on a large amount of text-image pair data to process retrieval requirements input in the form of images, and after fusing them with the retrieval requirements input in the form of text, extracts their feature representation information, and then compares it with the feature representation of the image in the historical data after CLIP encoding, to obtain the historical image that is closest to the fused retrieval requirement; in view of the fact that medical images have many fine-grained differences, the present invention improves the existing feature matching method, introduces a hierarchical retrieval design for the feature representation of historical data, and improves the accuracy of customized retrieval.

[0041] (2) The present invention combines the patient's microscopic images with the specific needs of the physician in diagnosis, retrieves and presents historical case images and diagnostic information, assists the physician in making judgments, and improves the accuracy of diagnosis; the present invention will be particularly helpful in the diagnosis of controversial cases and rare cases in microscopic examinations, and provide a reliable basis and reference for physicians' diagnosis and treatment; the present invention can also be used in scientific research and teaching, by customizing questions to solve problems in teaching in a targeted manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the structure of a customized microscopy case classification retrieval system based on multimodal learning;

[0043] Figure 2 This is a flowchart of the retrieval method of a customized microscopic case graded retrieval system based on multimodal learning;

[0044] Figure 3 This is a flowchart of step S3 of a customized microscopy case grading retrieval system based on multimodal learning. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Example 1

[0046] like Figures 1 to 3 As shown in the figure, a customized microscopic case classification retrieval system based on multimodal learning is constructed as follows: Figure 1 As shown, it includes an image acquisition module, a detection and diagnosis module, a customized retrieval module and a database module. The data interaction relationship between the modules is indicated by straight arrows. Figure 2 The complete workflow of this system in one use is shown, and the following is a detailed introduction to the relevant modules.

[0047] 1. Image acquisition module

[0048] The image acquisition module is used to scan, photograph and digitize the smear images under the microscope. In order to obtain high-quality images for subsequent use by doctors and retrieval modules, the module should also perform basic image enhancement processing (denoising, sharpening, etc.) on the acquired images. Finally, combined with the smear case information, the image is uniquely numbered and passed to the detection module.

[0049] 2. Detection and diagnosis module

[0050] The detection and diagnosis module is mainly composed of a software application that includes a user interface (doctor or other user). In this application, users can observe the collected case images and input the diagnosis results; they can also combine the current images, input the retrieval requirements for historical cases and diagnosis results, retrieve historical related cases, and give a diagnosis after synthesis.

[0051] On the whole, this module receives data from the image acquisition module, external devices and customized retrieval module, and after obtaining the diagnosis results, it packages the diagnosis results and images and transmits them to the database module for storage.

[0052] 3. Customized search module

[0053] The customized retrieval module is the core of this system. It analyzes the images and targeted retrieval requirements (text) passed in from the detection and diagnosis module, finds images that meet the user's requirements from the historical data, and returns the images and corresponding diagnosis results to the user. In this system, this module is implemented based on the correlation calculation between text-image representations; specifically, we use the CLIP model pre-trained on a large amount of text-image pair data to process the retrieval requirements input in the form of images, and after fusing them with the retrieval requirements input in the form of text, extract their feature representation information, and compare them with the feature representation of the images in the historical data after CLIP encoding to obtain the historical images that are closest to the fused retrieval requirements; in particular, based on this process, we have improved the existing feature matching method in view of the fact that medical images have many fine-grained differences, and introduced a hierarchical retrieval design for the feature representation of historical data to improve the accuracy of customized retrieval.

[0054] The specific process of this module is as follows Figure 3 , the details of each part are further introduced below; it should be noted that, although the CLIP model is mainly used as an example here, the present invention also supports the implementation of a pre-trained model with a similar structure to the CLIP model and obtained based on image-text matching.

[0055] 3.1 Multimodal Feature Extraction

[0056] like Figure 3 ,The present invention mainly extracts features from user input (text, image) and historical data (image) based on the ,image and text encoder of the CLIP model.

[0057] First, on the user input side, for retrieval requirements that appear in two modes (original retrieval text, microscopic image), the present invention uses a "template embedding" approach to form a unified representation of the retrieval requirements with text as the carrier, thereby realizing the fusion of retrieval requirements in the two modes.

[0058] Specifically, Figure 3 , using the CLIP image encoder , retrieve the image from Extract features , through the "Image-Text Feature Conversion Unit" , predict a vector representation of the image in the CLIP text representation space At the same time, the input search text is also represented by its vector in the CLIP text representation space through word segmentation and linear mapping (embedding). Since there is a correspondence between the word segmentation tokens used by CLIP and the vector representation of the text, it contains the representation of the image semantic information. Can be compared with the vector representation of the original text Rearrange and splice according to a certain template to get a new set of text This process is called “template embedding”. The template is pre-set by the user, such as (expressed in natural language) “an image and (Image embed Related and (Text Embedding ） "; the image embedding comes from the pseudo text obtained by the image encoder and text conversion unit, and the text embedding comes from the user input. When the user sets multiple templates, a template is randomly selected for embedding. Finally, the new text , input CLIP text encoder , from the new text that combines the image and the original retrieved text information In the above example, we extract its feature representation .

[0059] On the other side, for the image to be retrieved in the database , we directly use the CLIP image encoder based on convolutional networks , for the input image , extract it in the encoder The last 3 layers of output and , a set of feature representations of the image to be retrieved at different scales is obtained, which is directly input into the subsequent hierarchical similarity calculation unit (Section 3.2). In particular, in order to speed up the calculation in the retrieval, if the database capacity allows, the features of the image after CLIP encoding are and It should be saved together with the image and diagnosis, and directly extracted during retrieval for subsequent hierarchical similarity calculation.

[0060] Image Encoder , Text Encoder , Image-text feature conversion unit, CLIP text representation space, CLIP image encoder based on convolutional network All are models based on the ResNet50 structure; the subscript im indicates that the encoder is used for image input; (i, j) is the feature vector with coordinates (i, j) in the spatial dimension on the feature map extracted by the encoder;

[0061] The output is generally a four-dimensional tensor (Batch size, C, H, W), where batch size is the data batch, C represents the number of feature channels, H and W represent the spatial dimensions, corresponding to the height and width dimensions of the original image, and (i, j) represents the feature vector at the position indexed by (i, j) on the H and W dimensions of the feature map; Dimensionality reduction refers to compressing the number of feature channels to the same size as the last layer output. The same can be achieved through linear mapping.

[0062] 3.2 Hierarchical Similarity Calculation

[0063] The fused retrieval input features are used to calculate the similarity between the image features of different scales in the historical data, and the images with the closest semantic information, i.e., the highest feature similarity, are selected step by step. Figure 3 , the retrieved input features are first compared with the last layer of features extracted from the historical data ( Calculate the similarity for all historical data to be retrieved) according to the pre-defined similarity threshold , filter out a batch of historical data that is close to the retrieval input features in terms of deep semantic information (assuming that the number is ). Then, the penultimate layer features of these images are calculated Similarity with the retrieval input feature, based on a predefined similarity threshold , select a batch of historical data with the closest similarity to the retrieval input features at this scale (assuming that the number is ). Finally, in this batch of images, by calculating the third-to-last layer features of these images The similarity between the input feature to be retrieved is to filter out the features that are closest to the retrieval input feature and whose similarity exceeds the preset threshold In the similarity calculation of any of the above levels, if the similarity calculated by no image exceeds the preset threshold, the message "No similar images in historical data" is returned, prompting the user to modify the search requirements or modify the search similarity threshold setting.

[0064] In this module, The same as the retrieval feature dimension, the similarity calculation can directly apply cosine similarity Implementation, where and They are respectively the normalized composite texts and historical data Feature representation output by the CLIP encoder.

[0065] In calculation When retrieving the similarity between input features, In the spatial dimension, each position (here The vector at (denoted by) does not pass through the subsequent multi-head attention layer of the CLIP image encoder, but directly passes through the linear mapping layer to obtain the dimensionality reduction mapping vector of the specific area of ​​the corresponding image in the output space of the image encoder. , and then through the spatial dimension Taking the average, we get , press and In the same way, the cosine similarity between the input features and the retrieval is calculated.

[0066] According to the above hierarchical retrieval method, one or zero image indexes with the highest similarity to the input to be retrieved in the historical data at each scale can be returned in the end, and the historical image and other information of the case image in the index positioning database can be returned to the detection and diagnosis module.

[0067] in, , , There is no fixed size relationship, representing the similarity threshold level at different scales, which is determined according to user needs during actual use;

[0068] The similarity threshold is in the range of 0-1, and a value above 0.9 is generally taken as the threshold;

[0069] As the model deepens, the extracted features can better reflect the deep semantic information of the image, but at the same time, as the dimension is compressed, the high-frequency details of the image are also lost; therefore, similarity matching is performed from the last layer, mainly to filter the retrieved images from coarse to fine, that is, from semantic similarity to image detail similarity, and gradually refine the retrieval results;

[0070] The image with the largest similarity calculation value is the closest. The search result is the historical image index with the highest similarity obtained in "Similarity Calculation (III)" after threshold screening. The image file can be obtained based on this index.

[0071] In summary, Figure 3 In the algorithm shown, the feature extraction of the above multimodal data is mainly implemented based on the CLIP model launched by OpenAI, and the fusion of different modal data is mainly achieved through "image-text conversion unit" and "template embedding". Finally, in terms of similarity calculation, it is achieved through the cosine similarity commonly used in image retrieval.

[0072] 4. Database Module

[0073] The database module mainly stores the incoming images and diagnostic data, saves the new data after receiving it from the detection and diagnosis module, and accepts the retrieval and call of data by the customized retrieval module. As mentioned above, when the database capacity allows, the image features of the incoming images encoded by CLIP can be saved together to speed up the image retrieval process.

[0074] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A customized microscopic case classification retrieval system based on multimodal learning, characterized by: It includes an image acquisition module, a detection and diagnosis module, a customized retrieval module and a database module that can interact with the detection and diagnosis module and the customized retrieval module in sequence, and the customized retrieval module can also interact with the detection and diagnosis module; The image acquisition module performs imaging and image enhancement processing on the microscopic case and uniquely numbers the image in combination with the microscopic case information to obtain a microscopic case image and outputs it to the detection and diagnosis module; The detection and diagnosis module receives data input from the image acquisition module, the external device and the customized retrieval module, and after obtaining the diagnosis result, packages the diagnosis result and the microscopic case image and transmits it to the database module for storage; the detection and diagnosis module also includes a system use interface, through which the user can observe the microscopic case image acquired by the image acquisition module and input the diagnosis result; the user can also input the retrieval requirements for historical microscopic case images and diagnosis results in the customized retrieval module through the system use interface in combination with the current microscopic case image, retrieve historical related cases and give a diagnosis based on the retrieval results to generate a diagnosis result; The customized retrieval module analyzes the microscopic case images and text retrieval requirements transmitted from the detection and diagnosis module, finds historical microscopic case images that meet the user's requirements from the historical data of the database module, and returns the historical microscopic case images and corresponding diagnosis results to the user; The customized retrieval module uses a pre-trained model obtained based on image-text matching to process the retrieval requirements input in the form of images, fuses the retrieval requirements input in the form of images with the retrieval requirements input in the form of text to obtain a fused retrieval requirement, extracts the fused retrieval requirement feature representation information from the fused retrieval requirement, and then performs hierarchical similarity calculation with the feature representation of the historical microscopic case images of different scales encoded in the database module, and screens the images with the highest feature similarity level by level, obtains the historical microscopic case images closest to the fused retrieval requirement, and outputs them to the system user interface; The database module stores the input microscopic case images and diagnosis results. After receiving new data from the detection and diagnosis module, the database module stores and accepts the retrieval and call of data by the customized retrieval module. The retrieval method of the customized microscopic case graded retrieval system includes the following steps: S1, the image acquisition module performs smear scanning and image enhancement processing on the microscopic smear to be observed and uniquely numbers the image in combination with the microscopic case information to obtain the microscopic case image and upload it to the doctor for preview through the detection and diagnosis module; S2, the doctor makes a diagnosis and determines whether it is necessary to search for similar cases. If yes, proceed to step S3, if not, proceed to step 4; S3, the doctor customizes the search requirements in the system user interface of the detection and diagnosis module, and the customized search module parses the image and text requirements, searches for historical microscopic case images in the database module and returns the historical microscopic case images and diagnosis results that meet the requirements; Step S3 includes the following steps: S31, the user inputs text retrieval requirements and microscopic image retrieval requirements in the customized retrieval module through the detection and diagnosis module, and the customized retrieval module fuses the text retrieval requirements and the microscopic image retrieval requirements through a template embedding method to obtain the fused retrieval requirements, and the fused retrieval requirements are unified representations of the retrieval requirements with text as the carrier; the customized retrieval module extracts historical data image features of different scales of the images to be retrieved in the database module; Step S31 includes the following steps: S311: The user inputs text search requirements and microscopic image search requirements in the customized search module through the detection and diagnosis module, and the image encoder Retrieve requirements from microscopic images Extract features from ; S312, image-text feature conversion unit Predictive features Vector representation in text representation space , by segmenting and linearly mapping the text retrieval requirements, we can obtain the vector representation in the text representation space ; S313, vector representation With vector representation Fusion by template to get text , text encoder From the text Extract features from the fusion to obtain the retrieval requirement feature representation ; S314, Image Encoder Extract the historical microscopic case images to be retrieved from the database module The last 3 layers output and , obtain different scale feature representations of the historical microscopic case images to be retrieved; S32, the customized retrieval module performs similarity calculation with the historical data image features of different scales in the database module according to the fusion retrieval requirements, and gradually selects the images with the highest similarity with the semantic information features of the fusion retrieval requirements, obtains the historical images closest to the fusion retrieval requirements, and outputs them to the system user interface; Step S32 includes the following steps: S321, calculating the fusion retrieval requirement feature representation The historical microscopic case images to be retrieved The last layer of features The similarity of is the total number of all historical microscopic case images to be retrieved; according to the predefined first similarity threshold The number of screened The first screening of historical data; Using cosine similarity Calculate the similarity, where and They are the normalized texts And the historical microscopic case image data to be retrieved Feature representation output by the CLIP encoder; S322, calculating the second-to-last layer feature of the first screening historical data After the fusion, the retrieval requirement feature is expressed similarity, according to a predefined second similarity threshold The number of screened The second screening of historical data; S323, calculating the third-to-last layer feature of the second screening historical data After the fusion, the retrieval requirement feature is expressed similarity, according to a predefined third similarity threshold Screening out images with the highest similarity to the semantic information features of the fusion retrieval requirements and outputting them to the system user interface; returning 1 or 0 image indexes with the highest similarity in the historical data, and returning the historical images together with other information of the case images in the index positioning database to the detection and diagnosis module; In steps S322 and S323, In the spatial dimension, each position The vectors at the positions are respectively obtained through the linear mapping layer to obtain the dimension reduction mapping vectors of the specific regions of the corresponding image in the output space of the image encoder. ,, through the spatial dimension Take the average of the above , and then use cosine similarity , Perform similarity calculation; In steps S321, S322, and S323, when no historical data with a similarity exceeding a preset threshold is obtained, a message "no similar images in historical data" is returned and displayed in the system user interface, prompting the user to modify the search requirements or modify the search similarity threshold setting; S4. The doctor issues a diagnosis result, and the search of the customized microscopic case grading retrieval system is completed.

2. According to claim 1, a customized microscopic case hierarchical retrieval system based on multimodal learning is characterized by: The microscopic case image is a smear image under a digital microscope, and the image enhancement processing includes denoising and sharpening.

3. According to claim 1, a customized microscopic case hierarchical retrieval system based on multimodal learning is characterized by: The image encoder , the image-text feature conversion unit and the text encoder All are based on CLIP components; The template is pre-set by the user; when there is more than one search template, a template is randomly selected.

4. According to claim 3, a customized microscopic case hierarchical retrieval system based on multimodal learning is characterized by: The image encoder , the image-text feature conversion unit , the text encoder All of them are models with ResNet50 structure.

5. According to claim 1, a customized microscopic case hierarchical retrieval system based on multimodal learning is characterized by: , and The values ​​are all between 0.9 and 1.

Images