A microscopic image core semantic text generation method of a double clustering process

Abstract

This invention discloses a method for generating core semantic text from microscopic images using a dual-clustering process, belonging to the field of medical image processing technology. The invention obtains a general semantic set through a large language model, processes the microscopic images to obtain sets of microscopic target images for each category, and then inputs these preprocessed sets into a trained image-text model to generate text and image representation vectors. A category-semantic similarity matrix is ​​constructed using cosine similarity calculation. Different similarity sets are extracted from the matrix, and intra- and inter-class clustering is performed to obtain generalized and discriminative semantic text sets for each category. The intersection of the two sets yields the core semantic text. This invention achieves accurate semantic filtering of microscopic images through a dual-clustering strategy, and the generated core semantic text possesses both generalization and discriminative properties, improving semantic alignment quality and providing high-quality semantic data support for fine-tuning multimodal models.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a method for generating core semantic text from microscopic images using a dual clustering process. Background Technology

[0002] With the evolution of deep learning technology, the cross-integration of computer vision (CV) and natural language processing (NLP) has given rise to multi-modal foundation models. The core idea of ​​these models is to break the limitation of traditional visual models that can only recognize fixed categories (such as "cat" or "dog"), by training them with massive amounts of internet images and text. Among them, OPENAI's CLIP (Contrastive Language-Image Pre-training) is the most representative solution. It adopts a "dual-tower" structure, using an image encoder and a text encoder to map images and their descriptions to the same high-dimensional feature space. Through contrastive learning techniques, the model learns to "bring closer" related image-text pairs and "push apart" unrelated pairs, thus aligning the images with the meaning of the text.

[0003] Existing technologies, represented by CLIP (and later models such as GLIP which incorporate object detection capabilities), possess extremely strong zero-shot learning capabilities. This means that even without fine-tuning for specific scenes, the model can recognize objects it has never seen before based on text commands. This "base" property makes it perform excellently in image-text alignment and content retrieval in general scenarios (such as natural scenery, everyday items, and portraits).

[0004] Because general-purpose models are primarily trained on natural internet images, their visual feature distributions differ significantly from those of professional microscopic medical images (such as pathological sections and fluorescence micrographs). Furthermore, the textual corpus in the general domain lacks in-depth medical terminology logic. This results in the model's inability to effectively align visual features with medical semantics in microscopic medical scenarios, leading to misjudgments when dealing with complex anatomical structures, pathological features, or minute tissue lesions. Therefore, existing technologies suffer from the problem of inaccurate image-text alignment and insufficient generalization ability in medical microscopy scenarios due to visual feature mismatch and lack of professional semantics in general-purpose multimodal pedestal models. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this invention provides a method for generating core semantic text for microscopic images using a dual-clustering process. This method solves the problems of general multimodal models struggling to achieve precise alignment between visual features and medical semantics in the field of microscopic images, exhibiting weak generalization ability, and failing to meet the requirements for high-quality generalized semantic text needed for subsequent fine-tuning of multimodal models.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: a method for generating core semantic text from microscopic images using a dual-clustering process, comprising the following steps:

[0007] S1. Use a large language model to obtain a general semantic set;

[0008] S2. Cut out the target images from the microscopic images, scale them up, and then paste them onto the background images respectively, and classify them to obtain a set of microscopic target images for each category.

[0009] S3. After preprocessing the general semantic set and the image set of each category of microscopic target, input them into the trained image-text model to obtain the text representation vector and the image representation vector;

[0010] S4. Calculate the cosine similarity between the text representation vector and the image representation vector, and construct a category-semantic similarity matrix;

[0011] S5. Extract the image-text similarity set from the category-semantic similarity matrix and perform intra-class clustering to obtain the generalized semantic text set for each category;

[0012] S6. Extract the text-image similarity set from the category-semantic similarity matrix and perform inter-class clustering to obtain the discriminative semantic text set for each category;

[0013] S7. Take the intersection of the discriminative semantic text set and the generalized semantic text set of the same category to obtain the core semantic text.

[0014] Furthermore, S1 includes the following sub-steps:

[0015] S11. Construct structured text prompts;

[0016] S12. Based on the structured text prompts, a large language model is used to generate a candidate semantic text library;

[0017] S13. Extract a general semantic set containing microbial morphology, structure and biological characteristics from the candidate semantic text library.

[0018] Furthermore, S2 includes the following sub-steps:

[0019] S21. Obtain a microscopic image labeled with the target location coordinates and target category;

[0020] S22. Based on the target position coordinates, crop all targets in the microscopic image to obtain the target image;

[0021] S23. Scale each target image to 224 pixels along its longest side while keeping the aspect ratio of the target image unchanged to obtain the scaled target image;

[0022] S24. Paste the scaled-down image of the target onto the background image to obtain the microbial target-background image;

[0023] S25. According to the target category, the microbial target-background images are classified and placed into the corresponding folders to obtain a collection of microscopic target images for each category.

[0024] Furthermore, S3 includes the following sub-steps:

[0025] S31. Divide the general semantic set into the smallest semantic unit, token;

[0026] S32. Embed the smallest semantic unit, the token, into a vector to obtain a semantic vector;

[0027] S33. Input the semantic vector into the text input terminal of the trained image-text model to obtain the text representation vector;

[0028] S34. Normalize the pixel values ​​of each image in each category of microscopic target image set to obtain a preprocessed image;

[0029] S35. Input the preprocessed image into the image input terminal of the trained image-text model to obtain the image representation vector.

[0030] Furthermore, in S3, the loss function used during the training of the image-text model is: ,in, For loss function, Let be the similarity loss function between the i-th text representation vector and all image representation vectors. Let be the similarity loss function between the j-th image representation vector and all text representation vectors.

[0031] Furthermore, the similarity loss subfunction The expression is: ,

[0032] Similarity loss subfunction The expression is: ,

[0033] Where log is the logarithmic function. Let be the cosine similarity between the i-th text representation vector corresponding to the n-th candidate semantic text and the j-th image representation vector corresponding to the m-th category of microscopic target images during the training phase. The hyperparameter is used to adjust the discriminative power of the similarity distribution, and exp is an exponential function; Let be the cosine similarity between the i-th text representation vector corresponding to the n-th candidate semantic text and the i-th image representation vector corresponding to the m-th category of microscopic target image set. Let be the cosine similarity of the j-th image representation vector corresponding to the n-th candidate semantic text. The number of samples in the training batch is n, the candidate semantic text number is m, the category number is i, the text representation vector number is j, and the image representation vector number is j.

[0034] Furthermore, S4 includes the following sub-steps:

[0035] S41. Calculate the cosine similarity between each text representation vector and each image representation vector;

[0036] S42. Arrange the cosine similarity of all individual text representation vectors and individual image representation vectors in the order of "text dimension × image dimension" to obtain the category-semantic similarity matrix.

[0037] Furthermore, S5 includes the following sub-steps:

[0038] S51. Extract all cosine similarities corresponding to the m-th category from the category-semantic similarity matrix to obtain the image-text similarity set of the m-th category, where m is the category number;

[0039] S52. Based on the clustering objective function, perform clustering on the image-text similarity set of the m-th category to obtain multiple generalized semantic centers corresponding to the category.

[0040] S53. Extract the candidate semantic texts with the highest similarity from each generalized semantic center to form a set of generalized semantic texts for that category.

[0041] Furthermore, the expression for the clustering objective function in S52 is: ,

[0042] in, The clustering objective function is... For the m-th category, the set of image-text similarity is... For the m-th category, the set of image-text similarities The r-th cosine similarity in the equation, Let the k-th cluster center be the m-th category. Let k be the squared Euclidean distance, k be the cluster center number, and r be the image-text similarity set of the m-th category. The cosine similarity index is given by k, where k is the number of cluster centers.

[0043] Furthermore, S6 includes the following sub-steps:

[0044] S61. Find the cosine similarity between the nth candidate semantic text and the mth category in the category-semantic similarity matrix, and construct the text-image similarity set between the nth candidate semantic text and the mth category, where n is the number of the candidate semantic text and m is the number of the category;

[0045] S62. Calculate the mean of each cosine similarity between the nth candidate semantic text and the text-image similarity set of the mth category, and obtain the semantic center of the nth candidate semantic text for the mth category.

[0046] S63. Find the largest text semantic center of the m-th category among all candidate semantic text pairs, and obtain the comparative text semantic center of the m-th category;

[0047] S64. Subtract the semantic center of the nth candidate semantic text from the semantic center of the mth category from the semantic center of the contrasting text of the mth category to obtain the inter-class discrimination score of the candidate semantic text for the mth category.

[0048] S65. In the m-th category, select candidate semantic texts whose inter-class discrimination scores are greater than the threshold to form a set of discriminative semantic texts for the m-th category.

[0049] In summary, the beneficial effects of this invention are as follows:

[0050] 1. This invention generates a set of microbiological semantics through a large language model, which makes up for the lack of medical terminology in general multimodal models; combined with S2 dual-background image preprocessing and S3 image-text model encoding, it achieves accurate mapping between visual features of microscopic images and medical semantics, solves the core pain point of "image-text alignment failure" in existing technologies, and makes the matching degree between semantic text and microscopic target features conform to the laws of clinical laboratory medicine.

[0051] 2. This invention mines common semantics among similar microscopic targets through S5 intra-class clustering, ensuring that the core semantic text can cover target features under different imaging conditions (such as bacterial morphology under different staining and light sources), and improves the generalization and adaptation capabilities of subsequent multimodal models; then, S6 inter-class clustering filters semantics with class distinguishability to avoid semantic confusion, and finally, S7 intersection operation obtains core semantics that are both general and specific, effectively supporting the high-precision requirements of downstream tasks such as text-guided detection and semantic understanding.

[0052] 3. The core semantic text generated by this invention can be directly used for fine-tuning of multimodal models (such as GLIP, Grounding Dino, Sam series) without the need for additional manual annotation of massive amounts of professional semantic data, which greatly reduces the development cycle and annotation cost of clinical microbiology AI models. At the same time, the standardized image preprocessing process (S2) and semantic generation link improve the reusability of the technical solution and can be quickly adapted to different types of microscopic images (bacteria, fungi, pathological sections, etc.) scenarios. Attached Figure Description

[0053] Figure 1 A flowchart of a method for generating core semantic text from microscopic images using a bi-clustering process;

[0054] Figure 2 A flowchart for constructing a category-semantic similarity matrix. Detailed Implementation

[0055] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0056] This invention provides an effective semantic text synthesis method for microbial microscopic images based on a general multimodal pedestal model (image-text adversarial pre-training). It addresses the problem that current text-guided general target detection and segmentation models (such as the Sam series, GLIP, Grounding Dino, etc.) developed based on multimodal pedestal models lack generalization ability on microbial microscopic images, failing to effectively align image and text semantics and thus resulting in detection failure. This invention constructs a text semantic library specifically describing the features of microbial microscopic imaging using a large language model. Furthermore, it employs various general multimodal pedestal models (CLIP, BioMedClip, MetaClip, etc.) to perform data distillation and text semantic search on clinical microbiology image data. This identifies high-quality generalization prompts that can semantically align with the target features of microorganisms (such as color, arrangement, and shape) in the image data, meeting the fine-tuning needs of subsequent multimodal models (such as text-guided microbial microscopic target detection models and microbial microscopic image semantic understanding models).

[0057] A multimodal base model refers to a basic model that can simultaneously process two different modalities of data, namely "image" and "text", and establish semantic associations between images and text through pre-training.

[0058] like Figure 1As shown, a method for generating core semantic text from microscopic images using a bi-clustering process includes the following steps:

[0059] S1. Use a large language model to obtain a general semantic set;

[0060] S2. Cut out the target images from the microscopic images, scale them up, and then paste them onto the background images respectively, and classify them to obtain a set of microscopic target images for each category.

[0061] S3. After preprocessing the general semantic set and the image set of each category of microscopic target, input them into the trained image-text model to obtain the text representation vector and the image representation vector;

[0062] S4. Calculate the cosine similarity between the text representation vector and the image representation vector, and construct a category-semantic similarity matrix;

[0063] S5. Extract the image-text similarity set from the category-semantic similarity matrix and perform intra-class clustering to obtain the generalized semantic text set for each category;

[0064] S6. Extract the text-image similarity set from the category-semantic similarity matrix and perform inter-class clustering to obtain the discriminative semantic text set for each category;

[0065] S7. Take the intersection of the discriminative semantic text set and the generalized semantic text set of the same category to obtain the core semantic text.

[0066] In this embodiment, S1 includes the following sub-steps:

[0067] S11. Construct structured text prompts;

[0068] S12. Based on the structured text prompts, a large language model is used to generate a candidate semantic text library;

[0069] S13. Extract a general semantic set containing microbial morphology, structure and biological characteristics from the candidate semantic text library.

[0070] This invention constructs structured text prompts and uses mainstream large language models (such as GPT and Gemini) to extract general descriptions of microbial morphology, structure, and biological characteristics. These descriptions are no longer limited to simple category names but form a general semantic set containing rich biological connotations, laying a knowledge foundation for subsequent generalization alignment.

[0071] Step S1 constructs the text data used by the image-text pre-training model. Large language models have the ability to generate all human-used words, sentence structures, and semantic expressions. A rich target semantic text for optical microscopy images is generated using mainstream large language models, resulting in a candidate semantic text library. The words / sentence structures in this candidate semantic text library meet conditions 1, 2, 3, and 4: 1. All words are in English (because the best current image-text adversarial pre-training uses English text-image pairs); 2. Each word / sentence structure expresses only a single, simple semantic meaning, not a complex, compound semantic meaning; 3. Words are provided from multiple semantic perspectives, including morphology, color (staining), texture, technical terms, and common descriptions; 4. Each semantic meaning has a rich variety of different word expressions.

[0072] Structured text prompts such as: Act as an expert in microscopy and Bioimage Analysis. Please generate a comprehensive list of descriptive English keywords for objects observed under an optical microscope.

[0073] Organize the words into the following categories:

[0074] Morphology&Shape (eg, filamentous, granular, ellipsoidal)

[0075] Color&Staining Intensity (eg, eosinophilic, iridescent, punctatestaining)

[0076] Texture&Surface (eg, striated, reticulated, porous)

[0077] Professional Terminology (eg, cytoplasm, vacuole, birefringence,meniscus)

[0078] Common Analogies (eg, rod-like, needle-shaped, cluster)

[0079] Ensure the words are semantically independent and cover both biological and material science microscopy. Provide at least 20 words percategory.

[0080] The structured text prompt's Chinese definition is: As a microscopy and biological image analysis expert, please generate a comprehensive list of descriptive English keywords for objects observed under an optical microscope; categorize the keywords into "morphology and shape, color and staining intensity, texture and surface, technical terms, and common analogies" to ensure semantic independence and coverage of biological and materials science microscopy scenarios, with at least 20 words in each category.

[0081] By inputting structured English text prompts into a large language model, a general semantic set is obtained, including: "Morphology (Shape & Edge)" morphology (shape and edge): ["Spherical", "Round", "Ovoid", "Oval", "Elliptical", "Globular".

[0082] “Rod-shaped”, “Rod-like”, “Rod”, “Cylindrical”, “Elongated”, “Filamentous”.

[0083] “Curved”, “Comma-shaped”, “Spiral”, “Fusiform”, “Spindle-shaped”.

[0084] “Pleomorphic”, “Monomorphic”, “Regular shape”, “Irregular shape”, etc.

[0085] In this embodiment, S2 includes the following sub-steps:

[0086] S21. Obtain a microscopic image labeled with the target location coordinates and target category;

[0087] S22. Based on the target position coordinates, crop all targets in the microscopic image to obtain the target image;

[0088] S23. Scale each target image to 224 pixels along its longest side while keeping the aspect ratio of the target image unchanged to obtain the scaled target image;

[0089] S24. Paste the scaled-down image of the target onto the background image to obtain the microbial target-background image;

[0090] S25. According to the target category, the microbial target-background images are classified and placed into the corresponding folders to obtain a collection of microscopic target images for each category.

[0091] This invention utilizes statistical information from microscopic image data to transform full-image data into standardized individual datasets. This ensures that the feature extraction model can focus on the morphological details of individual microorganisms, reducing environmental noise interference.

[0092] In this embodiment, the objects marked in the microscopic image are indicated using either rotated bounding box coordinates (x1, x2, y1, y2, θ) or horizontal bounding box coordinates (x1, x2, y1, y2). The object target categories are: Gram-positive C-positive cocci, Gram-negative bacilli, Gram-positive Yeast fungi, Gram-positive B-positive bacilli, Gram-positive SC-positive streptococci, and Gram-positive staphylococci. All targets in the image data are cropped slightly larger than the bounding box to ensure target integrity. Each cropped image portion is scaled to a length of 224 pixels by its longest side and pasted onto a 224×224 pixel background image with a white or black background. These are then categorized and placed in folders corresponding to their respective target categories. Here, x1 is the first horizontal coordinate of the bounding box, y1 is the first vertical coordinate, x2 is the second horizontal coordinate, y2 is the second vertical coordinate, and θ is the rotation angle.

[0093] In this embodiment, S3 includes the following sub-steps:

[0094] S31. Divide the general semantic set into the smallest semantic unit, token;

[0095] S32. Embed the smallest semantic unit, the token, into a vector to obtain a semantic vector;

[0096] S33. Input the semantic vector into the text input terminal of the trained image-text model to obtain the text representation vector;

[0097] S34. Normalize the pixel values ​​of each image in each category of microscopic target image set to obtain a preprocessed image;

[0098] S35. Input the preprocessed image into the image input terminal of the trained image-text model to obtain the image representation vector.

[0099] In this embodiment, a contrastive learning architecture is adopted to project image tensors and text vectors onto a unified feature space. Using a trained image-text model (such as CLIP), the cosine similarity between image features and text semantics is calculated to establish a category-semantic similarity matrix, thereby obtaining similarity information between each type of microscopic object and the general semantic set.

[0100] Image-text models include text encoders and image encoders; text encoders use one-way or two-way Transformer models; image encoders include CNN models and VIT models.

[0101] The core structure of the image-text model is a dual-tower encoding structure combined with contrastive learning. Text and image data undergo their own preprocessing methods (text data is segmented into tokens and then embedded into vectors; image data undergoes pixel normalization) and encoder models (images use CNN-based or VIT-based models, while text uses Transformer-based one-way or two-way models) to obtain latent representation vectors for the text and image data. Contrastive learning ensures that matching image and text representation vectors are as close as possible in the latent space, while mismatched image and text representation vectors are as far apart as possible.

[0102] In this embodiment, the loss function used in training the image-text model in S3 is: ,in, For loss function, Let be the similarity loss function between the i-th text representation vector and all image representation vectors. Let be the similarity loss function between the j-th image representation vector and all text representation vectors.

[0103] In this embodiment, the similarity loss subfunction The expression is: ,

[0104] Similarity loss subfunction The expression is: ,

[0105] Where log is the logarithmic function. Let be the cosine similarity between the i-th text representation vector corresponding to the n-th candidate semantic text and the j-th image representation vector corresponding to the m-th category of microscopic target images during the training phase. The hyperparameter is used to adjust the discriminative power of the similarity distribution, and exp is an exponential function; Let be the cosine similarity between the i-th text representation vector corresponding to the n-th candidate semantic text and the i-th image representation vector corresponding to the m-th category of microscopic target image set. Let be the cosine similarity between the j-th text representation vector corresponding to the n-th candidate semantic text and the j-th image representation vector corresponding to the m-th category of microscopic target images. The number of samples in the training batch is n, the candidate semantic text number is m, the category number is i, the text representation vector number is j, and the image representation vector number is j.

[0106] In this embodiment, S4 includes the following sub-steps:

[0107] S41. Calculate the cosine similarity between each text representation vector and each image representation vector;

[0108] S42. Arrange the cosine similarity scores of all individual text representation vectors and individual image representation vectors in the order of "text dimension × image dimension" to obtain the category-semantic similarity matrix, such as... Figure 2 As shown.

[0109] In this embodiment, the formula for calculating cosine similarity in S41 is: ,in, Let be the cosine similarity between the i-th text representation vector corresponding to the n-th candidate semantic text and the j-th image representation vector corresponding to the m-th category of microscopic target images. Let be the i-th text representation vector corresponding to the n-th candidate semantic text. Let be the j-th image representation vector corresponding to the m-th category of microscopic target images, where n is the candidate semantic text number, m is the category number, i is the text representation vector number, and j is the image representation vector number.

[0110] In this embodiment, the expression for the category-semantic similarity matrix in S42 is: ,in, For category-semantic similarity matrix, Let be the set of all text representation vectors corresponding to the nth candidate semantic text. Let m be the set of microscopic target images of the m-th category.

[0111] To obtain semantics with strong generalization ability, this invention introduces clustering and filtering mechanisms. Intra-class clustering: Cluster analysis is performed on the category-semantic similarity matrix to identify common features of different microbial individuals in the semantic space. Inter-class clustering: Weakly correlated or redundant semantic terms are removed, retaining the most representative and discriminative descriptive words. Final result: An aligned set of generalized semantics is formed, enabling the model to perform effective reasoning and recognition even when faced with unseen microbial species.

[0112] S5, S6, and S7 are for obtaining the final set of generalized semantic text that is aligned with the microscopic image data.

[0113] In this embodiment, S5 includes the following sub-steps:

[0114] S51. Extract all cosine similarities corresponding to the m-th category from the category-semantic similarity matrix to obtain the image-text similarity set of the m-th category, where m is the category number;

[0115] S52. Based on the clustering objective function, perform clustering on the image-text similarity set of the m-th category to obtain multiple generalized semantic centers corresponding to the category.

[0116] S53. Extract the candidate semantic texts with the highest similarity from each generalized semantic center to form a set of generalized semantic texts for that category.

[0117] In this embodiment, the expression for the clustering objective function in S52 is: ,

[0118] in, The clustering objective function is... For the m-th category, the set of image-text similarity is... For the m-th category, the set of image-text similarities The r-th cosine similarity in the matrix represents the k-th cluster center of the m-th category. Let k be the squared Euclidean distance, k be the cluster center number, and r be the image-text similarity set of the m-th category. The cosine similarity index is given by k, where k is the number of cluster centers.

[0119] S5 starts from the image side, looking for the largest semantic coverage set that can cover the typical features of this type of bacteria.

[0120] The formula for the generalized semantic text set constituting this category in S53 is: ,

[0121] in, Let m be the set of generalized semantic texts for the m-th category. Extract the top m candidate semantic texts with the highest similarity to the k-th cluster center of the m-th category. Let k be the cluster center of the m-th category, where M is the number of candidate semantic texts and k is the number of cluster centers. This is to perform a set union operation on the results of k cluster centers of the m-th category.

[0122] To ensure that sufficiently rich semantic expressions corresponding to each type of bacteria can be screened, this invention limits the number of generalized semantic centers k to be equal to the total number of bacterial categories, and limits the number of candidate semantic texts M screened by each semantic center to at least 1.

[0123] In this embodiment, S6 includes the following sub-steps:

[0124] S61. Find the cosine similarity between the nth candidate semantic text and the mth category in the category-semantic similarity matrix, and construct the text-image similarity set between the nth candidate semantic text and the mth category, where n is the number of the candidate semantic text and m is the number of the category;

[0125] S62. Calculate the mean of each cosine similarity between the nth candidate semantic text and the text-image similarity set of the mth category, and obtain the semantic center of the nth candidate semantic text for the mth category.

[0126] S63. Among the text semantic centers of the m-th category of multiple candidate semantic text pairs, select the maximum value to obtain the comparative text semantic center of the m-th category;

[0127] S64. Subtract the semantic center of the nth candidate semantic text from the semantic center of the mth category from the semantic center of the contrasting text of the mth category to obtain the inter-class discrimination score of the candidate semantic text for the mth category.

[0128] S65. In the m-th category, select candidate semantic texts whose inter-class discrimination scores are greater than the threshold to form a discriminative semantic text set for the m-th category.

[0129] S6 starts from the text side and finds cls semantic text discriminant sets that can distinguish different types of bacteria based on semantic text, where cls is the total number of bacterial categories.

[0130] In this embodiment, the formula for calculating the semantic center of the text in S62 is: ,in, Let n be the semantic center of the nth candidate semantic text pair for the mth category. Let n be the set of text-image similarities between the nth candidate semantic text and the mth category. The set of text-image similarities between the nth candidate semantic text and the mth category. The c-th cosine similarity is given by C, where C is the number of cosine similarities between the n-th candidate semantic text and the text-image similarity set of the m-th category, and c is the index of the cosine similarity between the n-th candidate semantic text and the text-image similarity set of the m-th category.

[0131] In this embodiment, the expression for obtaining the inter-class discrimination score of the candidate semantic text for the m-th category in S64 is: ,in, The inter-class discrimination score between the nth candidate semantic text and the mth category is given. Let m be the semantic center of the contrastive text for the m-th category.

[0132] In this embodiment, the scoring threshold is generally set to 0.25 so that the text semantic set of each bacterium has sufficiently high discriminative power.

[0133] In S7, the core semantic text of each type of microscopic object is obtained by taking the intersection of the two and screening it to find that it has both "universality" and "exclusivity".

[0134] Results review and evaluation:

[0135] Each bacterial class should have at least one valid semantic text, and the valid semantic text description of each bacterial class should conform to human cognition and medical common sense.

[0136] If a bacterial category does not yield a sufficiently large and qualified set of semantic texts, the relevant screening process parameters can be adjusted: increase K and M to obtain a broader semantic text expression for each bacterial category, and decrease the scoring threshold to obtain more general bacterial semantic texts.

[0137] This embodiment uses six types of microscopic bacterial images (G+C positive cocci, GB negative bacilli, G+Y yeast fungi, G+B positive bacilli, G+SC positive streptococci, and G+ST positive staphylococci) as the object to verify the matching effect between the generalized semantic text synthesized by this technical solution and the bacterial images. The results are as follows:

[0138] After synthesizing generalized semantic text using this technique, the high similarity semantic text and matching logic for each bacterial category are as follows:

[0139] 1. Gram-positive cocci: The corresponding generalized semantic texts are "G+C", "Pairs", and "Cocci". "G+C" represents the bacterial category; "Pairs" aligns with the objective rule in laboratory medicine that "gram-positive cocci often appear in pairs"; and "Cocci" corresponds to the spherical morphology of the cocci. These semantic texts are fully aligned with the Gram-positive cocci image in the latent space of the multimodal model. Subsequently, in text-guided target detection and other multimodal models, using these three texts as input, Gram-positive cocci in the image can be accurately located.

[0140] 2. G+B positive bacilli: The corresponding generalized semantic text is "Gram-positive rod", which is consistent with the "type determination criteria for positive bacilli" in laboratory medicine, achieving a precise match between semantics and bacterial morphological characteristics.

[0141] 3. G+SC positive chain cocci: The corresponding generalized semantic texts are "single cells" and "long chains". This result is consistent with the facts in laboratory medicine: positive chain cocci are composed of multiple single positive cocci arranged in a chain-like structure. The two semantic texts correspond to its "cell composition" and "arrangement morphology" features, respectively.

[0142] 4. Grape-like clusters (G+ST positive): The corresponding generalized semantic text is "Grape-like Clusters", "singlecells", and "G+C". Among them, "Grape-like Clusters" corresponds to its grape-like arrangement, "single cells" and "G+C" correspond to its laboratory medical characteristic of "composed of a single positive coccus". The matching logic between semantics and image features conforms to the clinical laboratory test rules.

[0143] The above results show that the generalized semantic text synthesized in this invention can not only be effectively aligned with the image features of the corresponding bacterial categories in the multimodal model, but also conform to the objective laws of laboratory medicine, and can directly support the downstream tasks of the subsequent multimodal model.

[0144] This invention obtains a general semantic set containing microbial professional characteristics through a large language model, making up for the shortcomings of general models in lacking professional semantics. Then, the microscopic images are processed by background standardization to adapt to the model's visual feature learning needs. The image-text model is used to achieve the mapping and alignment between semantics and image features. Combined with the bi-clustering strategy of intra-class clustering to mine generalized semantics and inter-class clustering to filter discriminative semantics, as well as intersection filtering, the core semantic text with both generalization and discriminativeness is finally generated. This invention accurately solves the problems of failure of image-text alignment in microscopic scenes and insufficient generalization ability caused by visual feature mismatch and lack of professional semantics in existing general multimodal base models, and provides high-quality semantic support for subsequent fine-tuning of multimodal models.

[0145] Current multimodal pre-trained models such as CLIP, and open-set general detection models based on multimodal pre-training such as GLIP, Grounding Dino, and Sam3, exhibit poor generalization in the fields of optical microscopic objects and specialized laboratory medicine, lacking effective generalized semantic text. This invention utilizes image data from specialized laboratory medicine and rich semantic text data from large language models to synthesize and filter generalized semantic text data suitable for subsequent fine-tuning of visual language models (such as semantic text-guided microscopic object detection).

[0146] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating core semantic text from microscopic images using a dual-clustering process, characterized in that, Includes the following steps: S1. Use a large language model to obtain a general semantic set; S2. Cut out the target images from the microscopic images, scale them up, and then paste them onto the background images respectively, and classify them to obtain a set of microscopic target images for each category. S3. After preprocessing the general semantic set and the image set of each category of microscopic target, input them into the trained image-text model to obtain the text representation vector and the image representation vector; S4. Calculate the cosine similarity between the text representation vector and the image representation vector, and construct a category-semantic similarity matrix; S5. Extract the image-text similarity set from the category-semantic similarity matrix and perform intra-class clustering to obtain the generalized semantic text set for each category; S6. Extract the text-image similarity set from the category-semantic similarity matrix and perform inter-class clustering to obtain the discriminative semantic text set for each category; S7. Take the intersection of the discriminative semantic text set and the generalized semantic text set of the same category to obtain the core semantic text; S5 includes the following steps: S51. Extract the first from the category-semantic similarity matrix. The cosine similarity of all categories is obtained to get the i-th... A set of image-text similarity categories, For the category number; S52. Based on the clustering objective function, for the first... Clustering is performed on image-text similarity sets of each category to obtain multiple generalized semantic centers corresponding to that category; S53. Extract the candidate semantic texts with the highest similarity in each generalized semantic center to form a set of generalized semantic texts for that category. The expression for the clustering objective function in S52 is as follows: ， in, The clustering objective function is... For the first A set of image-text similarity categories, For the first Image-text similarity sets for each category The Middle Cosine similarity, For the first The first category Cluster centers, The square of the Euclidean distance. The cluster center number, For the first Image-text similarity sets for each category Numbering of cosine similarity The number of cluster centers; S6 includes the following sub-steps: S61. Find the first [item] in the category-semantic similarity matrix. The candidate semantic text and the first The cosine similarity corresponding to the category constitutes the _th _ ... The candidate semantic text and the first A set of text-image similarity categories, This is the ID of the candidate semantic text. For the category number; S62, Calculate the first The candidate semantic text and the first The mean of each cosine similarity in the text-image similarity set of each category is used to obtain the th... The candidate semantic text pair Text semantic center of each category; S63, Find all candidate semantic text pairs. The largest text semantic center in each category is obtained as the [number]th [class]. Contrast text semantic centers for each category; S64, the first The candidate semantic text pair The semantic center of the text in the first category and the first The difference between the semantic centers of the contrasting texts of each category is used to obtain the candidate semantic text pairs. Inter-class discrimination scores for each category; S65, in the In each category, candidate semantic texts with inter-class discrimination scores greater than a threshold are selected to constitute the first category. A set of discriminative semantic texts for each category.

2. The method for generating core semantic text from microscopic images using a dual-clustering process according to claim 1, characterized in that, S1 includes the following steps: S11. Construct structured text prompts; S12. Based on the structured text prompts, a large language model is used to generate a candidate semantic text library; S13. Extract a general semantic set containing microbial morphology, structure and biological characteristics from the candidate semantic text library.

3. The method for generating core semantic text from microscopic images using a dual-clustering process according to claim 1, characterized in that, S2 includes the following steps: S21. Obtain a microscopic image labeled with the target location coordinates and target category; S22. Based on the target position coordinates, crop all targets in the microscopic image to obtain the target image; S23. Scale each target image to 224 pixels along its longest side while keeping the aspect ratio of the target image unchanged to obtain the scaled target image; S24. Paste the scaled-down image of the target onto the background image to obtain the microbial target-background image; S25. According to the target category, the microbial target-background images are classified and placed into the corresponding folders to obtain a collection of microscopic target images for each category.

4. The method for generating core semantic text from microscopic images using a dual-clustering process according to claim 1, characterized in that, S3 includes the following steps: S31. Divide the general semantic set into the smallest semantic unit, token; S32. Embed the smallest semantic unit, the token, into a vector to obtain a semantic vector; S33. Input the semantic vector into the text input terminal of the trained image-text model to obtain the text representation vector; S34. Normalize the pixel values ​​of each image in each category of microscopic target image set to obtain a preprocessed image; S35. Input the preprocessed image into the image input terminal of the trained image-text model to obtain the image representation vector.

5. The method for generating core semantic text from microscopic images using a dual-clustering process according to claim 1, characterized in that, In S3, the loss function used during the training of the image-text model is: ,in, For loss function, For the first The similarity loss function between the text representation vector and all image representation vectors, For the first The similarity loss function between each image representation vector and all text representation vectors.

6. The method for generating core semantic text from microscopic images using a dual-clustering process according to claim 5, characterized in that, Similarity loss subfunction The expression is: ， Similarity loss subfunction The expression is: ， in, It is a logarithmic function. For the first training phase The candidate semantic text corresponding to the first The text representation vector and the first The set of microscopic target images corresponding to the first category Each image represents the cosine similarity of vectors. These are hyperparameters used to adjust the discriminative power of the similarity distribution. It is an exponential function; For the first The candidate semantic text corresponding to the first The text representation vector, and the matched first... The set of microscopic target images corresponding to the first category Each image represents the cosine similarity of vectors; For the first The candidate semantic text corresponding to the first The text representation vector, and the matched first... The set of microscopic target images corresponding to the first category Each image represents the cosine similarity of vectors. The number of samples in the training batch. This is the ID of the candidate semantic text. For the category number, This is the number representing the vector in the text representation. This is the number of the image representation vector.

7. The method for generating core semantic text from microscopic images using a dual-clustering process according to claim 1, characterized in that, S4 includes the following steps: S41. Calculate the cosine similarity between each text representation vector and each image representation vector; S42. Arrange the cosine similarity of all individual text representation vectors and individual image representation vectors in the order of "text dimension × image dimension" to obtain the category-semantic similarity matrix.

Images