Lung focus medical image segmentation method based on graphics and text information and knowledge embedding

Abstract

The invention relates to a lung focus medical image segmentation method based on graphics and text information and knowledge embedding. The method comprises the following steps: acquiring a lung medical image of a patient and a corresponding clinical diagnosis report; preprocessing the lung medical image to obtain an enhanced image; inputting the lung medical image and the clinical diagnosis report into the medical visual language model to obtain a focus prompt embedding vector; and inputting the lung medical image, the enhanced image and the focus prompt embedding vector into the medical image segmentation model to obtain a lung focus region segmentation image. By adopting the method, the lung focus can be quickly positioned by the segmentation model through the focus prompt embedding vector, the interference of a non-target area is reduced, and the segmentation accuracy and the target concentration are improved.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition, and in particular relates to a lung lesion medical image segmentation method based on graphic and text information and knowledge embedding. Background Art

[0002] With the development of medical imaging technology, medical images such as lung computed tomography (CT) and X-ray (X-ray) play an important role in the detection of lesions such as lung nodules, consolidation, and ground-glass opacities. However, medical images usually have complex backgrounds, blurred boundaries, and diverse morphologies, which pose great challenges to automated and accurate segmentation.

[0003] Traditional image segmentation methods rely on a single image modality and struggle to fully understand the semantic characteristics of lesions. This is especially true when clinical literature or diagnostic reports contain important information that cannot be utilized by the model. Segmentation models are prone to false positives, missed detections, and unclear boundaries. Furthermore, general semantic segmentation models (such as U-Net and DeepLab) cannot fully leverage medical semantic knowledge and lack the ability to structure modeling of medical terminology, disease location, and lesion manifestations.

[0004] Vision-Language Models (VLMs), such as CLIP (Contrastive Language-Image Pretraining) and BLIP (Self-Training Language-Image Pretraining), are beginning to be applied to joint image-text analysis tasks. These VLMs possess cross-modal understanding capabilities, capable of extracting semantic information from unstructured text such as diagnostic reports, guiding the model for more sophisticated image parsing. However, existing medical image-text models are mostly limited to classification or retrieval and have yet to be systematically applied to pixel-level medical image segmentation tasks. Summary of the Invention

[0005] Based on this, it is necessary to provide a medical image segmentation method for lung lesions based on graphic and text information and knowledge embedding, which can combine graphic and text information understanding and accurate segmentation to address the above technical problems.

[0006] In a first aspect, the present application provides a lung lesion medical image segmentation method based on image and text information and knowledge embedding, comprising:

[0007] Obtain the patient's lung medical images and corresponding clinical diagnosis reports;

[0008] Preprocessing lung medical images to obtain enhanced images;

[0009] Input lung medical images and clinical diagnosis reports into the medical visual language model to obtain the lesion hint embedding vector;

[0010] The lung medical image, enhanced image and lesion hint embedding vector are input into the medical image segmentation model to obtain the lung lesion area segmentation image.

[0011] In one embodiment, lung medical images and clinical diagnosis reports are input into a medical visual language model to obtain a lesion hint embedding vector, including:

[0012] Combine radiomics prior knowledge to embed clinical diagnosis reports and obtain regular text features;

[0013] Analyze clinical diagnosis reports to obtain lesion areas and corresponding infection information;

[0014] Align the lesion area and the corresponding infection information on the lung medical image to obtain a labeled image;

[0015] The labeled image is combined with regular text features to generate lesion hint embedding vectors.

[0016] In one embodiment, the lung medical image, the enhanced image, and the lesion hint embedding vector are input into a medical image segmentation model to obtain a segmented image of the lung lesion region, including:

[0017] Generate a prompt vector feature based on the lung medical image and the lesion prompt embedding vector;

[0018] Perform feature extraction on lung medical images and enhanced images to obtain dual-channel image feature maps;

[0019] Generate preliminary segmentation mask based on hint vector features and two-channel image feature map;

[0020] The initial segmentation mask is adjusted based on the attention mechanism to obtain the segmentation mask of the lung infection area;

[0021] The lung medical image is segmented according to the segmentation mask to obtain a segmentation image of the lung lesion area.

[0022] In one embodiment, knowledge embedding is performed on clinical diagnosis reports in combination with radiomics prior knowledge to obtain regular text features, including:

[0023] Perform semantic analysis on clinical diagnosis reports to obtain structured lesion information;

[0024] Match the structured lesion information with the corresponding position of the lung medical image to obtain lung image and text information;

[0025] The lung image and text information is optimized by combining radiomics prior knowledge to obtain regular text features.

[0026] In one embodiment, preprocessing a lung medical image to obtain an enhanced image includes:

[0027] Calculate the structural similarity of lung medical images based on fractal geometry and obtain high-level features of lung medical images;

[0028] Extract the tubular structure of the lung medical image based on high-level features and highlight the lesion area of ​​the lung medical image to obtain the initial enhanced image;

[0029] The initial enhanced image is normalized to obtain an enhanced image.

[0030] In one embodiment, the medical visual language model is constructed by the following method:

[0031] Construct a dataset using lung medical images, corresponding doctor diagnosis texts, and image-text alignment data;

[0032] The dataset is used to adjust the parameters of the pre-trained LLAVA-Med model, and the LoRA adaptation layer is inserted to adjust the low-rank features to obtain a medical visual language model.

[0033] The medical image segmentation model is constructed by the following method:

[0034] Construct a training set using lung medical images, corresponding lung lesion segmentation labels, and diagnosis reports;

[0035] The training set was used to adjust the parameters of the pre-trained Med-SAM2 model, and traditional gradient descent was used to optimize the parameters to obtain the medical image segmentation model.

[0036] In one embodiment, a lesion detection report is generated based on the lung lesion region segmentation image and the clinical diagnosis report; the lesion detection report includes lesion region statistics.

[0037] In a second aspect, the present application also provides a lung lesion medical image segmentation device based on image and text information and knowledge embedding, comprising:

[0038] Data acquisition module, used to obtain the patient's lung medical images and corresponding clinical diagnosis reports;

[0039] A feature enhancement module is used to preprocess lung medical images to obtain enhanced images;

[0040] The segmentation guidance module is used to input lung medical images and clinical diagnosis reports into the medical visual language model to obtain lesion hint embedding vectors;

[0041] The image segmentation module is used to input the lung medical image, enhanced image and lesion prompt embedding vector into the medical image segmentation model to obtain a segmented image of the lung lesion area.

[0042] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of any of the above-mentioned methods for medical image segmentation of lung lesions based on embedded graphic information and knowledge.

[0043] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned methods for medical image segmentation of lung lesions based on embedded graphic information and knowledge.

[0044] The above-mentioned lung lesion medical image segmentation method based on image and text information and knowledge embedding combines image enhancement processing, medical visual language model guidance, structured text prompts, and the coordinated optimization of image segmentation models. By preprocessing the original lung medical images, it can enhance the lung texture details and edge information of complex structures, improve the model's ability to recognize small lesions and fuzzy boundary areas, and help obtain more accurate segmentation contours. By jointly inputting clinical diagnosis reports and medical images into the medical visual language model, the semantic description and location prompts of the lesions can be automatically extracted and input into the segmentation model as guidance information, making the model pay more attention to the lesion area and reducing interference from non-target areas, thereby improving segmentation accuracy and target focus. By integrating multimodal knowledge from image and text modalities and utilizing a cross-attention mechanism or feature fusion strategy, the image and text semantic space is effectively aligned, enhancing the model's understanding of complex lesion manifestations while also improving the model's reasoning transparency and clinical interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 Schematic diagram of the process of the lung lesion medical image segmentation method based on graphic information and knowledge embedding of the present invention;

[0047] Figure 2 Schematic diagram of the step-by-step process of step S103;

[0048] Figure 3 Schematic diagram of the step-by-step process of step S104;

[0049] Figure 4 This is a structural diagram of the lung lesion medical image segmentation device based on graphic information and knowledge embedding of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] In one embodiment, Figure 1 As shown, a method for medical image segmentation of lung lesions based on embedded graphic information and knowledge is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0052] S101. Obtain the patient's lung medical images and corresponding clinical diagnosis report.

[0053] For lung infection diseases such as pneumonia, lung medical image lesion segmentation is performed. Schematically, lung medical images are mainly obtained through CT (computer tomography) scanners. The CT scanning equipment scans the patient's lungs based on the principle of X-ray attenuation, uses a detector to receive the X-ray attenuation information after passing through the human body, and then reconstructs the cross-sectional image of the lungs through computer processing to obtain multi-layer slice images. Optionally, lung medical images are stored in the Digital Imaging and Communications in Medicine (DICOM) format, containing the original lung CT image, scanning parameters such as layer thickness, resolution, radiation dose, and patient information. The clinical diagnosis report is written by the doctor based on the patient's symptoms, signs, preliminary judgment of the lung medical image and other relevant examination results, describing information such as the area of ​​the lung lesions and the degree of infection.

[0054] S102. Preprocess the lung medical image to obtain an enhanced image.

[0055] Medical image preprocessing is used to improve image quality and optimize feature expression. Schematically, preprocessing includes denoising using filtering algorithms such as Gaussian filtering and median filtering to remove image noise caused by equipment noise, environmental interference and other factors, making the image clearer and reducing interference with the observation of the lesion area.

[0056] Preprocessing also includes image enhancement techniques such as histogram equalization and image segmentation. Histogram equalization adjusts the grayscale distribution of the image to enhance image contrast, making the boundary between lung tissue and the lesion area more distinct. Image segmentation precisely separates the lung area from the entire chest image, removing interference from other irrelevant tissues and focusing on the lungs themselves.

[0057] S103. Input the lung medical image and clinical diagnosis report into the medical visual language model to obtain a lesion prompt embedding vector.

[0058] The Medical Visual Language Model (MVM) is an AI model that integrates computer vision and natural language processing technologies. It is designed to understand and correlate medical images and text. Pulmonary medical images and corresponding clinical diagnostic reports are fed into the model, which then uses its unique algorithm and architecture to perform a joint analysis of the images and text.

[0059] Schematically, the model performs natural language processing on clinical diagnostic reports, identifying key terms and semantic information, and determining the location of the lesion (e.g., left upper lobe, right lower lobe), morphology (e.g., nodules, patchy shadows), and possible disease types (e.g., pneumonia, pulmonary fibrosis). Furthermore, the model extracts features from lung medical images, using techniques such as convolutional neural networks to extract features such as texture, shape, and density. The model uses the knowledge and patterns learned through training to match and associate lesion information in the text with features in the image, determining the approximate location and extent of the lesion in the image, and then generating a lesion hint embedding vector, providing important guidance for subsequent medical image analysis and diagnosis.

[0060] S104: Input the lung medical image, enhanced image, and lesion prompt embedding vector into a medical image segmentation model to obtain a lung lesion region segmentation image.

[0061] Medical image segmentation models aim to use multi-source information to accurately segment target tissues or lesion areas from medical images.

[0062] Schematically, the model's front-end module, the segmentation information feature extraction module, extracts the cue vector features required by the model based on the lesion cue embedding vector. The model then determines the region of interest (ROI) based on the cue vector features, focusing attention on the ROI where lesions may exist, reducing analysis and calculation of irrelevant areas, and improving segmentation efficiency and accuracy. When processing lung medical images and enhanced images, the model uses methods such as convolutional neural networks for feature extraction to capture subtle features of lesions in the images, including shape, texture, and boundaries. Based on the extracted features, the model uses specific algorithms and neural network structures to perform pixel-level classification and labeling of lung lesion areas, distinguishing between normal tissue and lesion tissue, and generating preliminary segmentation results. Furthermore, the model optimizes and adjusts the preliminary segmentation results, using post-processing techniques such as morphological operations and rule-based corrections to further refine the segmentation boundaries and improve segmentation accuracy, ultimately outputting a precisely segmented image of the lung lesion area.

[0063] The aforementioned lung lesion medical image segmentation method based on image-text information and knowledge embedding obtains patient lung images and clinical diagnostic reports, integrating image and text information. This significantly improves segmentation accuracy compared to relying solely on image information. Preprocessing the lung images creates enhanced images, making lung lesions more prominent and enabling clearer distinction between lesion areas and normal tissue, thereby improving segmentation accuracy. The lung images and clinical diagnostic reports are then fed into a medical visual language model to generate lesion hint embeddings. These hints clearly indicate the likely location of lesions, providing prior knowledge for the medical image segmentation model. Based on these lesion hint embeddings, the segmentation model can more specifically analyze and segment key areas, reducing the blind search for lesions across the entire image and further improving segmentation accuracy. This technical solution automates the generation of segmented lung lesion images, significantly reducing manual effort compared to traditional methods where doctors manually annotate lesions. By integrating prior knowledge from radiomics into clinical diagnostic reports, regular text features are generated. Knowledge embedding enables the model to learn professional knowledge and experience in the medical field, enhancing the model's ability to understand and identify different types of lung lesions.

[0064] In one embodiment, Figure 2 As shown in the figure, lung medical images and clinical diagnosis reports are input into the medical visual language model to obtain the lesion hint embedding vector, including:

[0065] S201. Combining radiomics prior knowledge with clinical diagnosis reports to obtain regular text features.

[0066] Radiomics prior knowledge pre-extracts a large number of quantitative features from medical images through high-throughput analysis to reflect the status of lung lesions. Clinical diagnostic reports often include qualitative descriptions of lesions but lack systematic quantitative information. LLAVA-Med-based medical visual language models, combined with radiomics prior knowledge, can transform implicit knowledge into regular text features that can be understood by the model. Specifically, the LLAVA-Med-based medical visual language model uses its pre-trained language model to perform a preliminary analysis of clinical diagnostic reports, identifying key medical terms and concepts. Furthermore, it integrates the radiomics prior knowledge base with descriptions of lung disease features, such as the quantitative standards and typical manifestations of texture, density, and morphology of different lung lesions on images, to incorporate this knowledge into the understanding of diagnostic reports. For example, if a diagnostic report mentions "ground-glass opacity in the lungs," radiomics knowledge can supplement the quantitative features of this opacity in terms of CT values ​​and texture complexity, thereby generating more regular and professional text features, providing richer information for subsequent analysis.

[0067] S202: Analyze the clinical diagnosis report to obtain the lesion area and corresponding infection information.

[0068] Schematically, the LLAVA-Med-based medical visual language model incorporates a Transformer architecture that uses an attention mechanism to focus on key information in clinical diagnostic reports. After inputting the diagnostic report text, the model uses pre-trained medical terminology word vectors to encode each word, converting the text into a vector representation that the model can understand and process. Furthermore, the attention mechanism identifies key components describing lesion areas and infection information based on the semantic associations between words in the text. For example, higher weights are assigned to words describing location and words describing lesion characteristics and infection types, thereby accurately extracting the lesion area and corresponding infection information.

[0069] S203: Align the lesion area and the corresponding infection information on the lung medical image to obtain a labeled image.

[0070] Schematically, the medical visual language model based on LLAVA-Med uses visual language alignment technology to align lesion areas and infection information with lung medical images. During the training process, the model learned a large number of medical image-text pairs and established a mapping relationship between image features and text descriptions. Specifically, when a lung medical image and the parsed lesion area and infection information are input, the model extracts features from the image and converts the image into a feature vector using technologies such as Convolutional Neural Network (CNN) or Vision Transformer (ViT). Furthermore, the vector representations corresponding to the lesion area and infection information in the text are matched with the image feature vector. By calculating the similarity or correlation between the two, the location of the lesion area in the image is determined, and the infection information is associated with the area to generate a labeled image. For example, for the lesion area of ​​"right lower lobe of the lung", the model will find the corresponding anatomical location in the image and annotate the infection information "patchy shadows, high possibility of inflammation" near the area, so that the image and text information are closely integrated.

[0071] S204: The annotated image is combined with regular text features to generate a lesion hint embedding vector.

[0072] Specifically, the LLAVA-Med-based medical visual language model, after obtaining annotated image and regular text features, further generates a lesion hint embedding vector. The model fuses the annotated image features with the regular text features, again utilizing the attention mechanism to highlight key information related to the lesion. Based on this fused information, the model determines the boundaries of the lesion region in the image and generates a lesion hint embedding vector.

[0073] In one embodiment, Figure 3 As shown in the figure, the lung medical image, enhanced image and lesion hint embedding vector are input into the medical image segmentation model to obtain the lung lesion area segmentation image, including:

[0074] S301. Generate a prompt vector feature based on the lung medical image and the lesion prompt embedding vector.

[0075] Schematically, in the medical image segmentation model based on Med-SAM2, preliminary feature processing is performed on the lung medical image, and at the same time, the segmentation information feature extraction module performs semantic analysis on the lesion prompt embedding vector to extract the prompt vector features required by the medical image segmentation model based on Med-SAM2. Specifically, for lung medical images, the model will use its pre-trained visual module to extract the global features of the image, and for the lesion prompt embedding vector, projection and feature extraction are performed to obtain the prompt vector feature representation. Furthermore, the model will fuse the image features and the prompt vector feature representation to generate a prompt vector feature that can comprehensively reflect the position and semantic information of the prompt box in the image.

[0076] S302: Extract features from the lung medical image and the enhanced image to obtain a dual-channel image feature map.

[0077] Schematically, the medical image segmentation model based on Med-SAM2 adopts a dual-channel feature extraction mechanism. Different convolutional layers or Transformer modules are used for feature extraction for lung medical images and enhanced images, respectively. Specifically, for the original lung medical images, the model will extract its basic structure, texture and other features, reflecting the overall anatomical structure of the lungs and the general morphology of the lesions. For enhanced images, since they have been pre-processed to enhance certain features, such as texture contrast and edge clarity in the lesion area, the model can extract richer and more detailed features, especially features related to the lesions. These two sets of feature maps are combined to form a dual-channel image feature map, which provides more comprehensive and accurate information for subsequent segmentation.

[0078] S303: Generate a preliminary segmentation mask according to the hint vector feature and the dual-channel image feature map.

[0079] The Med-SAM2-based medical image segmentation model interacts with the cue vector features and the dual-channel image feature map. Schematically, the cue vector features are used to determine the region of interest (ROI), focusing attention on areas where lesions may be present. The decoder in the model, combining the cue vector features with the dual-channel image feature map, classifies each pixel and determines whether it belongs to a lesion region. This generates a preliminary segmentation mask, which typically represents each pixel in binary form as a lesion region.

[0080] S304. Adjust the preliminary segmentation mask based on the attention mechanism to obtain a segmentation mask of the lung infection area.

[0081] The Med-SAM2-based medical image segmentation model introduces an attention mechanism to optimize the preliminary segmentation mask. The attention mechanism can automatically focus on important features and areas in the image and ignore irrelevant information. Specifically, the model calculates the attention weight of each pixel based on the feature information in the dual-channel image feature map. Pixels with high correlation with lesion features and areas corresponding to the prompt box are assigned higher attention weights, while pixels in the background or irrelevant areas are assigned lower attention weights. The preliminary segmentation mask is then adjusted based on these attention weights, so that the mask more accurately reflects the boundaries and details of the lung infection area.

[0082] Optionally, the enhanced features in the dual-channel image features and the features extracted by the Med-SAM2-based medical image segmentation model are weightedly fused to optimize the segmentation boundary. Specifically, multi-scale fusion is used to calculate the feature weight matrix of the low-level features and the high-level features, and the lesion edge is smoothly optimized, where the low-level features are the edge information of the lesion and the high-level features are the shape of the lesion area.

[0083] Optionally, medical staff can manually modify the segmentation mask. For example, if the medical staff finds that the model misses the tiny lesions at the edge of the lungs when segmenting, they use the brush tool to accurately draw the missed lesion area on the original CT image to adjust the segmentation mask, and store the adjusted segmentation mask to optimize the model parameters.

[0084] S305 : Segment the lung medical image according to the segmentation mask to obtain a segmented image of the lung lesion area.

[0085] After obtaining an accurate segmentation mask of the infected lung area, the Med-SAM2-based medical image segmentation model applies the segmentation mask to the lung medical image. Schematically, through a pixel-by-pixel operation, pixels marked as lesion areas in the mask are highlighted in the original image, while pixels in non-lesion areas remain unchanged or are appropriately weakened. The resulting segmented image of the lung lesion area clearly shows the location and extent of the lung lesions, providing intuitive guidance for doctors' diagnosis and treatment.

[0086] In one embodiment, knowledge embedding is performed on clinical diagnosis reports in combination with radiomics prior knowledge to obtain regular text features, including:

[0087] S41. Perform semantic analysis on the clinical diagnosis report to obtain structured lesion information.

[0088] Schematically, the clinical diagnosis report is processed using methods such as lexical analysis, syntactic analysis, and semantic understanding in natural language processing (NLP) technology. Optionally, lexical analysis splits the text in the report into individual words or phrases, and identifies key words such as medical terms, symptom descriptions, and location information. Syntactic analysis determines the grammatical relationship between key words and constructs the structural framework of the sentence. Furthermore, the medical visual language model based on LLAVA-Med understands the semantics expressed by the entire sentence and paragraph, and extracts detailed information about the lesion from the text, including the location, morphology, size, etc. of the lesion, and organizes this information into a structured form for subsequent processing. For example, suppose the clinical diagnosis report describes "a quasi-circular nodule with a diameter of about 1.5 cm is seen in the left upper lobe of the patient's lung, with clear boundaries, and is considered to be likely to be benign." Through lexical analysis, key words such as "left upper lobe of the lung", "1.5 cm", "quasi-round nodule", "clear boundaries", and "benign" were identified; syntactic analysis determined the modification relationship between them; after the semantic understanding model integrated the information, the structured lesion information was {location: left upper lobe of the lung, morphology: quasi-round nodule, size: 1.5 cm in diameter, nature: highly likely to be benign}.

[0089] S42. Match the structured lesion information with the corresponding position of the lung medical image to obtain lung image and text information.

[0090] Medical image processing technology and deep learning-based object detection algorithms are used to match structured lesion information with lung medical images. Schematically, the LLAVA-Med-based medical visual language model searches for regions in the image that match the location and morphology of the structured lesion information. The location description in the structured lesion information is converted into a coordinate region in the image, and object detection is performed within this region in combination with the morphological information. Once a matching region is detected, the structured lesion information is associated with the image region to generate lung image-text information. This accurately maps the lesion information described in the text to the specific location and region in the image, achieving image-text integration. For example, the structured lesion information {Location: Left Upper Lobe, Morphology: Circular Nodule, Size: 1.5cm Diameter, Nature: Highly Probable Benign} is associated with the corresponding nodule region in the image to generate lung image-text information, for example {Image Region: Nodule Region within a certain coordinate range in the Left Upper Lobe, Textual Information: Location: Left Upper Lobe, Morphology: Circular Nodule, Size: 1.5cm Diameter, Nature: Highly Probable Benign}.

[0091] S43. Combine radiomics prior knowledge to optimize lung image and text information and obtain regular text features.

[0092] Radiomics prior knowledge contains a large amount of quantitative features and pattern information about lung diseases in images. The image area in the lung image information is extracted, and radiomics technology is used to extract multiple quantitative features such as texture features, morphological features, and density features from the image. The extracted features are analyzed and interpreted in combination with the known disease feature patterns in the radiomics knowledge base. For example, the knowledge base shows that a certain combination of texture features and density features is highly correlated with a specific disease, and the image area in the lung image information has a similar feature combination. Based on the original structured lesion information, the relevant information can be supplemented or optimized to make the text features more regular and professional. These optimized information are organized into regular text features for subsequent medical analysis and diagnosis.

[0093] In one embodiment, preprocessing a lung medical image to obtain an enhanced image includes:

[0094] S51. Calculate the structural similarity of lung medical images based on fractal geometry to obtain high-level features of lung medical images.

[0095] Fractal geometry can describe the complex shapes and structures of natural objects. In lung medical images, different tissues and lesions have unique fractal characteristics. Schematically, the FFM (Fractal Feature Map) module is used to calculate the structural similarity of different regions in lung medical images based on the principles of fractal geometry. Specifically, the FFM module performs multi-scale analysis on the image and calculates the fractal dimension (FD) of the neighborhood around each pixel. The fractal dimension reflects the complexity and irregularity of the objects in the image. Different lung tissues and lesions have different fractal dimension values. By comparing the fractal dimensions of different regions, the structural similarity between them can be obtained. Using this structural similarity information, high-order features of lung medical images are further extracted to more accurately describe the tissue structure and lesion characteristics of the lungs, providing richer information for subsequent analysis. For example, for a lung CT image, the FFM module divides the image into multiple small neighborhoods and calculates the fractal dimension of each neighborhood. Normal lung tissue has a relatively stable fractal dimension range, while the fractal dimension of lesion areas may deviate from this stable range. By comparing the fractal dimensions of different neighborhoods, we found that some areas have significantly different fractal dimensions from the surrounding normal tissue, identifying them as potential lesions. Furthermore, we integrated this fractal dimension information with structural similarity information to derive high-level features of lung medical images.

[0096] S52. Extract the tubular structure of the lung medical image based on the high-level features, and highlight the lesion area of ​​the lung medical image to obtain an initial enhanced image.

[0097] Schematically, methods such as feature matching and threshold segmentation are used to extract tubular structures from lung medical images. Tubular structures in lung medical images often represent important structures such as bronchi and blood vessels, and their morphology and distribution are crucial for the diagnosis of lung diseases. By analyzing texture, shape, and other information within high-level features, characteristic patterns of tubular structures are determined, and matching regions within the image are identified for extraction. For lesion regions, an appropriate threshold is set based on the abnormal information reflected in the high-level features to segment the lesion from normal tissue. The lesion is then highlighted by adjusting pixel values. For example, the pixel values ​​of the lesion region are increased or the color is changed to make it more prominent in the image, resulting in an initial enhanced image that enhances the visual quality and facilitates observation and analysis. Among the obtained high-level features, tubular structures are found to possess specific texture and shape characteristics. By setting matching conditions for texture and shape, matching regions within the lung CT image are identified and marked as tubular structures. For lesion areas, a fractal dimension threshold is set based on information such as fractal dimension abnormalities displayed in the advanced features. Areas exceeding this threshold are marked as lesions. The pixel values ​​of the lesion areas are increased by 50% to make them more visible in the image, while retaining the characteristics of the tubular structure. This results in an initial enhanced image. In this initial enhanced image, the tubular structure is clearly visible, and the lesion area is also highlighted, providing doctors with more intuitive diagnostic information.

[0098] S53: performing normalization processing on the initial enhanced image to obtain an enhanced image.

[0099] Schematically, the initially enhanced image is normalized by HU (Hounsfield Unit), and the HU value of the image is uniformly mapped to a specific standard range, so that the image after FFM enhancement is more uniform and comparable in value, thereby obtaining an enhanced image.

[0100] In one embodiment, the medical visual language model is constructed by the following method:

[0101] S61. Construct a dataset using lung medical images, corresponding doctor diagnosis texts, and image-text alignment data.

[0102] A large number of lung medical images are collected, covering different disease types, different severities, and different individual lung conditions to ensure data diversity. The doctor's diagnosis text corresponding to each image is obtained, which contains a detailed description of the location, morphology, and nature of the lung lesions. Image-text alignment data is generated through manual annotation or specific algorithms, accurately mapping the lesion description in the diagnostic text to the specific area in the image, clarifying the image location and range referred to by the text. The three types of data are combined to construct a complete dataset, providing rich and valuable data support for subsequent model training.

[0103] S62. Use the dataset to adjust the parameters of the pre-trained LLAVA-Med model, and insert the LoRA adaptation layer to adjust the low-rank features to obtain a medical visual language model.

[0104] Based on the LLAVA-Med model architecture, the model has been pre-trained on large-scale general-purpose data and possesses a certain level of foundational image and text understanding capabilities. When using the aforementioned specially constructed pulmonary medicine dataset, the model begins to adjust parameters based on this domain-specific data, attempting to understand the relationship between lung image features and corresponding text descriptions. Furthermore, a LoRA (Low-Rank Adaptation) adaptation layer is used for parameter adjustment. This layer is inserted between the Visual Transformer (ViT) and the Language Transformer (LLaMA). During training, the parameters of the ViT and LLaMA backbone networks are frozen, and only the parameters of the LoRA layer are updated. Backpropagation is used to calculate gradients based on the difference between the model's predictions and the image-text alignment data in the dataset, which are then used to update the parameters of the LoRA layer. This approach avoids the need to fully adjust the parameters of the large pre-trained model, reducing computational and memory requirements while also preventing catastrophic forgetting caused by over-adjusting pre-trained parameters.

[0105] The resulting medical visual language model parameters reflect the model's understanding of the relationship between lung medical images and text during the learning process, as well as its acquired medical knowledge. By loading these parameters, a medical visual language model was constructed that can accurately parse physician diagnostic text, extract lesion descriptions, achieve image-text alignment, and generate effective lesion description embedding vectors.

[0106] Optionally, a cross-entropy loss function is used to measure the difference between the text output extracted by the medical visual language model and the actual doctor's diagnosis text. The cross-entropy loss function can effectively handle multi-classification problems. For various diagnostic descriptions and labels in medical text, it can accurately calculate the gap between the model's predicted probability distribution and the true distribution, thereby guiding the model to learn more accurate text representations. Optionally, an L1 loss function is used to calculate the difference between the model-predicted lesion prompt embedding vector and the actual lesion. The L1 loss function is the sum of the absolute errors between the predicted value and the true value. It is robust to outliers and can enable the model to focus on the overall accuracy of the prompt box parameters, avoiding excessive influence from individual abnormal samples.

[0107] The medical image segmentation model is constructed by the following method:

[0108] S71. Construct a training set using lung medical images, corresponding lung lesion segmentation labels, and diagnosis reports.

[0109] Lung medical images, as fundamental data, contain information about various lung structures and potential lesions. Lung lesion segmentation labels accurately annotate the true lesion areas in the images, representing the actual lesion extent. Structured text information can be extracted from diagnostic reports, providing accurate targets for model learning.

[0110] S72. Use the training set to adjust the parameters of the pre-trained Med-SAM2 model and use traditional gradient descent to optimize the parameters to obtain a medical image segmentation model.

[0111] The Med-SAM2 model has been pre-trained on a certain amount of data and possesses basic image segmentation and feature extraction capabilities. When using a training set specifically for lung lesion segmentation, the model is adjusted based on the specific data, attempting to apply the general image segmentation knowledge learned during the pre-training phase to the lung lesion segmentation task, identifying the potential relationship between image features such as edges and textures and lesions.

[0112] During model training, for each training sample, the model processes the image based on the current parameter settings and generates a segmentation result. A loss function is used to calculate the error between the predicted result and the true label. Traditional gradient descent algorithms, such as stochastic gradient descent (SGD) and adaptive moment estimation (Adam), are further employed to minimize the difference between the model's predicted segmentation results and the true lung lesion segmentation labels. Through multiple iterations of training, the model gradually converges to an optimal set of model parameters, enabling the model to more accurately segment lesions when faced with new lung medical images.

[0113] Furthermore, the segmentation accuracy of the medical image segmentation model corresponding to the optimal model parameters is evaluated. The segmentation accuracy evaluation mainly measures the closeness of the segmented image of the lung lesion area to the actual situation by calculating some indicators. Common indicators include Dice coefficient (DICE) and intersection over union (IoU). The Dice coefficient evaluates the accuracy by calculating the ratio of the overlap between the predicted segmented area and the actual lesion area to the total area of ​​the two. The formula is Where A is the predicted segmentation area and B is the actual lesion area. IoU is the ratio of the intersection and union of the predicted area and the actual area, and the formula is The closer the values ​​of these two indicators are to 1, the higher the segmentation accuracy. During the calculation, the segmented image of the lung lesion area is used as the prediction area, while the actual lesion area is obtained from the lung lesion segmentation labels in the training set. For example, the loss between the predicted segmentation result and the actual segmentation label is calculated, the gradient is calculated through backpropagation, and the model parameters are updated using the Adam optimizer. The segmentation accuracy is evaluated again to verify the performance indicators of the model. When the performance no longer improves, training is stopped and the model parameters are retained and confirmed as the final medical image segmentation model parameters.

[0114] The parameters of the medical image segmentation model contain the lung image features, lesion patterns, and the associated knowledge with the lesion prompt embedding vectors learned by the model during the training process, enabling it to efficiently and accurately segment new lung images in practical applications.

[0115] In one embodiment, a lesion detection report is generated based on the lung lesion region segmentation image and the clinical diagnosis report; the lesion detection report includes lesion region statistics.

[0116] Schematically, the segmented image of the lung lesion area is analyzed to obtain relevant statistical information of the lesion area. For example, the number of pixels in each lesion area is calculated. Based on the number of pixels and the known image resolution information, the area of ​​the lesion area, i.e., the lesion area statistics, can be converted. The number of lesions can also be counted, and the distribution of lesions in different areas of the lungs can be analyzed. The report can also include the patient's basic information and key information in the clinical diagnosis report. The report is laid out in a certain format to make the content clear and easy to understand, providing doctors with comprehensive lung lesion analysis information to assist them in making more accurate diagnosis and treatment decisions.

[0117] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0118] Based on the same inventive concept, the embodiments of the present application also provide a device for medical image segmentation of lung lesions based on image-text information and knowledge embedding for implementing the above-mentioned method for medical image segmentation of lung lesions based on image-text information and knowledge embedding. The implementation solution provided by the device is similar to the implementation solution described in the above-mentioned method. Therefore, the specific limitations of one or more embodiments of the device for medical image segmentation of lung lesions based on image-text information and knowledge embedding provided below can be referred to the limitations of the method for medical image segmentation of lung lesions based on image-text information and knowledge embedding above, and will not be repeated here.

[0119] In an exemplary embodiment, Figure 4 As shown, a lung lesion medical image segmentation device based on image and text information and knowledge embedding is provided, comprising:

[0120] Data acquisition module, used to obtain the patient's lung medical images and corresponding clinical diagnosis reports;

[0121] A feature enhancement module is used to preprocess lung medical images to obtain enhanced images;

[0122] The segmentation guidance module is used to input lung medical images and clinical diagnosis reports into the medical visual language model to obtain lesion hint embedding vectors;

[0123] The image segmentation module is used to input the lung medical image, enhanced image and lesion prompt embedding vector into the medical image segmentation model to obtain a segmented image of the lung lesion area.

[0124] In one embodiment, it further includes:

[0125] The knowledge embedding module is used to embed clinical diagnosis reports into knowledge by combining radiomics prior knowledge to obtain regular text features;

[0126] Semantic parsing module, used to parse clinical diagnosis reports to obtain lesion areas and corresponding infection information;

[0127] The text-image alignment module is used to align the lesion area and the corresponding infection information on the lung medical image to obtain an annotated image;

[0128] The segmentation guidance module is also used to annotate images and generate lesion hint embedding vectors based on regular text features.

[0129] In one embodiment, it further includes:

[0130] Embedding vector parsing module, used to generate prompt vector features based on lung medical images and lesion prompt embedding vectors;

[0131] A feature extraction module is used to extract features from lung medical images and enhanced images to obtain dual-channel image feature maps;

[0132] A mask generation module, used to generate preliminary segmentation masks based on the hint vector features and the two-channel image feature map;

[0133] An optimization module is used to adjust the preliminary segmentation mask based on the attention mechanism to obtain the segmentation mask of the lung infection area;

[0134] The image segmentation module is also used to segment the lung medical image according to the segmentation mask to obtain a segmented image of the lung lesion area.

[0135] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0136] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0137] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0138] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A lung lesion medical image segmentation method based on image-text information and knowledge embedding, characterized in that: The method comprises: Obtain the patient's lung medical images and corresponding clinical diagnosis reports; Preprocessing the lung medical image to obtain an enhanced image; Inputting the lung medical image and the clinical diagnosis report into a medical visual language model to obtain a lesion prompt embedding vector; The lung medical image, the enhanced image and the lesion prompt embedding vector are input into a medical image segmentation model to obtain a lung lesion area segmentation image.

2. The method according to claim 1, characterized in that Inputting the lung medical image and the clinical diagnosis report into a medical visual language model to obtain a lesion prompt embedding vector includes: Combining radiomics prior knowledge to embed knowledge into the clinical diagnosis report, and obtaining regular text features; Analyze the clinical diagnosis report to obtain the lesion area and corresponding infection information; Aligning the lesion area and the corresponding infection information on the lung medical image to obtain a labeled image; The annotated image is combined with the regular text feature to generate the lesion prompt embedding vector.

3. The method according to claim 1, characterized in that The step of inputting the lung medical image, the enhanced image, and the lesion prompt embedding vector into a medical image segmentation model to obtain a lung lesion region segmentation image comprises: generating a prompt vector feature according to the lung medical image and the lesion prompt embedding vector; Performing feature extraction on the lung medical image and the enhanced image to obtain a dual-channel image feature map; generating a preliminary segmentation mask based on the hint vector features and the dual-channel image feature map; Adjusting the preliminary segmentation mask based on the attention mechanism to obtain a segmentation mask of the lung infection area; The lung medical image is segmented according to the segmentation mask to obtain the lung lesion area segmentation image.

4. The method according to claim 2, characterized in that The method of embedding the clinical diagnosis report with radiomics prior knowledge to obtain regular text features includes: Performing semantic parsing on the clinical diagnosis report to obtain structured lesion information; Matching the structured lesion information with corresponding positions of the lung medical image to obtain lung graphic information; The lung image and text information is optimized in combination with the radiomics prior knowledge to obtain regular text features.

5. The method according to claim 1, wherein Preprocessing the lung medical image to obtain an enhanced image includes: calculating the structural similarity of the lung medical image based on fractal geometry to obtain high-level features of the lung medical image; extracting the tubular structure of the lung medical image according to the high-level features, and highlighting the lesion area of ​​the lung medical image to obtain an initial enhanced image; Normalization is performed on the initial enhanced image to obtain the enhanced image.

6. The method according to any one of claims 1 to 5, characterized in that: The medical visual language model is constructed by the following method: Construct a dataset using lung medical images, corresponding doctor diagnosis texts, and image-text alignment data; The pre-trained LLAVA-Med model is adjusted using the dataset, and a LoRA adaptation layer is inserted to adjust the low-rank features to obtain the medical visual language model. The medical image segmentation model is constructed by the following method: Creating a training set using the lung medical images, corresponding lung lesion segmentation labels, and diagnosis reports; The training set is used to adjust the parameters of the pre-trained Med-SAM2 model, and traditional gradient descent is used to optimize the parameters to obtain the medical image segmentation model.

7. The method according to claim 6, characterized in that The method further comprises: A lesion detection report is generated based on the lung lesion area segmentation image and the clinical diagnosis report; the lesion detection report includes lesion area statistics.

8. A lung lesion medical image segmentation device based on image and text information and knowledge embedding, characterized in that: The device comprises: Data acquisition module, used to obtain the patient's lung medical images and corresponding clinical diagnosis reports; A feature enhancement module, configured to preprocess the lung medical image to obtain an enhanced image; a segmentation guidance module, configured to input the lung medical image and the clinical diagnosis report into a medical visual language model to obtain a lesion prompt embedding vector; An image segmentation module is used to input the lung medical image, the enhanced image and the lesion prompt embedding vector into a medical image segmentation model to obtain a lung lesion area segmentation image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images