Automatic lesion identification and grading method for medical image

Abstract

The invention provides an automatic focus identification and grading method for a medical image, and the method comprises the steps: carrying out the standardization of an obtained multi-modal original image based on anatomical constraint, and obtaining a standardized image; generating semantic enhancement features through a cross-modal feature compensation network based on the standardized image and associated radiological text description; performing dynamic feature adaptation processing on the semantic enhancement feature to generate a modal adaptive feature; performing context reasoning through a multi-scale feature interaction algorithm based on the modal adaptive features to generate context reasoning features; and lesion identification decoding processing is carried out on the context inference feature map, a lesion segmentation mask is generated, and the lesion segmentation mask is used for extracting lesion area feature parameters to carry out lesion classification. By adopting the method, the adaptability to the missing mode can be enhanced, and the focus identification and grading precision can be improved.

Description

Technical Field

[0001] This invention belongs to the field of medical image-assisted diagnosis, and in particular relates to an automatic lesion identification and grading method for medical images. Background Technology

[0002] With the development of medical image analysis and artificial intelligence technologies, multimodal magnetic resonance imaging (MRI) technology has been widely used in brain tumor segmentation due to its ability to improve segmentation accuracy through complementary information from modalities such as T1, T2, FLAIR, and T1ce. This has led to the development of deep learning-based cross-modal fusion techniques and traditional segmentation methods such as 3DU-Net and nnU-Net. Traditional techniques address the problem of missing modalities primarily through methods such as missing modality synthesis (e.g., conditional generative adversarial networks, diffusion models), knowledge distillation (transferring knowledge from complete modal models to missing modal models), and architecture-based design (e.g., asymmetric collaborative networks, multi-encoder frameworks). In recent years, lightweight and feature fusion technologies have also emerged.

[0003] However, the current approach has obvious problems: it is not robust enough to missing modalities, the efficiency of cross-modal information fusion is low, the computational complexity is high and it is difficult to meet the real-time needs of clinical practice, and feature fusion is difficult to effectively integrate multi-source information such as text, resulting in limited contextual reasoning ability. It generally faces the dilemma of balancing synthesis quality, computational efficiency and information fusion. Summary of the Invention

[0004] Therefore, it is necessary to provide an automatic lesion identification and grading method for medical images to address the above-mentioned technical problems. This method can enhance the adaptability to missing modalities and improve the accuracy of lesion identification and grading through a hierarchical cross-modal fusion network guided by dual features.

[0005] In a first aspect, this application provides an automatic lesion identification and grading method for medical images, including:

[0006] The acquired multimodal raw images are standardized based on anatomical constraints to obtain standardized images;

[0007] Based on standardized images and their associated radiological text descriptions, semantically enhanced features are generated through a cross-modal feature compensation network;

[0008] Dynamic feature adaptation processing is performed on semantic enhancement features to generate modality-adaptive features;

[0009] Based on modal adaptive features, contextual reasoning features are generated through a multi-scale feature interaction algorithm.

[0010] The context reasoning feature map is processed for lesion identification and decoding to generate a lesion segmentation mask. The lesion segmentation mask is used to extract feature parameters of the lesion region for lesion classification.

[0011] In one embodiment, the acquired multimodal raw images are standardized based on anatomical constraints to obtain standardized images, including:

[0012] Obtain a predefined brain anatomy atlas containing key anatomical landmarks of the lateral ventricles, basal ganglia, and subcortical nuclei;

[0013] Based on brain anatomical atlases, a nonlinear registration algorithm is used to align multimodal raw images to a standard anatomical spatial coordinate system to generate spatially registered images.

[0014] Intensity histogram distributions of each modality in the spatially registered image are calculated, and then normalized to the standard intensity range using a histogram matching algorithm to generate a standardized image.

[0015] In one embodiment, semantically enhanced features are generated using a cross-modal feature compensation network based on standardized imagery and its associated radiological text description, including:

[0016] Modal combination identification information is obtained from the original multimodal image. The modal combination identification information is used to characterize the existence status of four modes: T1, T2, FLAIR, and T1 ce.

[0017] Based on modal combination identifier information, the corresponding text description fragments are matched from the predefined radiological description template;

[0018] Semantic features are extracted from text description fragments using a pre-trained cross-modal coding model to generate text semantic feature vectors;

[0019] Standardized images are input into a 3D convolutional network to extract primary visual features and generate primary visual features.

[0020] A block-based cross-attention mechanism is used to fuse textual semantic feature vectors and primary visual features to generate semantically enhanced features.

[0021] In one embodiment, the semantically enhanced features are subjected to dynamic feature adaptation processing to generate modality-adaptive features, including:

[0022] Based on the modal combination identification information of the original multimodal images, a binary encoded vector is generated;

[0023] The binary encoded vector is input into the learnable embedding layer to transform the weight parameters and generate a modality-adaptive weight matrix.

[0024] The semantic enhancement features are split along the channel dimension to obtain independent feature subsets for each modality;

[0025] Based on the modality adaptation weight matrix, channel-weighted enhancement is performed on the independent feature subsets with existing modalities, and zero-value masking is performed on the independent feature subsets with missing modalities to obtain the enhanced feature subsets.

[0026] The enhanced feature subset is processed by merging the channel dimensions to generate modality-adaptive features.

[0027] In one embodiment, context reasoning is performed based on modal adaptive features using a multi-scale feature interaction algorithm to generate context reasoning features, including...

[0028] The modality adaptive features are divided into multiple spatial feature blocks;

[0029] Local convolution processing is performed on each spatial feature block, and 3×3×3 convolution kernels are used to extract the associated features of adjacent voxels;

[0030] Global self-attention processing is performed on all spatial feature blocks to calculate long-range dependencies across feature blocks;

[0031] Based on the gating mechanism, feature fusion processing is performed on the correlation features and long-range dependencies of adjacent voxels to generate block-level fused features;

[0032] Reorganize all block-level fusion features to generate contextual reasoning features.

[0033] Secondly, this application also provides an automatic lesion identification and grading device for medical images, comprising:

[0034] The image standardization module is used to standardize the acquired multimodal raw images based on anatomical constraints to obtain standardized images;

[0035] The text description enhancement module is used to generate semantically enhanced features based on standardized images and their associated radiological text descriptions through a cross-modal feature compensation network;

[0036] The dynamic adaptation module is used to perform dynamic feature adaptation processing on semantic enhancement features to generate modality adaptive features.

[0037] The context reasoning module is used to perform context reasoning based on modality adaptive features and through a multi-scale feature interaction algorithm to generate context reasoning features.

[0038] The lesion identification and grading module is used to perform lesion identification and decoding processing on the context reasoning feature map, and generate a lesion segmentation mask. The lesion segmentation mask is used to extract feature parameters of the lesion region for lesion grading.

[0039] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned automatic lesion identification and grading method for medical images.

[0040] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for automatic lesion identification and grading of medical images.

[0041] The aforementioned automatic lesion identification and grading method for medical images standardizes multimodal raw images based on anatomical constraints to obtain standardized images, providing a unified image foundation for subsequent processing. It utilizes a cross-modal feature compensation network to generate semantically enhanced features based on standardized images and radiological text descriptions, effectively compensating for missing modal information and improving the model's robustness in incomplete modal scenarios. Dynamic feature adaptation processing is applied to the semantically enhanced features to generate modality-adaptive features, enhancing the model's generalization ability and enabling it to adapt to different modality combinations. A multi-scale feature interaction algorithm is used to perform contextual reasoning on the modality-adaptive features, generating contextual reasoning features to enhance contextual reasoning capabilities and fully integrate text and image features. Lesion identification and decoding processing is performed on the contextual reasoning feature map to generate a lesion segmentation mask, extracting lesion region feature parameters for lesion grading. This achieves automatic lesion identification and grading of medical images, effectively solving the problems of insufficient robustness to missing modalities, low efficiency of cross-modal information fusion, high computational complexity that fails to meet real-time clinical needs, and limited contextual reasoning capabilities due to the difficulty in effectively integrating textual information in feature fusion in existing technologies. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating an automatic lesion identification and grading method for medical images provided in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the structure of an automatic lesion identification and grading device for medical images provided in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0047] Multimodal raw images refer to a combination of complementary image data that are acquired simultaneously through magnetic resonance imaging (MRI) and reflect the structure and pathological features of brain tissue. These include, but are not limited to, T1-weighted images, T2-weighted images, fluid attenuation inversion recovery sequences (FLAIR), and contrast-enhanced T1-weighted images (T1 ce). Each modality presents different tissue contrast characteristics to characterize the core, edema, and infiltrative areas of brain tumors. In clinical practice, some modalities may be missing due to equipment limitations or differences in scanning protocols, thus constituting an incomplete multimodal dataset. As the basic data for lesion identification and grading tasks, each modality can provide unique tissue contrast and structural information, which is of key value for the accurate segmentation and diagnosis of brain tumors.

[0048] Radiographic text description refers to a written record in which a professional physician systematically explains the characteristics of a lesion using standardized terminology based on the results of medical imaging examinations. Its core lies in transforming imaging manifestations into professional expressions that include localization (anatomical location), qualitative (morphological characteristics), quantitative (size range), and correlation (relationship with adjacent tissues). Through standardized language (such as terms like "nodule," "calcification," and "enhancement pattern"), it objectively presents abnormal signs of tissue structure, providing quantifiable and reproducible imaging evidence for clinical diagnosis.

[0049] Dynamic feature adaptation is a feature optimization mechanism for medical image processing, particularly in multimodal MRI brain tumor segmentation tasks. Modality gaps can lead to poor feature representation when the model processes different modality combinations. Dynamic feature adaptation can dynamically adjust feature extraction and fusion methods based on the characteristics of different modality combinations, enabling the model to adaptively optimize feature representation. This enhances the model's versatility and flexibility across different modalities, improves its robustness and segmentation accuracy in multimodal scenarios, and ensures accurate lesion identification and grading.

[0050] Based on the above definitions, the automatic lesion identification and grading method for medical images provided in this application embodiment will be described. Indicatively, the implementation environment includes: a sensor group, a terminal, and a processor. The terminal is connected to the sensor group, intelligent adjustment device, and processor via a network; various magnetic resonance signal sensors in a magnetic resonance imaging (MRI) scanner, such as T1, T2, FLAIR, and T1ce modal signal sensors; the processor can be a central processing unit, a neural network processor, or a multi-core processor, without limitation here.

[0051] Based on the above explanations of terms and implementation environments, the application scenarios of the embodiments of this application will be described. The automatic lesion identification and grading method for medical images provided in the embodiments of this application can be applied to scenarios including but not limited to the following:

[0052] In primary healthcare institutions or other resource-constrained settings, incomplete multimodal MRI data is often encountered due to limitations in equipment configuration and scanning protocol specifications (e.g., only T1 and FLAIR sequences can be acquired). This solution automatically identifies available modal combinations through a dynamic feature adaptation mechanism and generates semantically enhanced features by combining them with predefined standardized radiological text descriptions (e.g., "T1 sequence shows low-signal edema zone"), effectively compensating for missing T2 / T1 ce modal information. This process enables the model to stably output segmentation results of the brain tumor core area and edema zone even with a single 1.5T MRI device, providing reliable lesion identification capabilities for areas lacking advanced imaging equipment.

[0053] This is merely an illustrative example; the automatic lesion identification and grading method for medical images provided in this application embodiment can also be applied to other application scenarios. This is only an example and does not limit the specific application scenarios.

[0054] In one exemplary embodiment, such as Figure 1 As shown, an automatic lesion identification and grading method for medical images is provided. This embodiment illustrates the application of this method to a terminal in the aforementioned implementation environment. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps 101 to 105:

[0055] Step 101: Standardize the acquired multimodal raw images based on anatomical constraints to obtain standardized images.

[0056] Specifically, multimodal raw images typically include MRI images of different modalities such as T1, T2, FLAIR, and T1ce. These images contain rich information about brain tissue at different contrasts and sensitivities. However, due to differences in imaging equipment, inconsistencies in scanning parameters, and variations in individual patient anatomy, these raw images are usually located in different coordinate systems and intensity scales. Direct fusion and analysis can lead to significant errors. Therefore, an anatomically constrained standardization method is employed to achieve image standardization. For example, a standard brain anatomy template can be selected, representing the typical anatomical structure and morphology of the brain. Image registration algorithms align each multimodal raw image with this anatomical template. The image registration process aims to find a spatial transformation function that ensures, after transformation to the template coordinate system, the raw image is as anatomically consistent as possible with the template image. This spatial transformation function typically includes rigid transformations such as translation, rotation, and scaling, as well as non-rigid transformations such as elastic deformation, to accommodate individual anatomical differences. Furthermore, mutual information-based registration algorithms can be used to achieve image registration. Mutual information is a statistic that measures the degree of interdependence between two random variables. In image registration, it reflects the similarity of pixel intensity distributions in the spatially overlapping regions of two images. By maximizing mutual information, optimal spatial transformation parameters can be found, achieving the best anatomical match between the original image and the anatomical template. Furthermore, to further eliminate intensity differences between different images, intensity standardization can be performed on the registered images. For example, histogram matching can be used to adjust the intensity distribution of each multimodal image to match the intensity distribution of the anatomical template. Images standardized by anatomical constraints are not only unified in spatial coordinates but also comparable in intensity scale, providing a solid foundation for subsequent feature extraction and fusion.

[0057] Step 102: Based on the standardized image and its associated radiological text description, semantically enhanced features are generated through a cross-modal feature compensation network.

[0058] Specifically, radiological text descriptions are typically written by medical professionals and include expert interpretations of features such as the location, size, shape, and signal intensity of lesions in images, as well as semantic information such as possible diagnostic suggestions. By complementing image data with textual information, a more comprehensive basis for lesion identification can be provided. The core of the cross-modal feature compensation network lies in using a pre-trained CLIP model to encode these textual descriptions, generating semantic feature vectors closely related to the image modality. Subsequently, these semantic feature vectors are fused with feature vectors from standardized images to generate semantically enhanced features. For example, when T1 modality images are missing, fusing the semantic features encoded by the CLIP model with features from other available modalities can effectively compensate for the lack of information caused by the absence of the T1 modality, enabling the model to still obtain lesion feature information relatively comprehensively in incomplete modal scenarios, significantly improving the model's robustness. This cross-modal feature compensation network automatically learns the correlation mapping between text and image features through deep learning, achieving effective supplementation of cross-modal information and providing richer and more accurate feature representations for subsequent lesion identification and grading.

[0059] Step 103: Perform dynamic feature adaptation processing on the semantic enhancement features to generate modality adaptive features.

[0060] Specifically, the key to the dynamic feature adaptation module lies in its learnable embedding mechanism, which adaptively adjusts based on the data features of different modality combinations. For example, this module contains multiple learnable embedding layers, each learning the feature mapping relationship for a specific modality combination. When processing image data with different modality combinations, the dynamic feature adaptation module automatically selects and activates the corresponding embedding layer, performing a non-linear transformation on the semantic enhancement features to generate a feature representation adapted to the current modality combination. This data-driven process enables the model to flexibly adapt to various modality combinations, maintaining stable model performance regardless of which modality is missing by dynamically adjusting the feature representation. For instance, for image data missing both T1 and T2 modalities, the dynamic feature adaptation module can activate the corresponding embedding layer handling the missing modality combination, optimizing the semantic enhancement features to generate modality-adaptive features matching the FLAIR and T1 ce modalities, ensuring the model can still accurately identify lesion areas under this modality combination. This adaptive capability significantly enhances the model's generalization ability, maintaining high recognition accuracy across different datasets and clinical scenarios.

[0061] Step 104: Based on modal adaptive features, contextual reasoning is performed through a multi-scale feature interaction algorithm to generate contextual reasoning features.

[0062] Specifically, the multi-scale feature interaction algorithm aims to construct a fine-grained feature interaction network from local to global perspectives to fully explore the feature information of lesions at different scales and their relationship with surrounding tissues. This algorithm achieves dynamic fusion of text and image features through a block-based cross-attention mechanism. For example, modality-adaptive features are segmented into multiple block regions. Features within each block region undergo cross-attention calculation with global features and features from other block regions, thereby achieving interaction and information transfer between features. For instance, at the local scale, the block-based cross-attention mechanism can capture subtle feature changes at the lesion boundary, accurately locating the lesion's edge contour; at the global scale, this mechanism helps understand the overall structural relationship between the lesion and surrounding brain tissue, as well as its location distribution within the whole-brain anatomy. Through this multi-scale feature interaction, the model can more comprehensively understand the feature manifestations of the lesion, enhance contextual reasoning ability, and provide more accurate and richer feature basis for subsequent lesion grading. By constructing a hierarchical feature interaction network, the multi-scale feature interaction algorithm enables the model to simultaneously focus on the local details and global structural information of the lesion, effectively improving the accuracy of lesion identification and grading.

[0063] Step 105: Perform lesion identification and decoding processing on the context reasoning feature map to generate a lesion segmentation mask. The lesion segmentation mask is used to extract feature parameters of the lesion region for lesion classification.

[0064] Specifically, the context inference feature map undergoes lesion identification and decoding processing to generate a lesion segmentation mask. This mask is generated through a specific decoding network employing a convolutional neural network architecture, containing multiple convolutional layers, deconvolutional layers, and activation function layers. The context inference feature map first enters the convolutional layers of the decoding network for feature extraction and dimensionality transformation. Subsequently, it is progressively upsampled through deconvolutional layers to restore the spatial resolution of the feature map, ultimately generating a lesion segmentation mask that matches the original image size. This segmentation mask is presented as a binary image or probability map, clearly identifying the location and extent of lesion regions in the image. For example, in the generated lesion segmentation mask, lesion regions are represented by a pixel value of 1 (or a high probability value), while normal tissue regions are represented by a pixel value of 0 (or a low probability value), thus achieving accurate segmentation between lesion and normal tissue regions. Based on this lesion segmentation mask, further feature parameters of the lesion region are extracted, such as lesion size, shape, and signal intensity distribution. These feature parameters serve as the basis for lesion grading, providing important references for clinical diagnosis and treatment decisions. Furthermore, the size of the lesion can be calculated by converting the number of pixels in the lesion region within the segmentation mask into the image resolution; the shape features of the lesion can be analyzed based on the contour information of the lesion region in the segmentation mask to extract parameters such as roundness and irregularity; the signal intensity distribution of the lesion is characterized by statistically analyzing the signal intensity values ​​and distribution characteristics of each pixel within the lesion region. These feature parameters are comprehensively evaluated through specific grading models or rules to achieve lesion grading.

[0065] The aforementioned automatic lesion identification and grading method for medical images standardizes multimodal raw images based on anatomical constraints to obtain standardized images, providing a unified image foundation for subsequent processing. It utilizes a cross-modal feature compensation network to generate semantically enhanced features based on standardized images and radiological text descriptions, effectively compensating for missing modal information and improving the model's robustness in incomplete modal scenarios. Dynamic feature adaptation processing is applied to the semantically enhanced features to generate modality-adaptive features, enhancing the model's generalization ability and enabling it to adapt to different modality combinations. A multi-scale feature interaction algorithm is used to perform contextual reasoning on the modality-adaptive features, generating contextual reasoning features to enhance contextual reasoning capabilities and fully integrate text and image features. Lesion identification and decoding processing is performed on the contextual reasoning feature map to generate a lesion segmentation mask, extracting lesion region feature parameters for lesion grading. This achieves automatic lesion identification and grading of medical images, effectively solving the problems of insufficient robustness to missing modalities, low efficiency of cross-modal information fusion, high computational complexity that fails to meet real-time clinical needs, and limited contextual reasoning capabilities due to the difficulty in effectively integrating textual information in feature fusion in existing technologies.

[0066] In one embodiment, the acquired multimodal raw images are standardized based on anatomical constraints to obtain standardized images, including:

[0067] Obtain a predefined brain anatomy atlas containing key anatomical landmarks of the lateral ventricles, basal ganglia, and subcortical nuclei.

[0068] Specifically, the brain anatomical atlas includes precise markings of key anatomical structures such as the lateral ventricles, basal ganglia, and subcortical nuclei. These anatomical markings are obtained through precise localization and annotation by medical experts and stored in a specialized medical imaging database. The brain anatomical atlas provides crucial reference coordinates and structural information for subsequent image registration and standardization, forming the basis for achieving unified anatomical constraints across multimodal imaging.

[0069] Based on brain anatomical atlases, a nonlinear registration algorithm is used to align multimodal raw images to a standard anatomical spatial coordinate system, generating spatially registered images.

[0070] Specifically, based on the acquired brain anatomical atlas, this method uses a nonlinear registration algorithm to precisely align the multimodal raw images to a standard anatomical spatial coordinate system, generating spatially registered images. The core of the nonlinear registration algorithm lies in optimizing a specific energy function to achieve the best spatial match between the anatomical structures in the multimodal raw images and their corresponding structures in the brain anatomical atlas. For example, a nonlinear registration algorithm based on mutual information can be used. This algorithm automatically adjusts parameters such as image displacement, rotation, and scaling by maximizing the mutual information between the multimodal images and the brain anatomical atlas, thereby achieving precise alignment of the images in the standard anatomical spatial coordinate system. This process effectively eliminates spatial deviations caused by different imaging devices, parameter settings, and patient physiological differences, ensuring accurate correspondence of each modality in anatomical structures and laying a solid foundation for the generation of spatially registered images.

[0071] Intensity histogram distributions of each modality in the spatially registered image are calculated, and then normalized to the standard intensity range using a histogram matching algorithm to generate a standardized image.

[0072] Specifically, intensity histogram distribution calculation involves statistical analysis of pixel intensity values ​​across different image modalities to obtain the intensity distribution characteristics of the image data. Histogram matching algorithms, based on predefined standard intensity intervals, adjust and transform the intensity values ​​of the image data, ensuring that the processed image data maintains consistency with the standard interval in terms of intensity distribution. For example, for T1 modality images, the standard intensity interval might be defined based on the typical signal intensity ranges of normal brain tissue and lesion tissue. Through histogram matching algorithms, the intensity histogram of T1 modality data in spatially registered images is adjusted to this standard interval, thereby achieving intensity standardization of T1 modality images. This intensity standardization step not only eliminates systematic biases in intensity measurement between different image data but also enhances the comparability and consistency of multimodal images in subsequent processing, providing a high-quality data foundation for cross-modal information fusion and lesion identification.

[0073] Furthermore, in this embodiment, in addition to the mutual information-based method, a landmark-based registration algorithm can also be used to implement the nonlinear registration algorithm. This algorithm uses key anatomical structure markers in the brain anatomical atlas as landmarks, directly matching the corresponding anatomical structure points in the multimodal original images with the landmarks, and achieving nonlinear registration of the images by minimizing the matching error. In addition, during the intensity histogram distribution calculation and normalization process, different histogram matching strategies, such as piecewise linear matching and polynomial matching, can be selected according to the characteristics of different modalities of images and clinical needs to better adapt to practical application scenarios.

[0074] In one embodiment, semantically enhanced features are generated using a cross-modal feature compensation network based on standardized imagery and its associated radiological text description, including:

[0075] Modal combination identification information is obtained from the original multimodal image. The modal combination identification information is used to characterize the existence status of four modes: T1, T2, FLAIR, and T1ce.

[0076] Specifically, modality combination identification information is typically presented in the form of binary vectors. For example, when T1 and T2 modal images exist, but FLAIR and T1 ce modal images are missing, the modality combination identification information can be represented as [1,1,0,0]. This information can be pre-set and recorded by the image acquisition system or data management system to ensure accurate identification of the presence or absence of each modality during subsequent processing, providing a crucial basis for subsequent text description fragment matching and feature compensation.

[0077] Based on modal combination identifier information, the corresponding text description fragments are matched from the predefined radiological description template.

[0078] Specifically, the predefined radiological description template is a text database compiled by medical experts, covering typical lesion feature descriptions for different modality combinations. For example, when the modality combination identifier is [1,1,0,0], text description fragments applicable to T1 and T2 modality combinations are matched from the template, such as "T1-weighted images show low signal intensity in the lesion area with unclear boundaries; T2-weighted images show increased signal intensity in the lesion area with surrounding edema." Through text retrieval algorithms, the corresponding text description fragments are quickly located and extracted from the template database based on the modality combination identifier information, providing accurate text input for subsequent semantic feature extraction.

[0079] Semantic features are extracted from text description fragments using a pre-trained cross-modal coding model to generate text semantic feature vectors.

[0080] Specifically, a pre-trained cross-modal coding model is used to extract semantic features from the matched text description fragments, generating a text semantic feature vector. This pre-trained cross-modal coding model is trained on large-scale medical image text data, effectively capturing semantic information in the text description and transforming it into a fixed-dimensional feature vector. For example, a BERT model based on the Transformer architecture is used to encode the text description fragments. Its multi-layer neural network structure performs deep semantic parsing of the text, ultimately outputting a text semantic feature vector. This feature vector not only contains lexical semantic information in the text description but also reflects the manifestation of lesion features in different modalities, providing a semantic-level feature representation for subsequent feature fusion.

[0081] Standardized images are input into a 3D convolutional network for primary visual feature extraction, generating primary visual features.

[0082] Specifically, a 3D ResNet network architecture can be used to extract features from standardized images through multiple 3D convolutional layers, pooling layers, and activation function layers. After convolution, the image data extracts primary visual features containing information such as lesion shape, size, and edge features. These features are output in the form of multi-channel feature maps, providing a visual feature representation for subsequent feature fusion.

[0083] A block-based cross-attention mechanism is used to fuse textual semantic feature vectors and primary visual features to generate semantically enhanced features.

[0084] Specifically, the core of the block-based cross-attention mechanism lies in segmenting the primary visual features into multiple block-shaped regions, with the visual features within each block region undergoing cross-attention calculation with the text semantic feature vector. In detail, the primary visual feature map is segmented into multiple equally sized block-shaped regions, and the feature vector of each block region interacts with the text semantic feature vector through an attention mechanism. For example, the feature fusion process in the block-based cross-attention mechanism can be represented by the following formula:

[0085]

[0086] Where Q represents the text semantic feature vector, i.e., the text-side features extracted through a cross-modal coding model, used to query relevance information in visual features; K T The key feature vector, derived from the visual features of the block region, is used to interact with the query vector Q to calculate the relevance weights between the text and the visual features; d k The dimension of the preset feature vector is used to scale the dot product result and prevent gradient vanishing or numerical overflow during softmax calculation; V is the value feature vector, which is also a visual feature from the block region. After the attention weights are calculated, they are used to generate the final fused feature representation. Through this block-based cross-attention mechanism, textual semantic features and visual features are deeply fused in multiple local block regions, allowing visual features to fully absorb textual semantic information, thereby generating semantically enhanced features. This fusion process not only enhances the semantic expressive power of visual features but also effectively compensates for the lack of visual information caused by modality loss, significantly improving the model's ability to identify lesion features.

[0087] For example, when implementing the pre-trained cross-modal coding model in this embodiment, in addition to the BERT model, other pre-trained language models, such as RoBERTa and XLNet, can also be used. These models can be fine-tuned to adapt to the features of medical image text. Regarding the selection of the 3D convolutional network, different depths and structures of 3D CNN architectures, such as 3D DenseNet and 3D Inception, can be selected according to actual needs and computational resources to optimize feature extraction performance. The block region partitioning strategy in the block cross-attention mechanism can also be flexibly adjusted according to image resolution and lesion feature distribution. For example, for high-resolution images, a finer-grained block partitioning can be used to capture more detailed lesion features.

[0088] In one embodiment, the semantically enhanced features are subjected to dynamic feature adaptation processing to generate modality-adaptive features, including:

[0089] Based on the modal combination identification information of the original multimodal images, a binary encoded vector is generated;

[0090] The binary encoded vector is input into the learnable embedding layer to transform the weight parameters and generate a modality-adaptive weight matrix.

[0091] The semantic enhancement features are split along the channel dimension to obtain independent feature subsets for each modality;

[0092] Based on the modality adaptation weight matrix, channel-weighted enhancement is performed on the independent feature subsets with existing modalities, and zero-value masking is performed on the independent feature subsets with missing modalities to obtain the enhanced feature subsets.

[0093] The enhanced feature subset is processed by merging the channel dimensions to generate modality-adaptive features.

[0094] Specifically, based on the modality combination identifier information of the original multimodal image, a binary encoded vector is generated. The modality combination identifier information represents the existence state of four modalities—T1, T2, FLAIR, and T1 ce—in binary form. The generated binary encoded vector is input into a learnable embedding layer for weight parameter transformation, generating a modality-adaptive weight matrix. The learnable embedding layer is a parameter mapping layer learned through training, whose core function is to convert the binary encoded vector into a weight matrix that matches the dimension of the semantic enhancement features. For example, the embedding layer can be a fully connected network with the number of input nodes equal to the length of the modality combination identifier information and the number of output nodes equal to the number of channels of the semantic enhancement features. Through training, the embedding layer learns the feature channel weights corresponding to different modality combinations, thereby generating a modality-adaptive weight matrix. Each element of this weight matrix represents the importance of the corresponding channel in the current modality combination, used for subsequent feature weighting enhancement. The semantic enhancement features are split along the channel dimension to obtain independent feature subsets for each modality. The semantic enhancement feature is a multi-channel feature map, with each channel corresponding to specific feature information. The channel dimension splitting process divides the feature map into independent feature subsets corresponding to each modality based on the modality combination identifier information. For example, assuming the semantic enhancement feature has 256 channels, and each modality has 64 channels, the split independent feature subsets for modalities T1, T2, FLAIR, and T1ce each contain 64 channels, providing a foundation for subsequent differentiated processing of features for different modalities. Based on the modality adaptation weight matrix, channel-weighted enhancement is performed on the independent feature subsets containing modalities, and zero-value masking is applied to the independent feature subsets containing missing modalities, resulting in enhanced feature subsets. Specifically, for the independent feature subsets containing modalities, the corresponding weight vector in the modality adaptation weight matrix is ​​used to weight the features of each channel. For example, if modality T1 exists, and the weight vector corresponding to T1 in the modality adaptation weight matrix is ​​[W1, W2, ..., W...], then... 64If the weight vector is multiplied element-wise with each channel of the T1 independent feature subset, its feature expression is enhanced. For the independent feature subset with missing modalities, the feature values ​​of all its channels are directly set to zero to shield its influence on subsequent processing. This embodiment achieves dynamic adaptation of semantic enhancement features through the above detailed steps and algorithm design. Its technical advantages are mainly reflected in the following aspects: First, by generating binary encoded vectors and learnable embedding layers, personalized modality adaptation weight matrices can be generated for different modality combinations, realizing adaptive feature enhancement for different modality combinations; Second, the splitting and merging of channel dimensions makes the feature processing process more refined, which can highlight the features of existing modalities and effectively shield the interference of missing modalities; Third, the dynamic feature adaptation process is closely integrated with subsequent lesion identification and grading tasks, which significantly improves the robustness and accuracy of the model in incomplete multimodal scenarios and provides more reliable auxiliary support for clinical diagnosis.

[0095] In one embodiment, context reasoning is performed based on modal adaptive features using a multi-scale feature interaction algorithm to generate context reasoning features, including...

[0096] The modality adaptive features are divided into multiple spatial feature blocks;

[0097] Local convolution processing is performed on each spatial feature block, and 3×3×3 convolution kernels are used to extract the associated features of adjacent voxels;

[0098] Global self-attention processing is performed on all spatial feature blocks to calculate long-range dependencies across feature blocks;

[0099] Based on the gating mechanism, feature fusion processing is performed on the correlation features and long-range dependencies of adjacent voxels to generate block-level fused features;

[0100] Reorganize all block-level fusion features to generate contextual reasoning features.

[0101] Specifically, modality-adaptive features are divided into multiple spatial feature blocks. The block size can be dynamically determined based on the distribution characteristics of brain anatomical structures. For example, smaller block sizes can be used in the lateral ventricles and basal ganglia regions to preserve fine structural features, while larger blocks can be used in the white matter regions to improve computational efficiency. This process achieves non-uniform partitioning of feature maps by establishing a three-dimensional spatial index mapping table, enabling higher resolution processing of key anatomical regions. This anatomically guided partitioning strategy differs from conventional uniform grid partitioning and can specifically enhance the feature extraction capability of tumor-prone areas. When performing local convolution processing on each spatial feature block, a multi-level convolution kernel combination design is adopted. Specifically, a 3×3×3 standard convolution kernel can be used first to extract the basic association features of adjacent voxels, then a 5×5×5 separable convolution can be cascaded to expand the local receptive field, and finally a 1×1×1 convolution can be used to compress channel dimension information. For example, when processing edema regions, this combined convolution can simultaneously capture the association patterns of small nodule features and diffuse lesions, avoiding the feature expression limitations caused by single-layer convolution. When performing global self-attention processing, a relative position encoding mechanism is introduced. For example, the Euclidean distance matrix of the center points of all spatial feature blocks is calculated to generate a distance tensor describing the spatial topological relationship. This distance tensor is then converted into position bias coefficients and incorporated into the attention weight calculation process. Furthermore, a multi-head attention architecture can be used to compute the dependencies of different subspaces in parallel, merging the outputs of each head to maintain the spatial constraints of the anatomical structure when modeling long-range dependencies across regions. In this embodiment, feature fusion based on a gating mechanism employs a dual-path interaction strategy, inputting the feature vectors output by local convolution and the feature vectors output by global attention into independent fully connected layers for nonlinear transformation. A learnable gating weight matrix is ​​used to dynamically calculate the fusion ratio coefficient. Finally, the feature responses of the two paths are weighted and fused proportionally. For example, local feature weights are automatically enhanced in the tumor core region, while the global feature contribution is increased in the infiltrative edge region, achieving adaptive optimization of contextual information. When reconstructing block-level fusion features, the processed feature blocks are restored to their original positions according to a preset spatial index mapping table. This method uses sub-pixel convolution technology for block boundary interpolation compensation to eliminate artificial boundary effects caused by block segmentation. A complete contextual reasoning feature map is generated by stitching along the feature channel dimension, with a spatial resolution consistent with the input features, ensuring the continuity of subsequent processing. This embodiment achieves precise localization of lesion regions through an anatomically guided block segmentation mechanism, enhances local feature representation capabilities using a cascaded convolution architecture, maintains spatial topological consistency by incorporating global attention with positional constraints, and utilizes gated fusion to achieve adaptive optimization of multi-scale features, thereby improving the accuracy of brain tumor boundary segmentation.

[0102] In summary, the automatic lesion identification and grading method for medical images provided in this application ensures that different modalities are analyzed within a unified anatomical framework by standardizing multimodal raw images based on anatomical constraints. It utilizes a cross-modal feature compensation network, combined with predefined radiological description templates and a pre-trained cross-modal coding model, to generate semantically enhanced features that compensate for missing modal information. Through dynamic feature adaptation, modality-adaptive features are generated, enhancing the model's adaptability to different modal combinations. A multi-scale feature interaction algorithm, including local convolution and global self-attention processing, combined with a gating mechanism, generates contextual reasoning features that fuse local details and global contextual information. Finally, lesion identification and decoding processes generate lesion segmentation masks for extracting lesion region feature parameters for lesion grading. This technical solution, through a hierarchical cross-modal fusion network guided by dual features, effectively enhances adaptability to missing modalities, improves the accuracy of lesion identification and grading, meets the high-efficiency requirements of real-time clinical applications, and provides a reliable solution for the field of medical image-assisted diagnosis.

[0103] For example, this technical solution effectively solves the problems existing in the incomplete multimodal MRI brain tumor segmentation task in the prior art through the above-mentioned technical means. Its technical advantages are mainly reflected in the following aspects: First, it significantly improves the robustness of the model to missing modalities. Through cross-modal feature compensation networks and dynamic feature adaptation modules, it makes full use of available modal information and semantic information, effectively compensating for the data loss caused by missing modalities, and ensuring the stability and accuracy of the model in incomplete modal scenarios. Second, it enhances the efficiency of cross-modal information fusion, realizing deep fusion of text and image features and interaction of multi-scale features, fully mining lesion feature information, and improving the accuracy of lesion identification and grading. Third, it reduces computational complexity. Through lightweight network design and efficient algorithm optimization, it meets the high-efficiency requirements of real-time clinical applications, providing fast and reliable auxiliary support for clinical diagnosis, and has significant clinical application value and promotion prospects.

[0104] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0105] Based on the same inventive concept, this application also provides an automatic lesion identification and grading device 10 for implementing the automatic lesion identification and grading method for medical images as described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the automatic lesion identification and grading device 10 for medical images provided below can be found in the limitations of the automatic lesion identification and grading method for medical images described above, and will not be repeated here.

[0106] In one exemplary embodiment, such as Figure 2 As shown, an automatic lesion identification and grading device 10 for medical images is provided, comprising:

[0107] Image standardization module 11 is used to standardize the acquired multimodal raw images based on anatomical constraints to obtain standardized images;

[0108] The text description enhancement module 12 is used to generate semantically enhanced features through a cross-modal feature compensation network based on standardized images and their associated radiological text descriptions.

[0109] Dynamic adaptation module 13 is used to perform dynamic feature adaptation processing on semantic enhancement features to generate modality adaptive features;

[0110] Context reasoning module 14 is used to perform context reasoning based on modality adaptive features and through a multi-scale feature interaction algorithm to generate context reasoning features;

[0111] The lesion identification and grading module 15 is used to perform lesion identification and decoding processing on the context reasoning feature map to generate a lesion segmentation mask. The lesion segmentation mask is used to extract feature parameters of the lesion region for lesion grading.

[0112] In one embodiment, the image standardization module 11 includes:

[0113] The anatomical atlas loading unit is used to acquire predefined brain anatomical atlases, which include key anatomical landmarks of the lateral ventricles, basal ganglia, and subcortical nuclei.

[0114] The nonlinear registration unit is used to align multimodal raw images to a standard anatomical spatial coordinate system based on brain anatomical atlases using a nonlinear registration algorithm, thereby generating spatially registered images.

[0115] The intensity normalization unit is used to calculate the intensity histogram distribution of each modality data in the spatially registered image, and normalize it to the standard intensity range through the histogram matching algorithm to generate a standardized image.

[0116] In one embodiment, the text description enhancement module 12 includes:

[0117] The state recognition unit is used to acquire the modality combination identification information of the multimodal raw image. The modality combination identification information is used to characterize the existence state of four modes: T1, T2, FLAIR, and T1ce.

[0118] The template matching unit is used to match the corresponding text description fragment from a predefined radiological description template based on modality combination identification information;

[0119] The semantic coding unit is used to extract semantic features from text description fragments using a pre-trained cross-modal coding model, and generate text semantic feature vectors.

[0120] The visual extraction unit is used to input standardized images into a 3D convolutional network to extract primary visual features and generate primary visual features.

[0121] The cross-modal fusion unit is used to fuse text semantic feature vectors and primary visual features through a block-based cross-attention mechanism to generate semantically enhanced features.

[0122] In one embodiment, the dynamic adaptation module 13 includes:

[0123] The vector generation unit is used to generate binary encoded vectors based on the modality combination identification information of the multimodal original image;

[0124] The embedding transformation unit is used to input binary encoded vectors into the learnable embedding layer to transform weight parameters and generate a modality adaptation weight matrix.

[0125] The channel decoupling unit is used to perform channel-dimensional splitting of semantic enhancement features to obtain independent feature subsets for each modality;

[0126] The dynamic weighting unit is used to perform channel-weighted enhancement on independent feature subsets with existing modes based on the modality adaptation weight matrix, and to perform zero-value masking on independent feature subsets with missing modes to obtain enhanced feature subsets.

[0127] The feature integration unit is used to merge the enhanced feature subsets by channel dimension to generate modality-adaptive features.

[0128] In one embodiment, the context reasoning module 14 includes

[0129] The block processing unit is used to divide the modality adaptive features into multiple spatial feature blocks;

[0130] The local sensing unit is used to perform local convolution processing on each spatial feature block, and uses a 3×3×3 convolution kernel to extract the associated features of adjacent voxels;

[0131] The global modeling unit is used to perform global self-attention processing on all spatial feature blocks and calculate long-range dependencies across feature blocks.

[0132] The gated fusion unit is used to perform feature fusion processing on the correlation features and long-range dependencies of adjacent voxels based on the gating mechanism to generate block-level fused features.

[0133] The feature reorganization unit is used to reorganize all block-level fused features to generate contextual reasoning features.

[0134] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above-described method for automatic lesion identification and grading of medical images.

[0135] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0136] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0137] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. An automatic lesion identification and grading method for medical images, characterized in that, The method includes: The acquired multimodal raw images are standardized based on anatomical constraints to obtain standardized images; Based on the standardized images and their associated radiological text descriptions, semantically enhanced features are generated through a cross-modal feature compensation network; The semantic enhancement features are subjected to dynamic feature adaptation processing to generate modality adaptive features; Based on the modality adaptive features, context reasoning features are generated through a multi-scale feature interaction algorithm. The context reasoning feature map is subjected to lesion identification and decoding processing to generate a lesion segmentation mask. The lesion segmentation mask is used to extract feature parameters of the lesion region for lesion classification.

2. The method according to claim 1, characterized in that, The process of standardizing the acquired multimodal raw images based on anatomical constraints to obtain standardized images includes: Obtain a predefined brain anatomy atlas, which includes key anatomical landmarks of the lateral ventricles, basal ganglia, and subcortical nuclei. Based on the brain anatomical atlas, the multimodal original images are aligned to the standard anatomical spatial coordinate system using a nonlinear registration algorithm to generate spatially registered images. The intensity histogram distribution of each modality data in the spatially registered image is calculated, and then normalized to the standard intensity range using a histogram matching algorithm to generate the standardized image.

3. The method according to claim 1, characterized in that, The standardized imagery and its associated radiological text description, using a cross-modal feature compensation network to generate semantically enhanced features, include: The modality combination identifier information of the original multimodal image is obtained, and the modality combination identifier information is used to characterize the existence state of four modalities: T1, T2, FLAIR, and T1ce. Based on the modality combination identifier information, the corresponding text description fragment is matched from the predefined radiological description template; Semantic features are extracted from the text description fragments using a pre-trained cross-modal coding model to generate a text semantic feature vector; The standardized image is input into a three-dimensional convolutional network for primary visual feature extraction to generate primary visual features. The semantically enhanced features are generated by fusing the text semantic feature vector and the primary visual features through a block-based cross-attention mechanism.

4. The method according to claim 3, characterized in that, The step of performing dynamic feature adaptation processing on the semantic enhancement features to generate modality-adaptive features includes: Based on the modal combination identification information of the original multimodal image, a binary encoded vector is generated; The binary encoded vector is input into the learnable embedding layer to perform weight parameter transformation and generate a modality adaptation weight matrix. The semantic enhancement features are split along the channel dimension to obtain independent feature subsets for each modality; Based on the modality adaptation weight matrix, channel-weighted enhancement is performed on the independent feature subsets with existing modalities, and zero-value masking is performed on the independent feature subsets with missing modalities to obtain the enhanced feature subsets. The enhanced feature subset is then processed by merging the channel dimensions to generate the modality adaptive features.

5. The method according to claim 1, characterized in that, Based on the modality adaptive features, context reasoning is performed through a multi-scale feature interaction algorithm to generate context reasoning features, including... The modality adaptive features are divided into multiple spatial feature blocks; For each of the spatial feature blocks, perform local convolution processing and use a 3×3×3 convolution kernel to extract the associated features of adjacent voxels; Global self-attention processing is performed on all the aforementioned spatial feature blocks to calculate long-range dependencies across feature blocks; Based on the gating mechanism, the adjacent voxel association features and the long-range dependency relationship are fused to generate block-level fused features. Reorganize all block-level fusion features to generate the context reasoning features.

6. An automatic lesion identification and grading device for medical images, characterized in that, The device includes: The image standardization module is used to standardize the acquired multimodal raw images based on anatomical constraints to obtain standardized images; The text description enhancement module is used to generate semantically enhanced features based on the standardized image and its associated radiological text description through a cross-modal feature compensation network; The dynamic adaptation module is used to perform dynamic feature adaptation processing on the semantic enhancement features to generate modality adaptive features. The context reasoning module is used to perform context reasoning based on the modality adaptive features and through a multi-scale feature interaction algorithm to generate context reasoning features. The lesion identification and grading module is used to perform lesion identification and decoding processing on the context reasoning feature map to generate a lesion segmentation mask. The lesion segmentation mask is used to extract feature parameters of the lesion region for lesion grading.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images