Posterior fossa tumor classification and identification method, system and device and storage medium
By constructing a multimodal model framework, combining MRI images and imaging reports, and utilizing a contrastive learning strategy, the problem of scarce posterior fossa tumor data was solved, achieving higher classification accuracy and robustness.
Patent Information
- Application Number
- CN202511505847.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-21
- Publication Date
- 2026-01-06
AI Technical Summary
Existing MRI image classification models perform poorly in scenarios where posterior fossa tumor data is scarce. They require large amounts of training data, consume a lot of resources, and struggle to achieve high accuracy and robustness.
A multimodal model framework was constructed, which combined expert knowledge from MRI images and imaging reports. The model was trained using a contrastive learning strategy through a visual feature extraction module, a text feature extraction module, a cross-attention module, and a classification head to improve the model's classification and recognition capabilities.
It significantly improves the accuracy and robustness of posterior fossa tumor classification under scarce data conditions, and is applicable to MRI images from different hospitals and equipment, demonstrating higher performance and robustness.
Smart Images
Figure CN121280795A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision language multimodal, and in particular to a method, system, device and storage medium for classifying and recognizing posterior fossa tumors. Background Technology
[0002] In recent years, MRI (magnetic resonance imaging) image classification technology has developed rapidly and has been widely used in many fields. Compared with single-modal information, multimodal classification models process data from two modalities simultaneously, obtaining richer semantic information. In the field of tumor classification and recognition, where data is scarce, it can achieve greater accuracy and robustness under the guidance of text information.
[0003] Previous models for image classification tasks typically used image modalities, such as MRI image sequences, or required pre-training with a large amount of image and text data before fine-tuning on the classification task. While pre-training methods can perform well on such tasks, they require a large amount of training data, consume significant server resources, and have long training times. Furthermore, the scarcity of posterior fossa tumor data limits their applicability.
[0004] In summary, although image classification technology has made significant progress, there are still a series of problems and challenges in the data-scarce medical image classification task, which provide directions for further research.
[0005] In view of this, the present invention is hereby proposed. Summary of the Invention
[0006] The purpose of this invention is to provide a method, system, device, and storage medium for classifying and identifying posterior fossa tumors, which fully utilizes expert knowledge in imaging reports, enhances the model's ability to perceive key information, and significantly improves the accuracy and robustness of classification and identification.
[0007] The objective of this invention is achieved through the following technical solution: A method for classifying and identifying posterior fossa tumors, comprising: A multimodal model framework is constructed, including: a visual feature extraction module, a text feature extraction module, a text feature converter, a cross-attention module, and a classification head; Paired posterior fossa MRI images and image reports are collected, and a training dataset is constructed by combining corresponding category labels. The multimodal model framework is then trained, including: extracting visual features from the posterior fossa MRI images using a visual feature extraction module; extracting text features from the image reports using a text feature extraction module; converting the text features to the visual feature space using a text feature converter to obtain converted text features; enhancing the visual features using the converted text features using a cross-attention module to obtain enhanced visual features; classifying the posterior fossa MRI images using visual features and outputting a predicted category; calculating the corresponding contrastive learning loss using the visual features extracted by the visual feature extraction module and the enhanced visual features output by the cross-attention module; constructing a classification loss by combining the category results with the corresponding category labels; constructing a training loss function using all calculated losses and training the multimodal model framework; after training, extracting the visual feature extraction module and the classification head to construct a posterior fossa tumor classification and recognition model. The posterior fossa MRI image is input into the posterior fossa tumor classification and recognition model. After the visual features are extracted by the visual feature extraction module, the classification head outputs the predicted category.
[0008] A posterior fossa tumor classification and identification system for implementing the aforementioned method, comprising: The model framework building unit is used to build a multimodal model framework, including: a visual feature extraction module, a text feature extraction module, a text feature converter, a cross-attention module, and a classification head; The training dataset construction and model training unit is used to collect paired posterior fossa MRI images and image report sets, combine them with corresponding category labels to construct a training dataset, and train the multimodal model framework. This includes: extracting visual features from the posterior fossa MRI images using a visual feature extraction module; extracting text features from the image report sets using a text feature extraction module; converting the text features to the visual feature space using a text feature converter to obtain converted text features; enhancing the visual features using the converted text features using a cross-attention module to obtain enhanced visual features; classifying the posterior fossa MRI images using visual features and outputting a predicted category; calculating the corresponding contrastive learning loss using the visual features extracted by the visual feature extraction module and the enhanced visual features output by the cross-attention module; constructing a classification loss by combining the category results with the corresponding category labels; constructing a training loss function using all calculated losses and training the multimodal model framework; after training, extracting the visual feature extraction module and the classification head to construct a posterior fossa tumor classification and recognition model. The posterior fossa tumor classification and recognition unit is used to input posterior fossa MRI images into the posterior fossa tumor classification and recognition model. After visual features are extracted by the visual feature extraction module, the classification head outputs the predicted category.
[0009] A processing device includes: one or more processors; and a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.
[0010] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.
[0011] As can be seen from the technical solutions provided by the present invention above: (1) By utilizing the expert knowledge contained in MRI image reports, this method exhibits higher robustness and performance when dealing with scarce data scenarios; (2) By using the technique of using text features to guide visual feature learning to focus on the target pathological area, combined with the contrastive learning strategy, the model fully understands the expert knowledge contained in the report and effectively guides the learning of visual features to obtain more accurate prediction results; (3) The present invention adopts a supervised training method for training, effectively guiding visual feature learning with text information, thereby improving the accuracy of model prediction. Attached Figure Description
[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 A flowchart of a method for classifying and identifying posterior fossa tumors provided in an embodiment of the present invention.
[0014] Figure 2 This is a schematic diagram of the overall framework of a method for classifying and identifying posterior fossa tumors provided in an embodiment of the present invention.
[0015] Figure 3 This is a schematic diagram illustrating the data display and prediction effect provided in the embodiments of the present invention.
[0016] Figure 4 This is a schematic diagram of a posterior fossa tumor classification and identification system provided in an embodiment of the present invention.
[0017] Figure 5 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation
[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.
[0019] First, the following explanations are provided for the terms that may be used in this article: The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.
[0020] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.
[0021] The following is a detailed description of the method, system, device, and storage medium for classifying and identifying posterior fossa tumors provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, conventional conditions in the art or conditions recommended by the manufacturer shall apply. Reagents or instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.
[0022] Example 1 This invention provides a method for classifying and identifying posterior fossa tumors, such as... Figure 1 As shown, it mainly includes the following steps: Step 1: Construct a multimodal model framework.
[0023] In this embodiment of the invention, the constructed multimodal model framework mainly includes: a visual feature extraction module, a text feature extraction module, a text feature converter, a cross-attention module, and a classification head.
[0024] Step 2: Build a training dataset and train a multimodal model framework.
[0025] In this embodiment of the invention, paired posterior fossa MRI images and image report sets are collected, and a training dataset is constructed by combining the corresponding category labels, including: The paired posterior fossa MRI images and image report set are represented as follows: , ,in, Let t be the t-th pair of data in set X, containing posterior fossa MRI images. and its paired image report X represents a set of paired posterior fossa MRI images and image reports; posterior fossa MRI images It consists of a stack of T1, T1E, and T2 mode sequences, represented as: , This represents the sequence corresponding to mode m. T1, T1E, and T2 are different types of imaging sequences, which reflect the signal performance of human tissues under different physical properties.
[0026] Data The category label is denoted as This forms the t-th data item in the training dataset, ultimately constructing the training dataset. , N is the total number of data items in the training data.
[0027] In this embodiment of the invention, the training process is as follows: visual features of the posterior fossa MRI image are extracted by the visual feature extraction module; text features of the image report set are extracted by the text feature extraction module; the text features are converted to the visual feature space by the text feature converter to obtain the converted text features; the visual features are enhanced by the converted text features using the cross-attention module to obtain the enhanced visual features; the classification head classifies the images using the visual features of the posterior fossa MRI image and outputs the predicted category; the contrastive learning loss is calculated using the visual features extracted by the visual feature extraction module and the enhanced visual features output by the cross-attention module, and the classification loss is constructed by combining the category results with the corresponding category labels; the training loss function is constructed using all the calculated losses and the multimodal model framework is trained.
[0028] In this embodiment of the invention, the visual feature extraction module is implemented using an open-source visual feature extraction model pre-trained from natural images. Based on a residual mechanism, it uses a convolutional neural network for image feature extraction, as shown below: ; in, Representing MRI images of the posterior fossa The visual features are defined as follows: t represents the t-th data item in the training dataset, and VE represents the visual feature extraction module, which is used to extract visual features.
[0029] In this embodiment of the invention, the text feature extraction module is implemented using a pre-trained large language model, and the features output by the last hidden layer of the large language model are used as text features, represented as: ; Here, TE represents the text feature extraction module. Image report The text features are denoted by t, where t represents the t-th data item in the training dataset.
[0030] In this embodiment of the invention, the step of enhancing visual features using the transformed text features through a cross-attention module to obtain enhanced visual features includes: Cross-attention module with transformed text features As query Q, visual features As the key K and value V, the enhanced visual features are obtained, represented as: ; in, Represents the cross-attention module. This represents enhanced visual features.
[0031] In this embodiment of the invention, the calculation of the corresponding contrastive learning loss using the visual features extracted by the visual feature extraction module and the enhanced visual features output by the cross-attention module includes: The first contrastive learning loss is constructed by combining the visual features extracted from different MRI images by the visual feature extraction module. Furthermore, by combining the visual features extracted by the visual feature extraction module with the enhanced visual features output by the cross-attention module, a second contrastive learning loss is constructed. This second loss is the contrastive learning loss between knowledge-enhanced samples, and is expressed as: ; in, For hyperparameters, The second contrastive learning loss is used, where t represents the t-th data item in the training dataset, and N is the total number of data items in the training dataset. For the t-th data item in the training dataset, the posterior fossa MRI image Visual features For the corresponding enhanced visual features, Let sim be the enhanced visual features of the posterior fossa MRI image in the j-th data item of the training dataset, where sim is the cosine similarity, log is the logarithmic function, and exp is the natural exponential function.
[0032] In this embodiment of the invention, the first contrastive learning loss includes: contrastive learning loss between adversarial examples and contrastive learning loss between classes.
[0033] Among them, for posterior fossa MRI images Noisy MRI images are obtained by introducing random noise. The visual features of each posterior fossa MRI image and each noisy MRI image are extracted using a visual feature extraction module. These features are then used to calculate the contrast learning loss between adversarial examples using the following formula. : ; in, For hyperparameters, For noisy MRI images Visual features For noisy MRI images Visual features For MRI images of the posterior fossa Random noise is introduced to obtain noisy MRI images.
[0034] For posterior fossa MRI images Let its positive sample set be denoted as Let its negative sample set be denoted as The corresponding inter-class contrastive learning loss is calculated using the following formula. : ; The inter-class contrastive learning loss for the training dataset is obtained by averaging the following formula. : ; in, , This refers to the category label corresponding to the t-th and j-th data items in the training dataset. For a single positive sample, the visual features are... The visual features corresponding to a single negative sample.
[0035] In this embodiment of the invention, after training is completed, the visual feature extraction module and the classification head are extracted to construct a posterior fossa tumor classification and recognition model.
[0036] Step 3: Classify and identify tumors using a posterior fossa tumor classification and identification model.
[0037] In this embodiment of the invention, the posterior fossa MRI image is input into the posterior fossa tumor classification and recognition model. After the visual features are extracted by the visual feature extraction module, the classification head outputs the predicted category.
[0038] In this embodiment of the invention, the predicted category is generally the category probability or the category score (logits).
[0039] The above-mentioned solutions provided by the embodiments of the present invention have the following main advantages: (1) A training scheme for the classification and identification of posterior fossa tumors in MRI images was designed. By utilizing the expert knowledge contained in MRI image reports, this method exhibits higher robustness and performance when dealing with scarce data scenarios.
[0040] (2) A technique was proposed that text features guide visual feature learning to focus on the target pathological area. Combined with a multiple contrast learning strategy, the model can fully understand the expert knowledge contained in the report and effectively guide the learning of visual features to obtain more accurate prediction results.
[0041] (3) The supervised training method is adopted to effectively guide the learning of visual features with text information, which improves the accuracy of model prediction.
[0042] (4) This method demonstrated state-of-the-art performance in tests using MRI image data from nine different hospitals. This method not only improved the model's classification accuracy but also showed greater robustness and robustness in processing MRI images with different distributions from different hospitals and different MRI equipment.
[0043] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.
[0044] I. Overall Overview of the Plan
[0045] This invention provides a precise classification and identification technology for posterior fossa tumors based on expert knowledge-guided learning. For the first time, image report data is used to guide the visual feature extraction module in learning for posterior fossa tumor classification and identification tasks based on MRI images, thereby improving the model's output accuracy in this task. A novel learning paradigm is designed, combining multiple contrastive learning methods and using image reports as expert knowledge to guide the visual feature extraction module in learning key pathological regions of images. Compared to the pre-training and fine-tuning approach, the training paradigm proposed in this invention requires less data and is more suitable for applications in fields with scarce medical data. This technology not only achieves a significant improvement in visual effects but also reaches industry-leading levels in multiple quantitative performance indicators.
[0046] II. Detailed introduction of the plan.
[0047] The solution provided in this invention is used for model learning and inference in scenarios with scarce training data for posterior fossa tumors. It utilizes information from both image and text modalities, leveraging the powerful feature extraction capabilities gained during pre-training of a large language model. By employing a cross-attention mechanism and a multi-contrast learning strategy, a visual model is constructed for the classification and recognition of posterior fossa tumors. This solution significantly improves the accuracy and robustness of posterior fossa tumor classification and recognition. The following sections provide a detailed description of each step of this solution.
[0048] 1. Obtain training MRI image data and corresponding image text reports.
[0049] (1.1) Obtain paired posterior fossa MRI images and image report sets, denoted as... , ,in, Let t be the t-th pair of data in set X, containing posterior fossa MRI images. and its paired image report , N is the total number of paired samples; any MRI image It consists of a stack of T1, T1E and T2 mode sequences.
[0050] (1.2) For the t-th pair of data MRI image sequences in three modalities (T1, T1E, and T2) were collected. For modality m, , The shape is Where S represents the sequence size, H represents the image height, and W represents the image width. In this embodiment, S is 8, and both H and W are 224. The images of the three modalities are stacked together to obtain... Its shape and size are .
[0051] (1.3) Construct the training dataset, for each of X Obtain its tumor category label Construct a training dataset Here, t and N are the data item numbers and the total number of data items in the training dataset, and they correspond one-to-one with those in X.
[0052] 2. Construct a multimodal model framework.
[0053] like Figure 2 As shown, the multimodal model framework mainly includes: a visual feature extraction module, a text feature extraction module, a text feature adapter, a cross-attention module, and a classification head (not shown in the figure).
[0054] (2.1) The visual feature extraction module is an open-source visual feature extraction model that has been pre-trained on a large number of natural images. Based on the residual mechanism, it uses a convolutional neural network to extract and process image features. In this embodiment, ResNet-18 is used as the visual feature extraction module.
[0055] MRI image data The input is fed into the visual feature extraction module to extract visual features, represented as follows: ; in, This represents the visual feature extracted from the t-th data item in the dataset. VE stands for Visual Feature Extraction Module, which is used to extract visual features.
[0056] (2.2) The text feature extraction module comes from a pre-trained large language model. In this embodiment, Qwen2.5-72B is used as the text feature extraction module, and the features of its last hidden layer are taken as text features. The text information input to the model is initially extracted as follows: ; Here, TE represents the text feature extraction module, which comes from a pre-trained large language model and is used to extract features from the text information in the dataset. Represents text features.
[0057] (2.3) The text feature converter consists of a fully connected neural network and is used to convert text features into a visual feature space: ; in, Represents a text feature converter. This represents the text features adapted to the visual feature space, which are used for subsequent interaction with visual features.
[0058] (2.4) The cross-attention module is composed of multiple stacked transformer layers. The core of each transformer layer mainly consists of a cross-attention mechanism and a feedforward neural network. (For the training dataset...) The t-th data in the dataset, its characteristics As query Q, visual features As the key K and value V, the input to the cross-attention module yields new enhanced visual features: ; in, Represents the cross-attention module. This represents enhanced visual features. Through an attention mechanism, the model is guided by textual information during training, enabling the visual feature extraction module to focus on relevant tumor areas in MRI images.
[0059] (2.5) The classification head is a fully connected neural network used to process the extracted t-th visual feature. The model is used to classify tumors and obtain their predicted categories. ; In this context, MLP (Multilayer Perceptron) represents the classification head. This indicates the tumor category predicted by the model. It is usually a category probability or a category score, with each item representing the probability or score of the stated category.
[0060] 3. Construct the training paradigm and loss function.
[0061] (3.1) For all samples in the training dataset D , During model training, any MRI image Adding random noise to obtain This is used to enhance the robustness of the model and obtain new inputs to the model. .Will and The visual features are obtained by inputting them into the visual feature extraction module. and ;Will The text feature extraction module obtains text features, which are then input into the text feature adapter to obtain text features adapted to the visual feature space. ;Will and Enhanced visual features fed into the cross-attention module .
[0062] (3.2) Loss function construction. In (3.1) above, multiple features were obtained. Different loss functions were used for different features, including adversarial comparison learning loss, inter-class comparison learning loss, knowledge-enhanced comparison learning loss, and cross-entropy loss.
[0063] (3.2.1) Regarding the visual features mentioned in (3.1) above. And visual features extracted after adding noise Using contrastive learning loss between adversarial examples: ; in, Represents cosine similarity, where a and b refer to two pieces of information in the sim function, and T is the transpose sign; It is a hyperparameter. In this embodiment, Set to 1.0.
[0064] (3.2.2) Visual features extracted in (3.1) above For any of these features Learning loss using inter-class comparison: ; in, This represents the set of positive samples for the t-th data item. This represents the set of negative samples for the t-th data item. In this embodiment, Set to 1.0. Then, for each sample... Sum and average: .
[0065] (3.2.3) The visual features extracted in (3.1) above and enhanced visual features In this embodiment, knowledge-enhanced inter-sample contrastive learning loss is used. Set it to 0.1.
[0066] .
[0067] (3.2.4) The visual features extracted in (3.1) above The model predicts the tumor category by inputting the classification header. The classification loss is calculated using the cross-entropy function. : ; in, Category Labels The k-th item (i.e., the probability or score of belonging to the k-th category) has a value of 1 for the true category and 0 for the other categories. For example, if it belongs to the k-th category, the value is 1; otherwise, it is 0. For predicting categories The k-th item (i.e., the probability or score of belonging to the k-th category) is represented by K, where K represents the number of categories.
[0068] (3.2.5) Integrate the training loss function for training: ; in, All of these are hyperparameters. In this embodiment, All are set to 0.1.
[0069] 4. The entire multimodal model framework is trained using gradient descent. During the training process, the text feature extraction module is frozen and does not participate in the training; the other modules participate in the training.
[0070] 5. After training is completed, during the inference phase, the visual feature extraction module and the classification head are retained as the posterior fossa tumor classification and recognition model. The input data only needs to use MRI images without corresponding image reports. The classification head directly uses the visual features extracted by the visual feature extraction module for classification and prediction.
[0071] like Figure 3 The example shown provides a model prediction result, where the predicted value is the category score. The higher the score, the greater the probability of the corresponding category. It should be noted that the specific number of categories and the specific categories can be set by the user according to the actual situation, and this invention does not impose any limitations.
[0072] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0073] Example 2 This invention also provides a posterior fossa tumor classification and identification system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 4 As shown, the system mainly includes: The model framework building unit is used to build a multimodal model framework, including: a visual feature extraction module, a text feature extraction module, a text feature converter, a cross-attention module, and a classification head; The training dataset construction and model training unit is used to collect paired posterior fossa MRI images and image report sets, combine them with corresponding category labels to construct a training dataset, and train the multimodal model framework. This includes: extracting visual features from the posterior fossa MRI images using a visual feature extraction module; extracting text features from the image report sets using a text feature extraction module; converting the text features to the visual feature space using a text feature converter to obtain converted text features; enhancing the visual features using the converted text features using a cross-attention module to obtain enhanced visual features; classifying the posterior fossa MRI images using visual features and outputting a predicted category; calculating the corresponding contrastive learning loss using the visual features extracted by the visual feature extraction module and the enhanced visual features output by the cross-attention module; constructing a classification loss by combining the category results with the corresponding category labels; constructing a training loss function using all calculated losses and training the multimodal model framework; after training, extracting the visual feature extraction module and the classification head to construct a posterior fossa tumor classification and recognition model. The posterior fossa tumor classification and recognition unit is used to input posterior fossa MRI images into the posterior fossa tumor classification and recognition model. After visual features are extracted by the visual feature extraction module, the classification head outputs the predicted category.
[0074] Since the main technical details of this system have been described in detail in previous embodiments, they will not be repeated here.
[0075] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.
[0076] Example 3
[0077] The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.
[0078] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.
[0079] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc. The output device can be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.
[0080] Example 4 The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.
[0081] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.
[0082] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.
Claims
1. A method for classifying and identifying posterior fossa tumors, characterized in that, The method comprises the following steps: constructing a multi-modal model framework, including a visual feature extraction module, a text feature extraction module, a text feature converter, a cross-attention module, and a classification head; collecting paired posterior fossa MRI images and image report sets, combining corresponding category labels to construct a training data set, and training the multi-modal model framework, including: extracting visual features of the posterior fossa MRI images through the visual feature extraction module; extracting text features of the image report set through the text feature extraction module; converting the text features to the visual feature space through the text feature converter to obtain converted text features; using the converted text features to enhance the visual features through the cross-attention module to obtain enhanced visual features; the classification head uses the visual features of the posterior fossa MRI images for classification and outputs a predicted category; calculating a corresponding contrast learning loss using the visual features extracted by the visual feature extraction module and the enhanced visual features output by the cross-attention module, combining the category result with the corresponding category label to construct a classification loss, and using all the calculated losses to construct a training loss function and training the multi-modal model framework; after the training is completed, the visual feature extraction module and the classification head are taken out to construct a posterior fossa tumor classification and recognition model; inputting the posterior fossa MRI image into the posterior fossa tumor classification and recognition model, extracting the visual features through the visual feature extraction module, and outputting the predicted category by the classification head.
2. The method of claim 1, wherein the method is used for classifying and identifying a tumor in the posterior cranial fossa. The method comprises the following steps: The paired posterior fossa MRI images and radiology reports are represented as , , wherein, represents the tth pair of data in the set X, containing a posterior fossa MRI image and its paired radiology report , X is a set of paired posterior fossa MRI images and radiology reports; a posterior fossa MRI image is composed of T1, T1E, and T2 modality sequences stacks, represented as: , represents the sequence corresponding to modality m, T1, T1E, and T2 are different types of imaging sequences, which reflect the signal performance of human tissues under different physical properties; Data of the category label is marked as , the tth item of data in the training data set, and finally the training data set , , N is the total number of data items in the training data. 3.The method of claim 1 or 2, wherein, The visual feature extraction module is implemented by using an open-source visual feature extraction model pre-trained by a natural image, which is based on a residual mechanism and uses a convolutional neural network to extract image features, and is represented as: ； wherein, represents a visual feature of a posterior fossa MRI image t represents the tth item of data in the training data set, and VE represents a visual feature extraction module for extracting the visual feature. 4.The method of claim 1 or 2, wherein, The text feature extraction module is implemented by using a pre-trained large language model, and the features output by the last hidden layer of the large language model are used as text features, and are represented as: ； where TE denotes a text feature extraction module, represents an image report of a text feature, and t denotes the tth item of data in a training data set.
5. The method of claim 1, wherein the method is used for classifying and identifying a posterior fossa tumor. The method comprises the following steps: The cross-attention module takes the transformed text features As the query Q, the visual features As the key K and the value V, the reinforced visual features are obtained, denoted as: ； wherein, represent a cross-attention module, represent reinforced visual features.
6. The method for classifying and identifying posterior fossa tumors according to claim 1, characterized in that, The method comprises the following steps: The method comprises the following steps: The first contrast learning loss includes an inter-adversarial sample contrast learning loss and an inter-class contrast learning loss. ； wherein, is a hyper-parameter, is a second contrastive learning loss, t represents the t-th item of data in the training data set, and N is the total number of data items in the training data; is a visual feature of the posterior fossa MRI image in the t-th item of data in the training data set, is a visual feature of the posterior fossa MRI image in the t-th item of data in the training data set, is a corresponding reinforced visual feature, is a reinforced visual feature of the posterior fossa MRI image in the j-th item of data in the training data set, sim is a cosine similarity, and log is a logarithmic function, and exp is a natural exponential function.
7. The method of claim 6, wherein the method further comprises: The method comprises the following steps: Wherein, for the posterior fossa MRI image , a noise MRI image is obtained by introducing random noise , visual features of each posterior fossa MRI image and visual features of each noise MRI image are extracted by a visual feature extraction module, and the following formula is used to calculate the contrast learning loss between the adversarial samples : ； wherein, is a hyperparameter, is a noisy MRI image is a visual feature of the noisy MRI image, is a noisy MRI image is a visual feature of the noisy MRI image, is a noisy MRI image by introducing random noise. For posterior fossa MRI images Let the positive sample set be denoted as Let the negative sample set be denoted as The corresponding inter-class contrastive learning loss is calculated by : ； The inter-class contrastive learning loss corresponding to the training data set is obtained by averaging by the following formula : ； wherein, , denotes the class label corresponding to the t-th data in the training data set, the j-th data pair, is the visual feature corresponding to a single positive sample, is the visual feature corresponding to a single negative sample.
8. A system for classifying and identifying posterior fossa tumors, comprising: A model framework construction unit is configured to construct a multi-modal model framework, including a visual feature extraction module, a text feature extraction module, a text feature converter, a cross-attention module, and a classification head. The training data set construction and model training unit is used to collect paired posterior fossa MRI images and image report sets, construct a training data set in combination with corresponding category labels, and train the multi-modal model framework, including: extracting visual features of the posterior fossa MRI image through a visual feature extraction module; extracting text features of the image report set through a text feature extraction module; converting the text features to a visual feature space through a text feature converter to obtain converted text features; using the converted text features to enhance the visual features through a cross-attention module to obtain enhanced visual features; the classification head uses the visual features of the posterior fossa MRI image for classification and outputs a predicted category; the visual features extracted by the visual feature extraction module and the enhanced visual features output by the cross-attention module are used to calculate a corresponding contrast learning loss, a classification loss is constructed in combination with a category result and corresponding category labels, all losses are calculated to construct a training loss function and train the multi-modal model framework; after training, the visual feature extraction module and the classification head are taken out to construct a posterior fossa tumor classification and recognition model; The posterior fossa tumor classification and recognition unit is used to input the posterior fossa MRI image into the posterior fossa tumor classification and recognition model, extract visual features through the visual feature extraction module, and output a predicted category through the classification head.
9. A processing device, characterized by Comprise: One or more processors; Memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-7.
10. A readable storage medium, storing a computer program, characterized in that, When the computer program is executed by the processor, the method of any one of claims 1-7 is implemented.