A Method and System for Predicting Thymic Disease Risk Based on Cross-Modal Feature Interaction

Abstract

The application provides a thymus disease risk prediction method and system based on cross-modal feature interaction, which comprises the following steps: respectively preprocessing CT image data, MRI image data and image reports; sequentially performing word segmentation, vector conversion, feature extraction and pooling on the third data, inputting the labeled semantic feature vector into an image generation network to generate a PET-CT image; inputting the first data, the second data and the PET-CT image into the same medical mamba feature extraction network respectively to obtain the first modal feature corresponding to the first data, the second modal feature corresponding to the second data and the third modal feature corresponding to the PET-CT image; performing feature fusion interaction on the first modal feature, the second modal feature and the third modal feature to obtain a fusion feature, and obtaining the thymus disease risk according to the fusion feature. The application can greatly improve the thymus disease prediction accuracy.

Description

Technical Field

[0001] This invention relates to the field of thymic disease risk prediction technology, and in particular to a method and system for predicting thymic disease risk based on cross-modal feature interaction. Background Technology

[0002] In the medical field, accurate prediction of thymus disease risk is of great significance for early intervention and effective treatment. Traditionally, doctors rely on single medical imaging methods to assess the condition of the thymus. While commonly used CT images can clearly show the anatomical structure of the thymus and help detect obvious lesions, they have limitations in soft tissue resolution. MRI images, while excelling in soft tissue resolution and able to clearly show the internal tissues and subtle features of lesions of the thymus, are unable to fully present the overall spatial relationship between the thymus and surrounding tissues.

[0003] Moreover, relying solely on a single imaging examination often fails to fully utilize the rich information contained in different imaging modalities, making it difficult to achieve a comprehensive and accurate assessment of the risk of thymic disease. In addition, the textual information in imaging reports also contains a large amount of semantic content related to thymic diseases, but traditional methods have failed to effectively mine and utilize this information, resulting in a need to improve the accuracy and reliability of thymic disease risk prediction. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for predicting the risk of thymic disease based on cross-modal feature interaction, which aims to solve the problem that the effectiveness of knowledge tracking is low due to the neglect of the important impact of negative learning transfer on knowledge tracking in traditional technologies.

[0005] In a first aspect, the present invention provides a method for predicting the risk of thymic disease based on cross-modal feature interaction, the method comprising:

[0006] Acquire CT image data, MRI image data, and image reports, and preprocess the CT image data, MRI image data, and image reports respectively to obtain first data corresponding to the CT image data, second data corresponding to the MRI image data, and third data corresponding to the image reports;

[0007] The third data is sequentially segmented, vectorized, extracted, and pooled to obtain a semantic feature vector. A PET-CT label is then attached to the semantic feature vector. The labeled semantic feature vector is then input into the image generation network so that the image generation network can generate PET-CT images based on the PET-CT label.

[0008] The first data, the second data, and the PET-CT image are respectively input into the same Medical Mamba feature extraction network to obtain the first modality feature corresponding to the first data, the second modality feature corresponding to the second data, and the third modality feature corresponding to the PET-CT image. The Medical Mamba feature extraction network includes a patch embedding layer and four stages. The first three stages are composed of a hybrid spatial module and a patch merging layer, and the last stage contains only a hybrid spatial module.

[0009] The first modal feature, the second modal feature, and the third modal feature are fused and interacted to obtain a fused feature, and the risk of thymic disease is obtained based on the fused feature.

[0010] Secondly, the present invention provides a thymic disease risk prediction system based on cross-modal feature interaction, the system comprising:

[0011] The modal data acquisition module is used to acquire CT image data, MRI image data, and image reports, and to preprocess the CT image data, MRI image data, and image reports respectively to obtain first data corresponding to the CT image data, second data corresponding to the MRI image data, and third data corresponding to the image reports.

[0012] The image generation module is used to perform word segmentation, vector transformation, feature extraction, and pooling on the third data in sequence to obtain a semantic feature vector, and to label the semantic feature vector with a PET-CT label. The labeled semantic feature vector is then input into the image generation network so that the image generation network can generate PET-CT images based on the PET-CT label.

[0013] The feature extraction module is used to input the first data, the second data, and the PET-CT image into the same Medical Mamba feature extraction network to obtain the first modality feature corresponding to the first data, the second modality feature corresponding to the second data, and the third modality feature corresponding to the PET-CT image. The Medical Mamba feature extraction network includes a patch embedding layer and four stages. The first three stages are composed of a hybrid spatial module and a patch merging layer, and the last stage contains only a hybrid spatial module.

[0014] The feature fusion module is used to perform feature fusion interaction on the first modal feature, the second modal feature, and the third modal feature to obtain fused features, and to obtain the risk of thymus disease based on the fused features.

[0015] Thirdly, the present invention provides a storage medium that stores one or more programs that, when executed by a processor, implement the above-described method for predicting thymic disease risk based on cross-modal feature interaction.

[0016] Fourthly, the present invention provides an electronic device, the electronic device comprising a memory and a processor, wherein:

[0017] The memory is used to store computer programs;

[0018] When the processor executes the computer program stored in the memory, it implements the above-mentioned method for predicting the risk of thymus disease based on cross-modal feature interaction.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] This invention integrates multi-source data and deeply extracts features from various modalities to comprehensively and accurately predict the risk of thymoma and thymic cysts. Specifically, data is collected from multiple data sources, including CT and MRI images and image reports. A text-guided generation and multimodal image fusion model is then built, cleverly cropping adjacent slices along the z-axis to force them to fully utilize the three-dimensional connections between adjacent slices. Considering the limitations of convolutional neural networks in long-range modeling, which prevents them from effectively extracting features from medical images, and the limitations of visual converters due to their secondary computational complexity, a novel Medical Mamba architecture is designed. This architecture combines the local feature extraction capabilities of convolutional layers with the ability to capture long-range dependencies in the state space, thereby modeling medical images of different modalities. Secondly, for image report data, a unified multimodal generation framework is used. The input text is encoded into semantic vectors and guided by preset modality identifiers. Through a diffusion model, guided by the semantic vectors, PET-CT images with typical metabolic or structural features of thymoma and thymic cysts are automatically generated. These images are then also processed by the Medical Mamba architecture for image feature extraction. Finally, a cross-modal feature interaction module is used to achieve deep interactive fusion of data features from three different modalities, enabling accurate and reliable prediction of thymus disease risk. Attached Figure Description

[0021] Figure 1 This is a flowchart of a thymus disease risk prediction method based on cross-modal feature interaction proposed in an embodiment of the present invention;

[0022] Figure 2 This is a framework diagram of a thymus disease risk prediction method based on cross-modal feature interaction;

[0023] Figure 3 This is a schematic diagram of the structure of the Medical Mamba Feature Extraction Network;

[0024] Figure 4 This is a structural schematic diagram of the hybrid space module;

[0025] Figure 5This is a schematic diagram of the structure of a thymus disease risk prediction system based on cross-modal feature interaction proposed in an embodiment of the present invention.

[0026] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art. The terms "comprising" and similar expressions used herein mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, but does not exclude other elements or objects.

[0028] like Figure 1 and Figure 2 As shown, an embodiment of the present invention proposes a method for predicting the risk of thymic disease based on cross-modal feature interaction. The method includes steps S101 to S104, wherein:

[0029] Step S101: Acquire CT image data, MRI image data, and image reports, and preprocess the CT image data, MRI image data, and image reports respectively to obtain first data corresponding to the CT image data, second data corresponding to the MRI image data, and third data corresponding to the image reports;

[0030] It should be noted that the CT image data, MRI image data, and image reports obtained in this step are for the same person to predict their risk of thymic disease, specifically the risk of developing thymoma and thymic cyst.

[0031] During preprocessing, the CT and MRI image data were first normalized and the image size was adjusted to a uniform size of 512×512. Bilateral filtering was used to remove noise from the CT and MRI image data to reduce the negative impact on model training. The window width and window level of the CT and MRI images were set to 350 and 40, respectively, to more clearly display tumor details and cyst edges.

[0032] In addition, the denoised CT image data also needs to be processed. and MRI imaging data Take multiple slices along the Z-axis, with a total number of slices of t, and obtain the data. and The first data is obtained by filling the boundaries of the data. Second data .

[0033] in, , , These represent the first patch, the nth slice, and the Nth slice of the CT image data, respectively. , , These represent the first patch, the nth slice, and the Nth slice of the MRI image data, respectively. w, h, and t represent the width, length, and height of the image data before boundary filling, respectively. W, H, and T represent the width, length, and height of the image data after boundary filling, respectively.

[0034] Regarding the preprocessing of image reports, the image reports are cleaned, which includes removing duplicate content, filtering irrelevant information (such as punctuation marks, redundant words, etc.), correcting spelling errors (such as unifying "mass" and "tumor" as the same term) and grammatical problems, to obtain third-party data.

[0035] In summary, traditional preprocessing of image and text data is often inconsistent, resulting in issues such as noise and inconsistent image sizes, redundancy in text, and inconsistencies in terminology, all of which negatively impact model performance. Normalizing, resizing, and applying bilateral filtering to denoise CT and MRI images, along with image report cleaning, reduces the negative impact of data on the model, providing high-quality data for subsequent model processing and improving prediction reliability.

[0036] Step S102: The third data is sequentially segmented, vectorized, extracted, and pooled to obtain a semantic feature vector. A PET-CT label is then attached to the semantic feature vector. The attached semantic feature vector is then input into the image generation network so that the image generation network generates PET-CT images based on the PET-CT label.

[0037] It should be noted that ClinicalBERT is used to segment the cleaned third-party data, mapping the segmented medical terms to indices in the vocabulary. Then, each word or subword is transformed into a word embedding vector using the BioBERT word embedding method. This high-dimensional word embedding vector effectively captures the semantic information in the medical text. The word embedding vector is then input into the text feature extraction block to capture the contextual information of the text. Through the self-attention mechanism, the meaning of each word in its context can be understood. For example, in the context of the sentence "tumor shows irregular edges," "irregular edges" will be associated with "tumor" to more accurately generate the tumor morphology in the image, obtaining a contextual feature vector. The contextual feature vectors obtained in the previous step are then aggregated in the final layer, using max pooling and average pooling operations to obtain a fixed-length semantic feature vector.

[0038] Furthermore, in some embodiments, regarding PET-CT images, PET-CT tags are introduced and fused with the semantic feature vector to form a cross-modal latent representation. This latent representation is then input into an image generation network to obtain a synthesized PET-CT image. .

[0039] Step S103: Input the first data, the second data, and the PET-CT image into the same Medical Mamba feature extraction network to obtain the first modality feature corresponding to the first data, the second modality feature corresponding to the second data, and the third modality feature corresponding to the PET-CT image. The Medical Mamba feature extraction network includes a patch embedding layer and four stages. The first three stages are composed of a hybrid spatial module and a patch merging layer, and the last stage contains only a hybrid spatial module.

[0040] It should be noted that, in this embodiment, the second modal features are obtained. Third mode features The method is the same as that for the first modality features, only the input data is different. Based on this, let's take obtaining the first modality features as an example:

[0041] like Figure 3 and Figure 4 As shown, the first input data first enters the patch embedding layer. The patch embedding layer first divides the first data into non-overlapping patches, with a patch size of 4×4. The first data is then divided into... Each non-overlapping patch is flattened into a one-dimensional vector, and then this one-dimensional vector is transformed into a linear vector with dimension 1. The first embedding vector.

[0042] In some embodiments, the hybrid state space module is a simple dual-branch module without complex or redundant design. The module input is first divided into two sub-inputs of the same size through a channel separation operation, which are then passed to two sub-modules: the convolution branch and the state space branch.

[0043] In the convolution branch, the first embedding vector is sequentially subjected to block normalization, 2D convolution, ReLU activation, 2D convolution, ReLU activation, and pointwise convolution to obtain the first feature. In the state space branch, there are two sub-branches: The first embedding vector is block normalized to obtain the second feature, which is then transformed through a linear layer and an activation function to obtain the third feature; the first embedding vector is layer normalized to obtain the fourth feature, which is then processed through a linear layer, depthwise separable convolution, SiLU activation function, 2D selective scanning, and normalization to obtain the fifth feature; the third and fifth features are fused element-wise by multiplication, and the fused features are then passed through a linear layer to complete feature mixing, resulting in a feature map. ;

[0044] In addition, the patch merging layer will feature map The feature map is divided into multiple first patches using a 2×2 window. The features of each 2×2 first patch are concatenated to obtain a first vector, which is a 4C-dimensional vector (C being the number of channels in the original feature map). Then, a linear layer is used to reduce the dimensionality of the first vector to 2C. A 1x1 convolutional layer is then used for further dimensionality reduction of the number of channels. Ultimately, the size of the output feature map is halved, and the number of channels is doubled, resulting in the first modality feature map. .

[0045] In addition, each of the four stages mentioned above has [C, 2C, 4C, 8C] channels.

[0046] In summary, traditional feature extraction methods struggle to effectively extract local and long-range features from CT and MRI images, impacting subsequent diagnosis. Often, radiomics features are extracted, neglecting contextual information within the image. Therefore, this paper employs two similar Medical Mamba branches to extract CT and MRI features separately. The network comprises four stages, utilizing patch embedding layers and hybrid spatial module states to fully consider semantic relationships within the image context, preserving key feature information. Patch merging layers reduce the size and dimensionality of image features, lowering computational resource consumption. Furthermore, addressing the challenges of effectively extracting information from image reports for supplementing image data, lacking the ability to generate high-quality supplementary images, and the inconsistent data parameters in image reports, making it difficult to process and select important feature parameters, this paper cleans the image report text, uses ClinicalBERT word segmentation and BioBERT word embedding to obtain semantic vectors, and combines these with target modality information to generate synthetic PET / CT images through a diffusion model. Medical Mamba then extracts features, solving the problem of insufficient PET / CT data for most patients, further supplementing metabolic and structural features of lesions, and effectively improving prediction accuracy.

[0047] Step S104: Perform feature fusion interaction on the first modality feature, the second modality feature, and the third modality feature to obtain fused features, and obtain the risk of thymic disease based on the fused features.

[0048] It should be noted that, in some embodiments, the specific fusion process is as follows:

[0049] Modal features are extracted using global pooling and max pooling:

[0050] ;

[0051] in, , These are channel, spatial, and coordinate feature information, respectively. For average pooling operation, For max pooling operation, For connection operations, Pooling features in CT images Pooling features in MRI images For image report data pooling features;

[0052] Aggregate the obtained features:

[0053] ;

[0054] in, This is a channel aggregation feature. It is a spatial aggregation feature. Coordinate aggregation features;

[0055] Convolutionalize the aggregated features:

[0056] ;

[0057] in, As a channel feature, For spatial features, As coordinate features, For one-dimensional convolution, It is a two-dimensional convolution. Aggregate weights for channels. Spatial aggregation weights, Aggregate weights for coordinates. For activation functions;

[0058] Cross-modal interaction of weights:

[0059] ;

[0060] in, For channel space mixing weights, For channel coordinates, mixed weights, Mixed weights for spatial coordinates;

[0061] Multiply the weights after the interaction by the features of each modality:

[0062] ;

[0063] in, Mixed features of CT images Mixed features of MRI images Mixed features for image reports;

[0064] Will , , Adding them together yields the fusion characteristics. .

[0065] In summary, traditional multimodal fusion methods lack deep interaction and cannot fully leverage the synergistic effect of data from different modalities. Therefore, this paper uses three branches to extract the channel, spatial, and coordinate information of each modality, obtains weights through pooling and convolution, normalizes them using Softmax, and then performs cross-modal interaction to fuse the final features. This achieves deep interactive fusion of features from three modalities, while automatically calculating the influence weight of each modality on the final prediction, strengthening key features and improving fusion accuracy.

[0066] Furthermore, in some embodiments, to obtain the final thymic disease risk, the fused features need to be input into a fully connected layer and an activation function to obtain the final thymic disease risk.

[0067] ;

[0068] Where y represents the risk of thymic disease. For activation function, It is a fully connected layer. This is a feature of fusion.

[0069] In summary, traditional risk prediction is based on single or insufficiently integrated features, resulting in low prediction accuracy. By inputting the integrated features into a fully connected layer and activation function to obtain the final risk level, the accuracy of thymoma and thymic cyst risk prediction is improved.

[0070] Furthermore, it should be noted that this embodiment is described from the perspective of a pre-built prediction model. The model building process is basically the same as the solution in this embodiment. The difference is that multiple CT image data, MRI image data and image reports of a known population are obtained. At the same time, before inputting them into the model for training, the risk of thymic disease needs to be manually labeled for each fusion feature obtained from historical data, specifically including the risk of thymoma and thymic cyst, so as to complete the training of the prediction model.

[0071] For example, the risk of thymic disease is generally expressed as a percentage. If the risk of thymoma and thymic cyst is less than 50%, it is considered low risk. If the risk of thymoma and thymic cyst is greater than or equal to 50%, it is considered high risk.

[0072] In summary, this invention integrates multi-source data and deeply extracts features from various modalities to comprehensively and accurately predict the risk of thymoma and thymic cysts. Specifically, data is collected from multiple data sources, including CT and MRI images and image reports. A text-guided generation and multimodal image fusion model is then built, cleverly cropping adjacent slices along the z-axis to force the slices to fully utilize the three-dimensional connections between adjacent slices. Considering the limitations of convolutional neural networks in long-range modeling, which prevents them from effectively extracting features from medical images, and the limitations of visual converters due to their secondary computational complexity, a novel Medical Mamba architecture is designed. This architecture combines the local feature extraction capabilities of convolutional layers with the ability to capture long-range dependencies in the state space, thereby modeling medical images of different modalities. Secondly, for image report data, a unified multimodal generation framework is used. The input text is encoded into semantic vectors and guided by preset modality identifiers. Through a diffusion model, guided by the semantic vectors, PET-CT images with typical metabolic or structural features of thymoma and thymic cysts are automatically generated. These images are then also processed by the Medical Mamba architecture for image feature extraction. Finally, a cross-modal feature interaction module is used to achieve deep interactive fusion of data features from three different modalities, enabling accurate and reliable prediction of thymus disease risk.

[0073] like Figure 5 As shown, one embodiment of the present invention proposes a thymic disease risk prediction system based on cross-modal feature interaction, the system comprising:

[0074] The modal data acquisition module 10 is used to acquire CT image data, MRI image data and image reports, and preprocess the CT image data, MRI image data and image reports respectively to obtain first data corresponding to the CT image data, second data corresponding to the MRI image data and third data corresponding to the image reports.

[0075] The image generation module 20 is used to perform word segmentation, vector transformation, feature extraction, and pooling on the third data in sequence to obtain a semantic feature vector, and to label the semantic feature vector with a PET-CT label. The labeled semantic feature vector is then input into the image generation network so that the image generation network generates PET-CT images based on the PET-CT label.

[0076] The feature extraction module 30 is used to input the first data, the second data, and the PET-CT image into the same Medical Mamba feature extraction network to obtain the first modality feature corresponding to the first data, the second modality feature corresponding to the second data, and the third modality feature corresponding to the PET-CT image. The Medical Mamba feature extraction network includes a patch embedding layer and four stages. The first three stages are composed of a hybrid spatial module and a patch merging layer, and the last stage contains only a hybrid spatial module.

[0077] The feature fusion module 40 is used to perform feature fusion interaction on the first modal feature, the second modal feature, and the third modal feature to obtain fused features, and to obtain the risk of thymus disease based on the fused features.

[0078] In another aspect, the present invention also proposes a storage medium on which one or more programs are stored, which, when executed by a processor, implement the above-described method for predicting the risk of thymic disease based on cross-modal feature interaction.

[0079] In another aspect, the present invention also proposes an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to realize the above-mentioned method for predicting the risk of thymus disease based on cross-modal feature interaction.

[0080] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain stored, communicated, propagated, or transmitted programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0081] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0082] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0083] While embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations fall within the scope and spirit of the invention as set forth in the claims. Furthermore, the invention described herein may have other embodiments and can be implemented or carried out in various ways.

Claims

1. A method for predicting a thymus disease risk based on cross-modal feature interaction, characterized in that, The method comprises: obtaining CT image data, MRI image data and image reports, and preprocessing the CT image data, the MRI image data and the image reports respectively to obtain first data corresponding to the CT image data, second data corresponding to the MRI image data and third data corresponding to the image reports; performing word segmentation, vector conversion, feature extraction and pooling on the third data in sequence to obtain a semantic feature vector, and labeling the semantic feature vector with a PET-CT label, and inputting the labeled semantic feature vector into an image generation network to enable the image generation network to generate a PET-CT image according to the PET-CT label; inputting the first data, the second data and the PET-CT image into the same medical Mamba feature extraction network to obtain first modality features corresponding to the first data, second modality features corresponding to the second data and third modality features corresponding to the PET-CT image, the medical Mamba feature extraction network comprising a patch embedding layer and four stages, the first three stages each consisting of a hybrid space module and a patch merging layer, and the last stage only comprising a hybrid space module; the patch embedding layer first divides the first data into non-overlapping patches, and then flattens each non-overlapping patch into a one-dimensional vector, and then converts the one-dimensional vector into a first embedding vector with a linear layer; the hybrid space module comprises a convolution branch and a state space branch, in the convolution branch, performing block normalization, 2D convolution, ReLU activation, 2D convolution, ReLU activation and point-wise convolution on the first embedding vector in sequence to obtain first features; In the state space branch, two sub-branches are divided, specifically: the first embedding vector is subjected to block normalization to obtain a second feature, and then the second feature is converted through a linear layer and an activation function to obtain a third feature; the first embedding vector is subjected to layer normalization to obtain a fourth feature, and then the fourth feature is subjected to a linear layer, a depth separable convolution, a SiLU activation function, a 2D selective scan, and normalization to obtain a fifth feature; the third feature and the fifth feature are subjected to element-wise multiplication fusion, and the fused feature is subjected to a linear layer to complete feature mixing to obtain a feature map ; the patch merging layer splits the feature map into multiple first patches, splices the features of each first patch to obtain a first vector, and then reduces the dimension of the first vector through a linear layer to obtain a first modal feature ; the second modal feature , the third modal feature are obtained in the same way as the first modal feature. performing feature fusion and interaction on the first modality features, the second modality features and the third modality features to obtain fusion features, and obtaining a thymus disease risk according to the fusion features; extracting modality features using global pooling and max pooling: ； wherein, , are channel, space and coordinate feature information, respectively, is an average pooling operation, is a max pooling operation, is a concatenation operation, is a CT image pooled feature, is an MRI image pooled feature, is an image report data pooled feature; aggregating the obtained features: ； wherein, is a channel aggregation feature, is a spatial aggregation feature, is a coordinate aggregation feature; convolving the aggregated features: ； wherein, is a channel feature, is a spatial feature, is a coordinate feature, is a one-dimensional convolution, is a two-dimensional convolution, is a channel aggregation weight, is a spatial aggregation weight, is a coordinate aggregation weight, is an activation function; cross-modality interaction of weights: ； wherein, is a channel space mixing weight, is a channel coordinate mixing weight, is a spatial coordinate mixing weight; multiplying the interacted weights and the features of each modality: ； wherein, is a CT image hybrid feature, is an MRI image hybrid feature, is an image report hybrid feature; Adding , , together, the fusion features are obtained.

2. The method of claim 1, wherein, the steps of obtaining CT image data, MRI image data and image reports, and preprocessing the CT image data, the MRI image data and the image reports respectively comprise: normalizing the CT image data and the MRI image data, and removing noise in the CT image data and the MRI image data using bilateral filtering; The CT image data after denoising and the MRI image data A plurality of slices are taken along the Z axis, and the number of slices is t, to obtain data and Boundary padding is performed on the data to obtain first data and second data ; wherein, , , are respectively the first slice, the nth slice, the Nth slice of CT image data, , , are respectively the first slice, the nth slice, the Nth slice of MRI image data, w, h, t are respectively the width, length, height before the boundary padding of image data, W, H, T are respectively the width, length, height after the boundary padding of image data. cleaning the image reports, the cleaning including removing repetitive content, filtering irrelevant information, correcting spelling errors and grammar problems to obtain the third data.

3. The method of claim 2, wherein, the steps of performing word segmentation, vector conversion, feature extraction and pooling on the third data in sequence to obtain a semantic feature vector comprise: performing word segmentation on the third data, and converting each word obtained by the word segmentation into a word embedding vector by a BioBERT word embedding method; inputting the word embedding vector into a text feature extraction block to capture context information of the text to obtain a context feature vector; The semantic feature vectors are summarized and obtained by maximum pooling and average pooling operations.

4. The method of claim 3, wherein, The step of labeling the semantic feature vectors with a PET-CT label and inputting the labeled semantic feature vectors into an image generation network to enable the image generation network to generate a PET-CT image according to the PET-CT label comprises: The PET-CT label is introduced and fused with the semantic feature vector to form a cross-modal latent expression, and the latent expression is input into an image generation network to obtain a synthesized PET-CT image .

5. The method of claim 4, wherein, The step of obtaining the thymus disease risk according to the fusion feature comprises: The fusion feature is input into a full connection layer and an activation function to obtain a final thymus disease risk: ； wherein y is a risk of thymus disease, is an activation function, is a fully connected layer, is a fusion feature.

6. A system for implementing the method of predicting the risk of thymus disease based on cross-modal feature interaction according to any one of claims 1-5, characterized in that, The system comprises: A modality data acquisition module is configured to acquire CT image data, MRI image data, and an image report, and to pre-process the CT image data, the MRI image data, and the image report to obtain first data corresponding to the CT image data, second data corresponding to the MRI image data, and third data corresponding to the image report. An image generation module is configured to sequentially perform word segmentation, vector conversion, feature extraction, and pooling on the third data to obtain semantic feature vectors, label the semantic feature vectors with a PET-CT label, and input the labeled semantic feature vectors into an image generation network to enable the image generation network to generate a PET-CT image according to the PET-CT label. A feature extraction module is configured to input the first data, the second data, and the PET-CT image into the same medical Mamba feature extraction network to obtain first modality features corresponding to the first data, second modality features corresponding to the second data, and third modality features corresponding to the PET-CT image. A feature fusion module is configured to perform feature fusion interaction on the first modality features, the second modality features, and the third modality features to obtain fusion features, and to obtain a thymus disease risk according to the fusion features.

7. A storage medium, characterized by The storage medium stores one or more programs that are executed by the processor to implement the thymus disease risk prediction method based on cross-modality feature interaction according to any one of claims 1-5.

8. An electronic device, comprising a memory and a processor, wherein: The memory is configured to store a computer program; The processor is configured to execute the computer program stored on the memory to implement the thymus disease risk prediction method based on cross-modality feature interaction according to any one of claims 1-5.

8. An electronic device, comprising a memory and a processor, wherein: The memory is configured to store a computer program; The processor is configured to execute the computer program stored on the memory to implement the thymus disease risk prediction method based on cross-modality feature interaction according to any one of claims 1-5.

Images