Tumor identification method based on multi-modal deep learning

Abstract

The invention discloses a tumor identification method based on multi-modal deep learning. The method comprises the following steps: S1, collecting multi-modal data, and respectively forming an image feature matrix and a gene feature matrix; and S2, inputting the image modal feature matrix into a convolutional neural network model encoder to obtain image feature representation, and inputting the gene modal feature matrix into a multistage progressive embedded encoder to obtain gene feature representation. And S3, designing a mode collaborative attention fusion mechanism which comprises a cross-mode attention mechanism and gating residual connection and is used for generating fusion feature representation. And S4, classification prediction: inputting the fused feature representation into a designed classification model, and outputting a corresponding tumor category prediction label. According to the tumor identification method provided by the invention, the accuracy of feature fusion expression and the classification boundary distinguishing capability of the classifier can be improved, and the accuracy and robustness of tumor identification are further improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence-assisted tumor recognition, and in particular to a tumor recognition method based on multimodal deep learning, which is suitable for an automatic diagnosis system that integrates medical imaging and genomic information. Background Art

[0002] At present, tumor recognition methods based on a single modality are difficult to fully characterize the heterogeneous characteristics of tumors, and their accuracy and generalization capabilities are limited. With the rapid development of artificial intelligence technology in the medical field, especially in tumor recognition and diagnosis, researchers have tried to improve the accuracy and robustness of diagnosis by fusing multimodal data. However, existing multimodal tumor recognition methods still have certain limitations in feature extraction, fusion strategies, and classifier design. Multimodal learning methods can effectively fuse complementary information, but they still face challenges in feature alignment, fusion methods, and discrimination capabilities. The present invention provides a structurally innovative method to overcome the above-mentioned defects and improve recognition performance.

[0003] For example, patent CN117352164A proposes an AI-based multimodal tumor detection and diagnosis platform that integrates imaging features and symptom information to detect and diagnose tumors. However, this approach is limited in its ability to deeply fuse multimodal data and extract nonlinear features, potentially limiting the model's expressive power and diagnostic accuracy. Existing technologies in the field of multimodal tumor recognition primarily suffer from the following challenges: limited feature extraction capabilities; a single fusion strategy; and a lack of innovation in classifier design.

[0004] Therefore, there is an urgent need for a new method that can effectively extract nonlinear features in multimodal data, adopt more flexible and in-depth fusion strategies, and design classifiers with stronger discriminative capabilities to improve the accuracy and robustness of tumor recognition. Summary of the Invention

[0005] The purpose of the present invention is to provide a tumor recognition method based on nonlinear structure and integrating multimodal medical data to achieve high-precision, multi-feature collaborative classification decision-making.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] S1: Collect multimodal data to form an image feature matrix and a gene feature matrix respectively.

[0008] S2: Input the image modality feature matrix into a convolutional neural network model encoder to obtain image feature representation, and input the gene modality feature matrix into a multi-level progressive embedded encoder to obtain gene feature representation.

[0009] S3: Design a modality-co-attention fusion mechanism. The fusion module includes a cross-modality attention mechanism and a gated residual connection to generate a fused feature representation.

[0010] S4: Classification prediction: input the fused feature representation into a designed classification model and output the corresponding tumor category prediction label.

[0011] Step S1 specifically includes: constructing the image feature matrix X img ∈R N×p and gene feature matrix X gen ∈R N×g , where N is the number of samples, p and g are the image and gene feature dimensions respectively, and R is the feature matrix set.

[0012] As an embodiment of the present application, step S2 specifically includes:

[0013] S21: Input image modality feature matrix X img and the gene modality feature matrix X gen They are normalized to have a uniform scale and transformed to a fixed dimension through linear mapping:

[0014]

[0015] Where: Norm(·) represents the normalization operation; W img ,W gen is the transformation matrix; b img ,b gen is the bias term.

[0016] S22: Normalized image features Input is a multi-branch convolutional neural network encoder that fuses frequency domain and spatial domain, whose structure consists of two parallel convolution branches.

[0017] S23: The genetic modality encoder is a designed multi-level progressive embedded encoder with the following structure: For the first layer, calculate:

[0018] e1=δ(λ1⊙(Θ1·e0+b1))

[0019] Among them, ⊙ represents the multiplication of corresponding elements; Θ1 is the weight parameter matrix of the first layer; b1 is the bias vector of the first layer; δ(·) is the nonlinear activation function; e1 is the encoding result of the first layer; The initial input is the normalized gene feature; λ1 is the regulatory factor, which is related to the input.

[0020] For the kth layer, 2≤k≤L, use the formula:

[0021] e k=δ(λ k ⊙(Θ k ·e k-1 +b k ))

[0022] Among them, L is the total number of layers, Θ k is the weight parameter matrix of the kth layer; b k is the bias vector of the kth layer; e k is the encoding result of the kth layer; k It is the adjustment factor of each dimension feature and is related to the input.

[0023] In the multi-level progressive structure of gene modality feature encoding, the adjustment factor λ is introduced into the nonlinear mapping of each layer k. k , which is used to adaptively control each dimension of features. Its generation method adopts a multi-network joint modeling mechanism to enhance the expression ability and nonlinear adjustment ability:

[0024]

[0025] in: It is a lightweight perception network based on a fully connected layer; is the global correlation module; is a nonlinear residual enhancement module; δ is a Sigmoid activation function; It represents the cross-module fusion operation and weighted average.

[0026] As an embodiment of the present application, step S22 specifically includes:

[0027] S221: ScaleFusionConv branch uses a self-designed multi-scale feature fusion convolution layer to extract multi-band feature information. The specific formula is as follows:

[0028]

[0029] in, N (s) are low-frequency and high-frequency convolution kernels respectively; is the normalized image feature of the kth scale; n is the total number of scales; It represents weighted fusion.

[0030] S222: mid-pass filter convolution branch, which introduces a mid-pass filtering mechanism to enhance the preservation of intermediate frequency information. The specific operations are as follows:

[0031]

[0032] in, is the normalized image feature, K low ,K mid ,Khigh are low, medium and high frequency convolution kernels respectively; β1 and β2 are learnable adjustment parameters.

[0033] S223: Finally, the output features of the two branches are cascaded to obtain the fusion coding representation of the image modality Φ img .

[0034] As an embodiment of the present application, step S3 specifically includes:

[0035] S31: Attention calculation between the first modality (image modality) and the second modality (gene modality), calculate the query vector of the first modality, and extract the feature representation Φ from the image modality. img The Query vector is calculated in:

[0036]

[0037] Calculate the Key and Value vectors of the second modality: From the feature representation Φ of the gene modality gen Calculate the Key and Value vectors:

[0038]

[0039] Calculate the attention weight between the first modality query and the second modality key, use the dot product attention mechanism to calculate the correlation between the image modality query and the gene modality key, and normalize it:

[0040]

[0041] The weighted summation is used to obtain the cross-modal representation. The calculated attention weights are used to perform weighted summation on the values of the gene modalities to obtain the cross-modal fusion representation:

[0042]

[0043] S32: Reverse attention calculation between the second modality (genetic modality) and the first modality (image modality) to calculate the query vector of the second modality. Similarly, from the feature representation Φ of the genetic modality gen The Query vector is calculated in:

[0044]

[0045] Calculate the Key and Value vectors of the first modality: From the feature representation Φ of the image modality img Calculate the Key and Value vectors:

[0046]

[0047] Calculate the attention weight between the second modality query and the first modality key, calculate the correlation between the genetic modality query and the imaging modality key, and normalize them:

[0048]

[0049] The weighted summation is used to obtain the reverse cross-modal representation. The calculated reverse attention weight is used to perform weighted summation on the value of the image modality to obtain the reverse cross-modal fusion representation:

[0050]

[0051] S33: Fusion of attention outputs through multimodal data feature regulator. To better fuse the attention outputs of the two modalities, a multimodal data feature regulator can be used for weighted fusion to adjust the fusion ratio of the attention outputs of the two modalities:

[0052] R final =γ·(λ·R (I→G) +(1-λ)·R (G→I) )+(1-γ)·η·(R (I→G) ⊙R (G→I) )

[0053] Among them: γ is a regulatory factor that controls the direction of the bidirectional attention fusion; λ is a learnable regulatory factor that determines the ratio between the mapping from the imaging modality to the genetic modality and the mapping from the genetic modality to the imaging modality. This parameter can be automatically adjusted through training to optimize the weight distribution between the modalities; η is a modality correlation parameter that enhances the nonlinear relationship between different modalities and further strengthens R by element-by-element product. (I→G) and R (G→I) The joint features between them can improve the expressiveness of fusion features; (I→G) represents the attention mapping matrix from image modality I to gene modality G; R (G→I) Represents the attention mapping matrix from genetic modality G to image modality I.

[0054] As an embodiment of the present application, step S4 specifically includes:

[0055] The fused multimodal feature vectors are fed into two asymmetric subspace projection branches. The first branch uses Sigmoid activation to obtain sparse response features, and the second branch uses hyperbolic tangent activation function to obtain boundary compression expression. Finally, the outputs of the two branches are concatenated and fed into the classification layer to generate tumor type prediction labels. The structure is as follows:

[0056]

[0057] Among them, R final The multimodal feature representation obtained by the previous fusion module; W1, W2 are weight matrices respectively; b1, b2 are bias function branches corresponding to their respective branches; σ(·) is the inhibitory feature extractor; tanh(·) is the emphasis feature extractor; represents weighted sum; Output layer weights; Output layer bias; is the final classification prediction vector.

[0058] The beneficial effects of the present invention are:

[0059] (1) The present invention enhances the image modality discrimination capability and key feature expression capability by designing a convolutional neural network encoder and a multi-level progressive embedded encoder. The multi-level progressive embedded gene modality encoder proposed in the present invention introduces a regulation factor λ generated by multi-network joint modeling in each encoding layer. k , and adaptively control gene expression data of different dimensions. This design can highlight the expression contribution of highly correlated functional genes or mutation sites, while suppressing the noise dimension, thereby more effectively capturing microscopic molecular information related to tumor development and improving the discriminability and interpretability of features. In the imaging modality encoder, the two branch paths of multi-scale feature fusion convolution and mid-pass filter convolution are integrated, which can extract the texture information, boundary shape and medium-frequency lesion features of the tumor area at different scales and frequency ranges. This design breaks through the local receptive field limitation of traditional convolution kernels and significantly improves the model's perception of early-stage microtumors and complex morphological variations.

[0060] (2) This paper designs a bidirectional cross-modal attention mechanism, which combines a gated adjustment module with a residual fusion structure to fully align the medical imaging modality and the genetic modality at the semantic level. This mechanism can dynamically capture the collaborative dependency between the two modalities and effectively alleviate the distribution differences between the modalities, thereby significantly improving classification accuracy and model robustness in actual tumor recognition tasks.

[0061] (3) The present invention enhances the discriminability of category boundaries by utilizing an innovative dual-branch asymmetric classifier. The classifier proposed in the present invention adopts a dual-branch path: one uses Sigmoid activation to emphasize the sparse sensitive dimension, and the other uses Tanh activation to compress and enhance the boundary gradient information. The outputs of the two branches are spliced and input into the Softmax classification head, which effectively improves the distribution separability of the fused features near the category boundary and solves the problem that the existing Softmax layer is insufficiently responsive to complex nonlinear features. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of the structure of a convolutional neural network model encoder provided in an embodiment of the present invention;

[0063] Figure 2 A schematic diagram of the structure of a multi-stage progressive embedded encoder provided in an embodiment of the present invention;

[0064] Figure 3 A schematic diagram of the structure of a cross-modal attention mechanism provided in an embodiment of the present invention;

[0065] Figure 4 Flowchart of the identification method of the present invention. DETAILED DESCRIPTION

[0066] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0067] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0068] In the present invention, unless otherwise specified or limited, the terms "connection" and "fixation" should be understood in a broad sense. For example, "fixation" can mean fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two elements or interaction between two elements, unless otherwise specified. Those skilled in the art will be able to understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0069] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or a scheme in which A and B are satisfied at the same time. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0070] Reference Figures 1 to 4 In a first aspect, the present invention provides a tumor recognition method based on multimodal deep learning, the method comprising the following steps:

[0071] S1: Collect multimodal data to form an image feature matrix and a gene feature matrix respectively.

[0072] Specifically, the image feature matrix X is formed respectively img ∈R N×p and gene feature matrix X gen ∈R N×g , where N is the number of samples, p and g are the image and gene feature dimensions respectively, and R is the feature matrix set.

[0073] S2: Input the image modality feature matrix into a convolutional neural network model encoder to obtain image feature representation, and input the gene modality feature matrix into a multi-level progressive embedded encoder to obtain gene feature representation.

[0074] Specifically, this step adopts a customized strategy for representation learning by designing a modality-specific encoder structure, targeting the spatial continuity of images and the high-dimensional sparsity of genetic data. The obtained feature representation Φ img , Φ gen They are highly complementary in subsequent cross-modal fusion, which is beneficial to enhancing the accuracy and generalization ability of the tumor recognition model.

[0075] S3: Design a modality-co-attention fusion mechanism, where the fusion module includes a cross-modality attention mechanism and a gated residual connection to generate a fused feature representation.

[0076] Specifically, through the modal collaborative attention fusion mechanism, the present invention can improve the inter-modal alignment capability and semantic interaction quality; enhance the nonlinear expression capability of fused features; adaptively control the information fusion strength; adapt to multiple modal combination structures, and provide more accurate feature representation for subsequent classification.

[0077] S4: Classification prediction: input the fused feature representation into a designed classification model and output the corresponding tumor category prediction label.

[0078] The classification model is an asymmetric two-branch classification model with an innovative structure to output tumor category prediction labels. This classification model comprehensively considers the sparse responsiveness and boundary enhancement of fused features. The fused structure has stronger expressive power, enhanced adaptability to heterogeneous feature dimensions, strong adaptability, and supports end-to-end training optimization.

[0079] This invention uses a multi-branch feature encoder to extract representations of the imaging and genetic modalities separately and introduces an adaptive feature adjustment mechanism based on multi-network generation. During the modality fusion stage, a bidirectional cross-attention mechanism between modalities and a multimodal conditioned residual connection are designed to fuse these representations into a joint feature representation. This fused representation is then input into a nonlinear two-branch classifier, which is then concatenated and fed into the output layer to generate a tumor class prediction. This invention improves the accuracy, robustness, and feature interpretability of tumor recognition and is suitable for multimodal intelligent medical diagnostic systems.

[0080] As an embodiment of the present application, step S2 specifically includes:

[0081] S21: Input image modality feature matrix X img and the gene modality feature matrix X gen They are normalized to have a uniform scale and transformed to a fixed dimension through linear mapping:

[0082]

[0083] Where: Norm(·) represents the normalization operation; W img ,W gen is the transformation matrix; b img ,b gen is the bias term.

[0084] S22: Normalized image features Input is a multi-branch convolutional neural network encoder that fuses frequency domain and spatial domain, whose structure consists of two parallel convolution branches.

[0085] S23: The gene modality encoder is a designed multi-stage progressive embedded encoder, and its structure is as follows:

[0086] For layer 1, calculate:

[0087] e1=δ(λ1⊙(Θ1·e0+B1))

[0088] Among them, ⊙ represents the multiplication of corresponding elements; Θ1 is the weight parameter matrix of the first layer; B1 is the bias vector of the first layer; δ(·) is the nonlinear activation function; e1 is the encoding result of the first layer; The initial input is the normalized gene feature; λ1 is the regulatory factor, which is related to the input.

[0089] For the kth layer, 2≤k≤L, use the formula:

[0090] e k =δ(λ k ⊙(Θ k ·e k-1 +b k ))

[0091] Among them, L is the total number of layers, Θ k is the weight parameter matrix of the kth layer; b k is the bias vector of the kth layer; e k is the encoding result of the kth layer; k It is the adjustment factor of each dimension feature and is related to the input.

[0092] In the multi-level progressive structure of gene modality feature encoding, the adjustment factor λ is introduced into the nonlinear mapping of each layer k. k , which is used to adaptively control each dimension of features. Its generation method adopts a multi-network joint modeling mechanism to enhance the expression ability and nonlinear adjustment ability:

[0093]

[0094] in: It is a lightweight perception network based on a fully connected layer; is the global correlation module; is a nonlinear residual enhancement module; δ is a Sigmoid activation function; It represents the cross-module fusion operation and weighted average.

[0095] Specifically, multimodal data normalization and dimension alignment achieve data alignment in scale, range, and dimension; reduce the impact of inter-modal distribution offsets on subsequent modeling; ensure that encoders of different modalities can share some structures or fusion layers, improving computational efficiency. The multi-branch convolutional structure captures multi-frequency and multi-scale tumor regional characteristics; and solves medical image detection problems such as small lesions and blurred boundaries. The multi-level progressive embedded structure introduces hierarchical control factors, allowing the model to dynamically adjust the weights of features of different dimensions; and solves problems such as high-dimensional sparsity and large redundant interference in gene expression. Multi-level encoding can compress redundancy and purify signals layer by layer; and has strong nonlinear characterization capabilities, suitable for capturing key pathways or mutant gene characteristics.

[0096] As an embodiment of the present application, step S22 specifically includes:

[0097] S221: ScaleFusionConv branch uses a self-designed multi-scale feature fusion convolution layer to extract multi-band feature information. The specific formula is as follows:

[0098]

[0099] in, are low-frequency and high-frequency convolution kernels respectively; is the normalized image feature of the kth scale; n is the total number of scales; It represents weighted fusion.

[0100] S222: mid-pass filter convolution branch, which introduces a mid-pass filtering mechanism to enhance the preservation of intermediate frequency information. The specific operations are as follows:

[0101]

[0102] in, is the normalized image feature, K low ,K mid ,K high are low, medium and high frequency convolution kernels respectively; β1 and β2 are learnable adjustment parameters.

[0103] S223: Finally, the output features of the two branches are cascaded to obtain the fusion coding representation of the image modality Φ img .

[0104] In the present invention, the normalized image modality features are input into a multi-branch convolutional neural network model encoder that fuses the frequency domain and the spatial domain to extract multi-scale discriminant information from the image. Through the collaborative design of ScaleFusion convolution and mid-pass filter convolution, the model can fully perceive the morphology, density and edge changes of the tumor area at multiple levels of low frequency, medium frequency and high frequency, effectively making up for the limitations of a single convolution scale. The medium frequency region usually corresponds to the transition area between tumor tissue and normal tissue. This design effectively enhances the response of this type of "gray area" through an adjustable filter, thereby improving the detection rate of early micro-lesions.

[0105] As an embodiment of the present application, step S3 specifically includes:

[0106] S31: Attention calculation between the first modality (image modality) and the second modality (gene modality), calculate the query vector of the first modality, and extract the feature representation Φ from the image modality. img The Query vector is calculated in:

[0107]

[0108] Calculate the Key and Value vectors of the second modality: From the feature representation Φ of the gene modality gen Calculate the Key and Value vectors:

[0109]

[0110] Calculate the attention weight between the first modality query and the second modality key, use the dot product attention mechanism to calculate the correlation between the image modality query and the gene modality key, and normalize it:

[0111]

[0112] The weighted summation is used to obtain the cross-modal representation. The calculated attention weights are used to perform weighted summation on the values of the gene modalities to obtain the cross-modal fusion representation:

[0113]

[0114] S32: Reverse attention calculation between the second modality (genetic modality) and the first modality (image modality) to calculate the query vector of the second modality. Similarly, from the feature representation Φ of the genetic modality gen The Query vector is calculated in:

[0115]

[0116] Calculate the Key and Value vectors of the first modality: From the feature representation Φ of the image modalityimg Calculate the Key and Value vectors:

[0117]

[0118] Calculate the attention weight between the second modality query and the first modality key, calculate the correlation between the genetic modality query and the imaging modality key, and normalize them:

[0119]

[0120] The weighted summation is used to obtain the reverse cross-modal representation. The calculated reverse attention weight is used to perform weighted summation on the value of the image modality to obtain the reverse cross-modal fusion representation:

[0121]

[0122] S33: Fusion of attention outputs through multimodal data feature regulator. To better fuse the attention outputs of the two modalities, a multimodal data feature regulator can be used for weighted fusion to adjust the fusion ratio of the attention outputs of the two modalities:

[0123] R final =γ·(λ·R (I→G) +(1-λ)·R (G→I) )+(1-γ)·η·(R (I→G) ⊙R (G→I) )

[0124] Among them: γ is a regulatory factor that controls the direction of the bidirectional attention fusion; λ is a learnable regulatory factor that determines the ratio between the mapping from the imaging modality to the genetic modality and the mapping from the genetic modality to the imaging modality. This parameter can be automatically adjusted through training to optimize the weight distribution between the modalities; η is a modality correlation parameter that enhances the nonlinear relationship between different modalities and further strengthens R by element-by-element product. (I→G) and R (G→I) The joint features between them can improve the expressiveness of fusion features; (I→G) represents the attention mapping matrix from image modality I to gene modality G; R (G→I) Represents the attention mapping matrix from genetic modality G to image modality I.

[0125] Specifically, the modality-coordinated attention fusion mechanism achieves information extraction dominated by the imaging modality and image structure regulation dominated by the gene modality, strengthening the selective attention of image regions to gene expression, helping the model capture potential molecular clues from visual structure; it also helps locate key image regions related to specific gene expression, such as the distribution of tumors corresponding to a mutation site. It dynamically adjusts the fusion weights of the two modal directions to avoid the phenomenon of single-modality dominance; nonlinear symmetric interaction strengthens the response of modal commonalities, improving the model's expressiveness of joint features, preserving modality specificity, and strengthening fusion discriminability, providing high-quality input for subsequent classification.

[0126] As an embodiment of the present application, step S4 specifically includes:

[0127] The fused multimodal feature vectors are fed into two asymmetric subspace projection branches. The first branch uses Sigmoid activation to obtain sparse response features, and the second branch uses hyperbolic tangent activation function to obtain boundary compression expression. Finally, the outputs of the two branches are concatenated and fed into the classification layer to generate tumor type prediction labels. The structure is as follows:

[0128]

[0129] Among them, R final The multimodal feature representation obtained by the previous fusion module; W1, W2 are weight matrices respectively; b1, b2 are bias function branches corresponding to their respective branches; σ(·) is the inhibitory feature extractor; tanh(·) is the emphasis feature extractor; represents weighted sum; Output layer weights; Output layer bias; is the final classification prediction vector.

[0130] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A tumor recognition method based on multimodal deep learning, characterized in that: The method comprises the following steps: S1: Collect multimodal data to form image feature matrix and gene feature matrix respectively; S2: Input the image modality feature matrix into a convolutional neural network model encoder to obtain image feature representation, and input the gene modality feature matrix into a multi-level progressive embedded encoder to obtain gene feature representation; S3: Design a modality-co-attention fusion mechanism, where the fusion module includes a cross-modality attention mechanism and a gated residual connection to generate a fused feature representation. S4: Classification prediction: input the fused feature representation into a designed classification model and output the corresponding tumor category prediction label.

2. The characteristic matrix according to claim 1, characterized in that: The step S1 specifically includes: Constitute the image feature matrix X img ∈R N×p and gene feature matrix X gen ∈R N×g , where N is the number of samples, p and g are the image and gene feature dimensions respectively, and R is the feature matrix set.

3. The convolutional neural network model encoder and the multi-stage progressive embedded encoder according to claim 1, characterized in that The step S2 specifically includes: S21: Input image modality feature matrix X img and the gene modality feature matrix X gen They are normalized to have a uniform scale and transformed to a fixed dimension through linear mapping: Where: Norm(·) represents the normalization operation; W img ,W gen is the transformation matrix; b img ,b gen is the bias term; S22: Normalized image features Input a multi-branch convolutional neural network encoder that integrates frequency domain and spatial domain. Its structure consists of two parallel convolution branches. S23: The gene modality encoder is a designed multi-stage progressive embedded encoder, and its structure is as follows: For layer 1, calculate: e1=δ(λ1⊙(Θ1·e0+b1)) Among them, ⊙ represents the multiplication of corresponding elements; Θ1 is the weight parameter matrix of the first layer; b1 is the bias vector of the first layer; δ(·) is the nonlinear activation function; e1 is the encoding result of the first layer; The initial input is the normalized gene feature; λ1 is the regulatory factor, which is related to the input; For the kth layer, 2≤k≤L, use the formula: e k =δ(λ k ⊙(Θ k ·e k-1 +b k )) Among them, L is the total number of layers, Θ k is the weight parameter matrix of the kth layer; b k is the bias vector of the kth layer; e k is the encoding result of the kth layer; k It is the adjustment factor of each dimension feature, which is related to the input; In the multi-level progressive structure of gene modality feature encoding, the adjustment factor λ is introduced into the nonlinear mapping of each layer k. k , which is used to adaptively control each dimension of features. Its generation method adopts a multi-network joint modeling mechanism to enhance the expression ability and nonlinear adjustment ability: in: It is a lightweight perception network based on a fully connected layer; is the global correlation module; is a nonlinear residual enhancement module; δ is a Sigmoid activation function; It represents the cross-module fusion operation and weighted average.

4. The convolutional neural network model encoder according to claim 1, wherein The step S22 specifically includes: S221: ScaleFusionConv branch uses a self-designed multi-scale feature fusion convolution layer to extract multi-band feature information. The specific formula is as follows: in, N (s) are low-frequency and high-frequency convolution kernels respectively; is the normalized image feature of the kth scale; n is the total number of scales; It represents weighted fusion; S222: mid-pass filter convolution branch, which introduces a mid-pass filtering mechanism to enhance the preservation of intermediate frequency information. The specific operations are as follows: in, is the normalized image feature, K low ,K mid ,K high are low, medium and high frequency convolution kernels respectively; β1 and β2 are learnable adjustment parameters; S223: Finally, the output features of the two branches are cascaded to obtain the fusion coding representation of the image modality Φ img .

5. The method according to claim 1, wherein The cross-modal attention mechanism includes: S31: Attention calculation between the first modality (image modality) and the second modality (gene modality), calculate the query vector of the first modality, and extract the feature representation Φ from the image modality. img The Query vector is calculated in: Calculate the Key and Value vectors of the second modality: From the feature representation Φ of the gene modality gen Calculate the Key and Value vectors: Calculate the attention weight between the first modality query and the second modality key, use the dot product attention mechanism to calculate the correlation between the image modality query and the gene modality key, and normalize it: The weighted summation is used to obtain the cross-modal representation. The calculated attention weights are used to perform weighted summation on the values of the gene modalities to obtain the cross-modal fusion representation: S32: Reverse attention calculation between the second modality (genetic modality) and the first modality (image modality) to calculate the query vector of the second modality. Similarly, from the feature representation Φ of the genetic modality gen The Query vector is calculated in: Calculate the Key and Value vectors of the first modality: From the feature representation Φ of the image modality img Calculate the Key and Value vectors: Calculate the attention weight between the second modality query and the first modality key, calculate the correlation between the genetic modality query and the imaging modality key, and normalize them: The weighted summation is used to obtain the reverse cross-modal representation. The calculated reverse attention weight is used to perform weighted summation on the value of the image modality to obtain the reverse cross-modal fusion representation: S33: Fusion of attention outputs through multimodal data feature regulator. To better fuse the attention outputs of the two modalities, a multimodal data feature regulator can be used for weighted fusion to adjust the fusion ratio of the attention outputs of the two modalities: R final =γ·(λ·R (I→G) +(1-λ)·R (G→I) )+(1-c)·h ·(R (I→G) ⊙R (G→I) ) Among them: γ is a regulatory factor that controls the direction of the bidirectional attention fusion; λ is a learnable regulatory factor that determines the ratio between the mapping from the imaging modality to the genetic modality and the mapping from the genetic modality to the imaging modality. This parameter can be automatically adjusted through training to optimize the weight distribution between the modalities; η is a modality correlation parameter that enhances the nonlinear relationship between different modalities and further strengthens R by element-by-element multiplication. (I→G) and R (G→I) The joint features between them can improve the expressiveness of fusion features; (I→G) represents the attention mapping matrix from image modality I to gene modality G; R (G →I) Represents the attention mapping matrix from genetic modality G to image modality I.

6. The method according to claim 1, characterized in that The classification model is a nonlinear two-branch fusion classification model, characterized in that the fused multimodal feature vectors are fed into two asymmetric subspace projection branches respectively. The first branch uses Sigmoid activation to obtain sparse response features, and the second branch uses hyperbolic tangent activation function to obtain boundary compression expression. Finally, the outputs of the two branches are concatenated and fed into the classification layer to generate tumor type prediction labels. Its structure is as follows: Among them, R final The multimodal feature representation obtained by the previous fusion module; W1, W2 are weight matrices respectively; b1, b2 are bias function branches corresponding to their respective branches; σ(·) is the inhibitory feature extractor; tanh(·) is the emphasis feature extractor; represents weighted sum; Output layer weights; Output layer bias; is the final classification prediction vector.

Images