A multimodal tumor data fusion method and device

Abstract

The present invention discloses a multimodal tumor data fusion method and device. The method includes: extracting a first morphological feature from the target's imaging data; extracting a second morphological feature and a structural feature from the target's pathological data; extracting a cluster feature from the target's genetic data; fusing the first morphological feature, the second morphological feature, the structural feature, and the cluster feature through a fusion network, wherein the fusion network includes a first-layer network and a second-layer network, and the first-layer network fuses the first morphological feature with the second morphological feature, the first morphological feature with the structural feature, the second morphological feature with the cluster feature, and the structural feature with the cluster feature in pairs from microscopic to macroscopic changes; the second-layer network fuses the pairwise fusion features to obtain a first fusion feature and a second fusion feature; and classifying or predicting the target using the first fusion feature and the second fusion feature. The present invention realizes comprehensive analysis of multimodal data and provides auxiliary reference for clinical practice.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and more specifically, to a multimodal tumor data fusion method and device. Background Art

[0002] Cancer is a complex disease with significant heterogeneity, characterized by varying macroscopic manifestations and cellular morphology across patients, at different stages of disease progression, and within different tissues and organs. To accurately diagnose cancer and design treatment plans, patients undergo a variety of clinical examinations, such as imaging, pathology, and genome sequencing. This data, combined with the patient's clinical manifestations, allows doctors to analyze various aspects of cancer.

[0003] Imaging examination is one of the most commonly used examination methods in clinical practice, including X-rays, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound imaging, positron emission tomography (PET), etc. The combination of structural and functional imaging can intuitively reflect the condition of the lesion. Pathological examination is the gold standard for cancer diagnosis, and it determines the benign or malignant nature and subtype of the tumor through direct observation of the tissue under a microscope. Clinical genomic testing can obtain the patient's DNA molecular information, such as transcriptome sequencing information (RNA-Seq), copy number variation (CNV), methylation ratio, etc., reflecting the molecular level of the lesion. With the rapid development of computers and information technology, the data obtained from clinical examinations can be stored digitally, which is a prerequisite for realizing computer intelligent analysis of data.

[0004] The data obtained from clinical examinations is diverse, and the analysis process becomes complex, often requiring collaboration across multiple departments or institutions. For example, the acquisition and diagnosis of radiomics data requires the participation of the imaging department, the observation and diagnosis of pathological sections requires the participation of the pathology department, and genomic data requires analysis by professional bioinformatics researchers. The final diagnosis and treatment plan design must be completed by the patient's attending physician. Data analysis and communication between all parties are arduous tasks, requiring high professional expertise in each part, and some links are inevitably influenced by subjective experience, which reduces work efficiency and analytical accuracy. Therefore, cancer diagnosis and treatment have a strong demand for objective and efficient data analysis methods.

[0005] In recent years, with technological advancements and growing demand, intelligent data analysis in the medical field has gradually become a research hotspot. Compared to traditional decentralized data analysis methods, intelligent multimodal data analysis methods extract data features based on the characteristics of different data and then integrate them for analysis. This, on the one hand, avoids the inefficiency caused by communication between multiple departments and institutions, and on the other hand, avoids the influence of physicians' subjective experience on the analysis. Compared with existing computer intelligent data analysis methods, traditional algorithms only use single-modal data or multimodal imaging data for analysis, and rarely combine imaging with genetic, macroscopic and microscopic, or epigenetic and molecular data for joint multimodal analysis. The fusion methods used by traditional algorithms are also relatively simple, usually direct feature splicing or majority voting decisions, which makes it difficult to model the potential connections between multimodal data and rarely analyzes redundant and complementary information between modalities.

[0006] Currently, most computer-based intelligent analysis research on cancer data is still focused on single-modal data such as imaging or genes. Few research results have combined multiple data types, including structural images, functional images, genomic data, pathological images, and clinical information. Using single-modal data can only obtain relatively partial information, so it is difficult to achieve significant breakthroughs in diagnostic and prognostic accuracy. Using multimodal data to form a more complete representation of the object from different aspects is a feasible approach. However, existing research on multimodal data fusion analysis faces the following difficulties:

[0007] 1) The data types are diverse and the distribution is complex, which requires computer-aided integrated analysis of the data. Different modalities need to be processed separately according to their characteristics, and a subspace representation that can retain their specificity and facilitate joint coordination needs to be found.

[0008] 2) The connections between modal data are unknown. Some connections have been explored in biology, while others are still unclear. This makes it difficult to integrate and analyze the characteristics of data from each modality. Establishing a mathematical model of the connections between modalities through computational methods is a major challenge.

[0009] 3) The information contained in digital pathology images is difficult to fully utilize. For example, the features extracted by traditional machine learning or deep learning methods often lack structural information. They only extract morphological features or pixel features without representing structural features such as cell distribution.

[0010] In summary, the existence of the above issues poses a significant challenge to the implementation of multimodal intelligent analysis. Existing research primarily extracts unilateral features from data from each modality, making it difficult to form a complete description of the data. Furthermore, the multimodal feature fusion methods used are relatively shallow, such as feature concatenation, tensor multiplication, and multi-model majority voting or weighted decision-making. These methods cannot effectively construct the underlying connections between features from each modality. Summary of the Invention

[0011] The purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a multimodal tumor data fusion method and device.

[0012] According to a first aspect of the present invention, a multimodal tumor data fusion method is provided. The method comprises the following steps:

[0013] Extracting a first morphological feature from the target image data, where the first morphological feature reflects the overall geometric morphological representation of the diseased tissue;

[0014] Extracting secondary morphological features and structural features from the target pathological data. The secondary morphological features describe the degree of abnormality of the tumor cell nucleus, and the structural features describe the distribution characteristics of the cells.

[0015] Extract cluster features based on the target gene data, which reflect the expression characteristics of the tumor at the gene molecular level;

[0016] The first morphological feature, the second morphological feature, the structural feature, and the cluster feature are fused through a trained fusion network, wherein the fusion network includes a first-layer network and a second-layer network, and the first-layer network fuses the first morphological feature with the second morphological feature, the first morphological feature with the structural feature, the second morphological feature with the cluster feature, and the structural feature with the cluster feature in pairs based on changes from micro to macro; the second-layer network further fuses the pairwise fused features to obtain the first fused feature and the second fused feature;

[0017] The first fusion feature and the second fusion feature are used to classify and identify the target or perform regression prediction.

[0018] According to a second aspect of the present invention, a multimodal tumor data fusion device is provided. The device comprises:

[0019] Feature extraction module: This module is used to extract the first morphological feature from the target's image data. The first morphological feature reflects the overall geometric morphology of the diseased tissue. From the target's pathological data, it extracts the second morphological feature and structural feature. The second morphological feature describes the degree of abnormality of the tumor cell nucleus, and the structural feature describes the distribution characteristics of the cells. From the target's genetic data, it extracts the cluster feature, which reflects the expression characteristics of tumor genes at the molecular level.

[0020] Feature fusion module: used to fuse the first morphological feature, the second morphological feature, the structural feature and the cluster feature through a trained fusion network, wherein the fusion network includes a first-layer network and a second-layer network. The first-layer network fuses the first morphological feature with the second morphological feature, the first morphological feature with the structural feature, the second morphological feature with the cluster feature, and the structural feature with the cluster feature in pairs based on the changes from micro to macro. The second-layer network further fuses the pairwise fused features to obtain the first fused feature and the second fused feature.

[0021] Prediction module: used to classify and identify or perform regression prediction on the target using the first fusion feature and the second fusion feature.

[0022] Compared with the existing technology, the advantage of the present invention lies in the design of an algorithm for processing multimodal tumor data and realizing joint analysis, using machine learning and deep learning to extract the features of each modality data and perform multi-level deep fusion, and applying the fused features to tasks such as diagnosis and prognosis to achieve comprehensive analysis of multimodal data and provide auxiliary reference for clinical practice.

[0023] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0025] Figure 1 is a flowchart of a multimodal tumor data fusion method according to an embodiment of the present invention;

[0026] Figure 2 is a process diagram of a multimodal tumor data fusion method according to an embodiment of the present invention;

[0027] Figure 3 FIG. 4 is a schematic diagram of a multi-level fusion process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0029] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0030] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0031] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0032] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0033] Combine Figure 1 and Figure 2 As shown, the provided multimodal tumor data fusion method includes the following steps.

[0034] In step S110 , multimodal features at the microscopic and macroscopic levels are extracted for the tumor data.

[0035] The present invention as a whole includes feature extraction and feature fusion. Feature extraction includes imaging feature extraction, gene data (RNA expression matrix) feature extraction, pathological image morphological feature extraction and structural feature extraction, etc., which are used to extract feature data of different modalities.

[0036] For example, when extracting features from image data, the collected images are first preprocessed based on their characteristics, including image resampling, window width and level adjustment, image normalization, and image denoising. Next, a machine learning segmentation algorithm is used to identify the tumor region of interest within the image. Finally, features such as the tumor's shape, size, and spatial distribution are extracted from the region of interest. These extracted image features represent the tumor at the organ level, reflecting the overall geometric morphology of the diseased tissue.

[0037] Pathology image feature extraction involves both morphological and structural feature extraction. First, the digital pathology panoramic image is preprocessed, cutting the large panoramic image into smaller image blocks. Next, color enhancement and equalization, image standardization, and other techniques are performed. Furthermore, the K-means clustering algorithm is used to cluster the small image blocks. The core of K-means clustering is to minimize the variance of samples within each class. The intra-class variance is calculated as:

[0038]

[0039] in, represents the pixel value at the coordinate position (i, j, c) of the k-th image x, (i, j) is the length and width coordinates of the image, c is the color channel coordinate of the image, n represents the n-th cluster category, u nRepresents the pixel value at (i, j, c) of the cluster center image of category n.

[0040] The small image blocks at the center of each cluster are selected as the image blocks that best represent the original pathological image for the next step of processing.

[0041] The cell segmentation algorithm is used to extract cells from small image blocks and extract the morphological characteristics of the cells, such as the size, density, texture, and roundness of the cell nucleus. Each cell nucleus is then used as a node, and the distance between each node is obtained through the K-nearest neighbor algorithm. A distance threshold is set to determine whether the nodes are in a neighboring relationship. If a neighboring relationship exists, the distance is the weight value of the edge between the nodes, thereby obtaining the weighted adjacency matrix A, which is as follows:

[0042]

[0043] Where i and j are different nodes, KNN(i) is the set of neighboring nodes of node i divided according to the k value set by the K-nearest neighbor algorithm, D(i,j) is the Euclidean distance between two nodes, and t is the set distance threshold.

[0044] Through the above process, a structural graph of each small image block is constructed. Then, spatial feature description information such as vertex count, edge length, number of edges, K-walks number, and Wiener index is extracted. Features extracted from pathological images represent the tumor at the cellular level. Morphological features can describe the degree of abnormality in tumor cell nuclei, while structural features can describe the distribution of cells, particularly the topological properties of cell adjacency within the lesion. Combining these two types of features provides a more comprehensive description of the lesion at the microscopic level.

[0045] Gene data uses RNA-seq expression data. In one embodiment, feature extraction of gene data includes: first, data preprocessing, including deletion of missing values, completion, expression normalization, difference screening, etc. Weighted gene co-expression network analysis (Zhang, B. & Horvath, S. (2005). A General Framework for Weighted Gene Co-Expression Network Analysis. Statistical Applications in Genetics and Molecular Biology, 4 (1). https: / / doi.org / 10.2202 / 1544-6115.1128) is used to extract features. Specifically, it includes: clustering genes into multiple cluster modules (Modules) according to gene expression, and the gene expression patterns in each cluster module are similar; selecting modules with obvious expression differences, and using the differential gene expression in the module as the extracted gene features. Through this method, the expression characteristics of the sample at the gene molecular level can be obtained.

[0046] It should be noted that other methods can also be used to extract features of each modal data. For example, for images, a convolutional neural network can be used for fully automatic extraction. The present invention does not limit the method of extracting various features.

[0047] Step S120: performing feature selection and feature dimensionality reduction on the extracted multimodal features.

[0048] The high-dimensional features extracted from each modality require feature screening and dimensionality reduction, followed by dimensionality unification before fusion. For example, feature embedding selection methods are first used to filter out highly relevant features and remove redundant information. Principal component analysis (PCA) is then used to reduce the dimensionality of these filtered features and unify the feature dimensions of each modality.

[0049] Specifically, the PCA algorithm includes the following steps:

[0050] Step S121, calculate the mean of N sample vectors x (D×M dimensions)

[0051] Step S122, calculate the variance S of the sample vector:

[0052] Step S123: Decompose the variance matrix to obtain a mapping matrix U consisting of eigenvalues ​​and corresponding eigenvectors. D×K , K is the number of eigenvectors;

[0053] Step S124, calculate the sample vector after dimensionality reduction: The sample vector is reduced from D dimension to K dimension, K<D.

[0054] Step S130 , based on the change from micro to macro, the features after dimensionality reduction are fused pairwise.

[0055] After obtaining the morphological features of imaging, the morphological and structural features of pathological data, and the cluster features of genetic data, the four types of features are fused through a two-layer fusion network. For example, in the order of changes from microscopic to macroscopic, that is, gene cluster features to pathological morphological and structural features and then to imaging morphological features, the first layer of the network fuses each type of feature in pairs. The fused features can be used to explore the potential connection between pathology and imaging, and pathology and genes. In one embodiment, the first layer of the network performs a Kronecker product between each pair of features, and the formula is as follows:

[0056]

[0057] Where A and B are matrices of dimensions m×n and p×q respectively.

[0058] See also Figure 3 As shown in the figure, four two-modal fusion features can be obtained through the first layer of the network, and the fusion is completed through the neural network. Furthermore, the fusion features obtained in the first layer are fused through two neural network paths. To avoid information redundancy, the fusion features with complementary information are paired and input into one path, so that the input information of each path contains multimodal information of images, pathological morphology, pathological structure and genes. Figure 3 In the embodiment, the neural network on each path includes four layers (one input layer, two fully connected layers as hidden layers, and one output layer), and adopts a neuron dropout strategy (dropout) and a linear rectifier activation function (ReLU) to reduce the dimension of the fusion features layer by layer, and finally obtain the fusion features from the two paths respectively.

[0059] In summary, the feature fusion method provided in step S130 takes into account at least the following factors: fusion of features of different modalities; different modal features include macroscopic and microscopic features; the feature fusion process takes into account the changes from microscopic features to macroscopic features. By designing medical multimodal data analysis for tumors, multimodal data in medicine can be effectively utilized to complete tasks such as diagnosis and prognosis. The multimodal data used include three major categories of data, from macroscopic to mesoscopic to microscopic medical imaging, pathological images, and genomics, covering a wide range of data. The proposed multi-level fusion method of multimodal data features, the fusion features at different levels can be used to analyze the potential connections between different modalities through visualization and other methods.

[0060] It should be noted that in feature fusion, other fusion methods can be used at different levels, such as using other tensor product calculations instead of Kronecker products, or using other machine learning models instead of neural network fusion and dimensionality reduction.

[0061] Step S140: Using the fusion features to achieve the target task of tumor data analysis.

[0062] After further integrating the two fusion features, target tasks such as classification and regression can be performed.

[0063] For example, for a classification task, such as a binary classification task to determine tumor subtypes, the loss function is:

[0064]

[0065] Where N represents the total number of samples, y n For sample x n The true label, Represents sample x n The predicted probability of x n is the feature vector obtained by the first layer fusion.

[0066] For regression tasks, such as predicting sample survival time, the loss function is:

[0067]

[0068] Among them, t i and t j are the i-th and j-th time nodes respectively, R(t i ) is the time node t i The risk set at time δ i is the data censoring mark, δ i =0,1.

[0069] The neural network is trained using the loss function for the target task above until convergence, obtaining the weights of each layer of the neural network. The model can then be applied to new data to achieve diagnosis or prognosis prediction.

[0070] Accordingly, the present invention also provides a multimodal tumor data fusion device for implementing one or more aspects of the above-mentioned method. For example, the device includes: a feature extraction module for extracting a first morphological feature from the target's image data; a second morphological feature and a structural feature from the target's pathological data, wherein the second morphological feature describes the degree of abnormality of the tumor cell nucleus and the structural feature describes the distribution characteristics of the cells; a cluster feature from the target's genetic data, wherein the cluster feature reflects the expression characteristics of the tumor at the genetic molecular level; a feature fusion module for fusing the first morphological feature, the second morphological feature, the structural feature, and the cluster feature through a trained fusion network, wherein the fusion network includes a first layer network and a second layer network, wherein the first layer network performs pairwise fusion of the first morphological feature with the second morphological feature, the first morphological feature with the structural feature, the second morphological feature with the cluster feature, and the structural feature with the cluster feature based on changes from microscopic to macroscopic; a second layer network further fuses the pairwise fusion features to obtain a first fusion feature and a second fusion feature; and a prediction module for using the first fusion feature and the second fusion feature to perform classification recognition or regression prediction on the target. Each module in the provided device can be implemented using a processor, FPGA or other dedicated hardware.

[0071] In summary, the present invention addresses the high heterogeneity of malignant tumors by utilizing multimodal data to provide a more comprehensive description of malignant tumors from different perspectives. Based on the characteristic representations of each modality, it computationally analyzes and analyzes patient tumor subtypes, performs prognostic predictions, and explores potential connections between the macroscopic and microscopic, molecular and phenotypic, addressing the challenges of combined analysis and application of multimodal data. Compared to existing technologies, the present invention offers at least the following advantages:

[0072] 1) Existing methods mostly focus on the fusion and analysis of two modal data. The present invention uses more modal data and extracts four types of features for fusion analysis, so the information contained in the data is richer.

[0073] 2) The present invention extracts morphological features and structural features from pathological image data respectively, and effectively utilizes the information contained in the data.

[0074] 3) The multi-level fusion proposed in this invention realizes the fusion representation of four types of features on the one hand, and on the other hand, the fusion features at different levels can be used to analyze the potential connections between different modalities.

[0075] 4) The present invention quantifies and streamlines the complex clinical multimodal diagnostic process, providing auxiliary reference for clinical diagnosis and prognosis.

[0076] 5) On the one hand, the present invention has important clinical value in realizing intelligent analysis of multimodal cancer data and promoting the automation and intelligence of diagnosis and prognosis; on the other hand, it is also of great significance for exploring the connection between the molecular level and the individual level of cancer, assisting in the selection of therapeutic targets, improving the occurrence and development mechanism of cancer, and unleashing the huge potential of computers in data analysis.

[0077] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0078] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0079] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0080] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0081] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0082] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0083] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0084] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0085] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A multimodal tumor data fusion method comprising the following steps: Extracting a first morphological feature from the target image data, where the first morphological feature reflects the overall geometric morphological representation of the diseased tissue; Extracting secondary morphological features and structural features from the target pathological data. The secondary morphological features describe the degree of abnormality of the tumor cell nucleus, and the structural features describe the distribution characteristics of the cells. Extract cluster features based on the target gene data, which reflect the expression characteristics of the tumor at the gene molecular level; The first morphological feature, the second morphological feature, the structural feature, and the cluster feature are fused through a trained fusion network, wherein the fusion network includes a first-layer network and a second-layer network, and the first-layer network fuses the first morphological feature with the second morphological feature, the first morphological feature with the structural feature, the second morphological feature with the cluster feature, and the structural feature with the cluster feature in pairs based on changes from micro to macro; the second-layer network further fuses the pairwise fused features to obtain the first fused feature and the second fused feature; The first fusion feature and the second fusion feature are used to classify and identify the target or perform regression prediction.

2. The method according to claim 1, characterized in that The second-layer network includes two path branches, one path branch fuses the first morphological feature with the second morphological feature, the cluster feature with the structural feature to obtain a first fusion feature, and the other path branch fuses the cluster feature with the second morphological feature, the first morphological feature with the structural feature to obtain a second fusion feature. The two path branches have the same structure, each including an input layer, a multi-layer fully connected hidden layer and an output layer.

3. The method according to claim 1, characterized in that The first layer of the network performs Kronecker product on each pair of features, which is expressed as follows: Among them, A and B are feature matrices with dimensions of m×n and p×q respectively.

4. The method according to claim 2, characterized in that The second-layer network includes two path branches, each of which includes two fully connected layers as hidden layers, and adopts a neuron dropout strategy and a linear rectification activation function to reduce the dimension of the fusion features layer by layer, and then output the first fusion feature and the second fusion feature respectively.

5. The method according to claim 1, wherein According to the target classification task, the loss function for training the fusion network is set to: Where N represents the total number of samples, y n For sample x n The true label, Represents sample x n The predicted probability of , n is the sample index subscript.

6. The method according to claim 1, wherein According to the target regression task, the loss function for training the fusion network is set to: Among them, t i is the i-th time node, R(t i ) is the time node t i The risk set at time δ i Marks data as missing.

7. The method according to claim 1, characterized in that The features of each modality input to the fusion network are obtained according to the following steps: For the extracted first morphological features, second morphological features, structural features and cluster features, the feature embedding selection method is used to screen out high-correlation features. The principal component analysis algorithm is used to reduce the dimension of the filtered features and unify the feature dimensions of each mode.

8. A multimodal tumor data fusion device, comprising: Feature extraction module: used for extracting a first morphological feature from the target image data, where the first morphological feature reflects the overall geometric morphological representation of the diseased tissue; Based on the target's pathological data, secondary morphological features and structural features are extracted. The secondary morphological features describe the degree of abnormality of the tumor cell nucleus, and the structural features describe the distribution characteristics of the cells. Based on the target's genetic data, cluster features are extracted. The cluster features reflect the expression characteristics of the tumor at the genetic molecular level. Feature fusion module: used to fuse the first morphological feature, the second morphological feature, the structural feature and the cluster feature through a trained fusion network, wherein the fusion network includes a first-layer network and a second-layer network. The first-layer network fuses the first morphological feature with the second morphological feature, the first morphological feature with the structural feature, the second morphological feature with the cluster feature, and the structural feature with the cluster feature in pairs based on the changes from micro to macro. The second-layer network further fuses the pairwise fused features to obtain the first fused feature and the second fused feature. Prediction module: used to classify and identify or perform regression prediction on the target using the first fusion feature and the second fusion feature.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of being run on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images