Multi-modal structure learning system and method for pathological image and gene data

Abstract

The invention discloses a pathological image and gene data multi-modal structure learning system and method, and relates to the technical field of multi-modal data processing, and the system comprises a training data obtaining module which is used for obtaining training data; the data processing module is used for sequentially inputting the training data into the initial encoder group and the graph structure learning device group to obtain a fine graph set; the pre-training module is used for inputting the fine image set into the image network encoder group to construct a first loss function and training the components to obtain a pre-trained component; the fine tuning data acquisition module is used for acquiring fine tuning data and constructing a cache region; the branch construction module is used for respectively constructing an online branch model and a target branch model as fine tuning models based on the pre-training component; the fine tuning module is used for inputting the fine tuning data into the fine tuning model and combining the fine tuning data with the cache region to obtain a second loss function training online branch model; and the result output module is used for obtaining a to-be-processed pathological image and inputting the to-be-processed pathological image into the trained online branch model to obtain a fusion result.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing technology, and more specifically to a multimodal structure learning system and method for pathological images and gene data. Background Technology

[0002] Whole-section images of pathological tissue contain rich morphological visual information, while genetic data objectively describes a patient's health status and individual specificity at the molecular level. Combining information from both modalities can provide a more comprehensive description of the patient's condition. However, the high cost and difficulty of acquiring genetic data currently result in a lack of complete genetic data in many application scenarios. Existing multimodal fusion methods still require complete image-gene pairing data during the inference stage. To alleviate this predicament, some studies are gradually focusing on multimodal fusion tasks with missing genetic information. Currently, the main approach is to reconstruct genetic information based on pathological image information.

[0003] However, current methods for addressing missing genetic data rely solely on individual patient pathological image data for genetic information reconstruction. This neglects the inter-case relationships and the guiding role of existing real-world genetic data in the database during the generation of missing genetic data, resulting in insufficient data accuracy and ultimately poor inference. Furthermore, these methods do not utilize the real-world genetic data of already acquired cases with diagnostic information during the inference phase; however, this existing database knowledge can significantly improve the efficiency and accuracy of disease diagnosis.

[0004] Therefore, how to improve the authenticity of the generated genetic data paired with pathological images, and thus improve the final inference effect, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a multimodal structure learning system and method for pathological images and gene data, which improves the authenticity of the generated gene data paired with pathological images, thereby improving the final inference effect.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal structure learning system for pathological images and genetic data includes: The training data acquisition module is used to acquire the first pathological image and the paired first gene data as training data. The data processing module is used to input the training data into the initial encoder group to obtain a modality map set, and input the modality map set into the graph structure learner group to obtain a fine map set; The pre-training module is used to input the fine graph set into the graph network encoder group, obtain the corresponding result to construct a first loss function, and train the encoder group, the graph structure learner group and the graph network encoder group to obtain the pre-trained components; The fine-tuning data acquisition module is used to acquire second pathological images with task labels and paired second gene data as fine-tuning data and build a cache area; A branch construction module is used to construct an online branch model and a target branch model based on the pre-trained components, respectively. The fine-tuning module is used to input the fine-tuning data into the online branch model, the target branch model combined with the cache, obtain the output result, construct a second loss function, and train the online branch model to obtain the fine-tuned online model. The results output module is used to obtain pathological images without gene data and input them into the fine-tuning online model to obtain fusion results with fusion gene hints.

[0007] In one embodiment, the data processing module includes: an image characterization acquisition unit, a gene characterization acquisition unit, a fusion characterization acquisition unit, a modality map acquisition unit, and a fine map acquisition unit; The image representation acquisition unit is used to divide the first pathological image into multiple image blocks and map them into multiple feature vectors to obtain a feature matrix, and input the feature matrix into the image encoder to obtain the image representation; The gene characterization acquisition unit is used to obtain a genome feature matrix based on the first gene data preprocessing and input it into the gene encoder to obtain gene characterization. The fusion characterization acquisition unit is used to obtain a fusion characterization by splicing the image characterization and the gene characterization. The modality graph acquisition unit is used to perform nearest neighbor clustering based on the image representation, the gene representation, and the fusion representation respectively to obtain the corresponding adjacency matrix, and to construct the modality graph set based on the corresponding representations, which consists of the image modality graph, the gene modality graph, and the fusion modality graph. The fine-map acquisition unit is used to input the image modality map, the gene modality map, and the fusion modality map into the corresponding graph structure learner to obtain the fine-map set composed of the image fine-map, the gene fine-map, and the fusion fine-map.

[0008] In one embodiment, the pre-training module includes: an image enhancement unit, a representation extraction unit, a first function construction unit, and a pre-training unit; The image enhancement unit is used to remove and add random edges based on the adjacency matrix corresponding to the image modality map and the gene modality map to obtain the image enhancement map and the gene enhancement map; The characterization extraction unit is used to input the image fine map, the gene fine map, the fusion fine map, the image enhancement map, and the gene enhancement map into the corresponding graph network encoder to obtain the corresponding characterization. The first function construction unit is used to construct a pre-trained loss function based on the corresponding representation; The pre-training unit is used to pre-train the encoder group, the graph structure learner group, and the graph network encoder group based on the pre-training loss function to obtain a pre-trained optimized encoder group, optimized graph structure learner group, and optimized graph network encoder group. The optimized encoder group includes an optimized image encoder and an optimized gene encoder. The optimized graph structure learner group includes an optimized image structure learner, an optimized gene structure learner, and an optimized fusion structure learner. The optimized graph network encoder group includes an optimized image graph network encoder, an optimized fusion graph network encoder, and an optimized gene graph network encoder.

[0009] In one embodiment, the characterization extraction unit includes: a first extraction subunit, a second extraction subunit, and a third extraction subunit; The first extraction subunit is used to input the image fine map and the image enhancement map into the image graph network encoder to obtain the image fine map representation and the enhancement map representation respectively; The second extraction subunit is used to input the gene fine map and the gene enhancement map into the gene map network encoder to obtain the gene fine map representation and the enhanced gene representation accordingly. The third extraction subunit is used to obtain a fusion fine map representation based on the fusion fine map input to the fusion map network encoder. The image fine map representation, the enhancement map representation, the gene fine map representation, the enhancement gene representation, and the fusion fine map representation together serve as the corresponding representation.

[0010] In one embodiment, the first function construction unit includes: a first constraint subunit, a second constraint subunit, a third constraint subunit, and a first loss construction subunit; The first constraint subunit is used to construct a first alignment constraint based on the image fine graph representation, the enhancement graph representation, the gene fine graph representation, and the enhancement gene representation; The second constraint subunit is used to construct a second alignment constraint based on the image fine graph representation and the gene fine graph representation; The third constraint subunit is used to construct a third alignment constraint based on the image fine graph representation, the gene fine graph representation, and the fusion fine graph representation; The first loss construction subunit is used to obtain the pre-trained loss function by adding the first alignment constraint, the second alignment constraint and the third alignment constraint.

[0011] In one embodiment, the fine-tuning data acquisition module includes: a fine-tuning data acquisition unit and a cache construction unit; The fine-tuning data acquisition unit is used to acquire the second pathological image with task tags and the paired second gene data as the fine-tuning data; The buffer construction unit is used to construct the buffer by inputting the second pathological image and the second gene data into the optimized image encoder and the optimized gene encoder, obtaining corresponding image coding features and gene coding features, and splicing them together to obtain real fusion features.

[0012] In one embodiment, the branch building module includes: an online model building unit and a target model building unit; The online model building unit is used to build the online branch model based on the optimized image encoder, the optimized image structure learner, the optimized image graph network encoder, and the inductor. The target model construction unit is used to construct the target branch model based on the optimized image encoder, the optimized gene encoder, the optimized image structure learner, and the optimized image graph network encoder.

[0013] In one embodiment, the fine-tuning module includes: a view enhancement unit, an online result output unit, a target result output unit, a second loss construction unit, and a fine-tuning unit; The view enhancement unit is used to perform data enhancement based on the second pathological image to obtain a first enhanced view and a second enhanced view. The online result output unit is used to input the first enhanced view into the online branch model, process it through the optimized image encoder to obtain a first encoded feature, input the first encoded feature into the inductor to obtain a gene cue feature and fuse it with the first encoded feature to obtain a first fused feature, input the first fused feature and the buffer data into the optimized image structure learner to obtain a second fused feature, and input the first fused feature and the second fused feature into the optimized image graph network encoder to obtain a third fused feature as the online result; The target result output unit is used to input the second enhanced view and the second gene data into the target branch model, process them through the optimized image encoder and the optimized gene encoder respectively, obtain the second coding feature and the gene coding feature, and fuse them to obtain the first splicing feature. The first splicing feature and the buffer data are input into the optimized image structure learner to obtain the second splicing feature. The first splicing feature and the second splicing feature are input into the optimized image graph network encoder to obtain the third splicing feature as the target result. The second loss construction unit is used to construct the fine-tuning loss function based on the first fusion feature, the second fusion feature, the third fusion feature, the first splicing feature, the second splicing feature, and the third splicing feature; The fine-tuning unit is used to train the online branch model based on the fine-tuning loss function to obtain the fine-tuned online model.

[0014] In one embodiment, the second loss construction unit includes: a first loss subunit, a second loss subunit, a third loss subunit, a fourth loss subunit, and a total loss subunit; The first loss subunit is used to construct a first loss function based on the third fusion feature; The second loss subunit is used to construct a second loss function based on the first fusion feature and the second splicing feature; The third loss subunit is used to construct a third loss function based on the third fusion feature and the third splicing feature; The fourth loss subunit is used to construct a fourth loss function based on the second fusion feature and the second splicing feature. The total loss subunit is used to construct the fine-tuning loss function based on the first loss function, the second loss function, the third loss function, and the fourth loss function.

[0015] A multimodal structure learning method for pathological images and genetic data includes: The first pathological image and the paired first gene data were used as training data. The training data is input into the initial encoder group to obtain the modality map set, and the modality map set is input into the graph structure learner group to obtain the fine map set; Based on the fine graph set input to the graph network encoder group, the corresponding result is used to construct the first loss function, and the encoder group, the graph structure learner group and the graph network encoder group are trained to obtain the pre-trained components; Acquire second pathological images with task tags and paired second gene data as fine-tuning data and construct a cache area; Based on the pre-trained components, an online branch model and a target branch model are constructed respectively; Based on the fine-tuning data input to the online branch model, the target branch model combined with the cache, the output result is used to construct a second loss function and train the online branch model to obtain the fine-tuned online model. Pathological images without gene data are obtained and input into the fine-tuned online model to obtain fusion results characterized by fusion gene suggestions.

[0016] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a multimodal structure learning system and method for pathological images and gene data, which has the following beneficial effects: 1. This invention employs graph structure learning to pre-train multimodal representation graphs and enables dynamic graph construction for downstream tasks, thereby achieving high-performance inference using a single histopathological image.

[0017] 2. This invention can capture the structural relationships between diagnostic cases by introducing data-driven graph structure pre-training. During the inference process, real genomic data is introduced as auxiliary supervision to make full use of genomic data with diagnostic information to alleviate the dilemma of modality loss, thus achieving superior performance.

[0018] 3. This invention promotes the reconstruction of missing genomic features by utilizing multimodal relationships between cases and jointly enhances the characterization capabilities of the whole-slice encoder, thereby greatly promoting the development of data-efficient precision oncology.

[0019] 4. This invention utilizes real genomic data as auxiliary supervision to reconstruct missing modality data during inference, thereby maximizing the use of existing gene data with diagnostic information. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a multimodal structure learning system for pathological images and gene data provided by the present invention.

[0022] Figure 2 The present invention provides a reasoning flowchart for a multimodal structure learning system for pathological images and gene data.

[0023] Figure 3The flowchart of a multimodal structure learning method for pathological images and gene data provided by the present invention is shown. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Example 1 like Figures 1-2 As shown in the figure, an embodiment of the present invention discloses a multimodal structure learning system for pathological images and gene data, comprising: The training data acquisition module is used to acquire the first pathological image and the paired first gene data as training data.

[0026] Furthermore, the training data acquisition module acquires case data that are paired with the first pathological image and the first gene data in the database as training data. There are no task labels, and the data usually reaches tens of thousands of cases. The purpose is to train graph structure learners with different modal data and fused modal data.

[0027] The data processing module is used to input the training data into the initial encoder group to obtain the modality map set, and then input the modality map set into the graph structure learner group to obtain the fine map set.

[0028] Furthermore, the data processing module includes: an image representation acquisition unit, a gene representation acquisition unit, a fusion representation acquisition unit, a modality map acquisition unit, and a fine-scale map acquisition unit; The image representation acquisition unit is used to divide the first pathological image into multiple image blocks and map them into multiple feature vectors to obtain a feature matrix, and input the feature matrix into the image encoder to obtain the image representation; The gene characterization acquisition unit is used to obtain a genome feature matrix based on the first gene data preprocessing and input it into the gene encoder to obtain gene characterization. The fusion characterization acquisition unit is used to obtain a fusion characterization by splicing together image and gene characterizations. The modality acquisition unit is used to perform nearest neighbor clustering based on image representation, gene representation and fusion representation respectively to obtain the corresponding adjacency matrix, and to construct a modality set composed of image modality map, gene modality map and fusion modality map based on the corresponding representation; The fine-map acquisition unit is used to input image modality maps, gene modality maps, and fusion modality maps into the corresponding graph structure learners to obtain a fine-map set composed of image fine-maps, gene fine-maps, and fusion fine-maps.

[0029] Furthermore, the image representation acquisition unit is used to segment the input first pathological image into multiple 256×256 pixel image patches, and map all image patches to multiple corresponding data using the Gigapath pre-trained DINO v2 framework. d p From eigenvectors, we obtain the feature matrix. Among them, China M k Indicates the first k The number of image patches contained in a patient's pathology image. I Represented as image modal data, the feature matrix is ​​input to the image encoder, which uses a Gigapath pre-trained LongNet encoder to aggregate the representations of all image patches, thus obtaining the image representation. h k = , ,in, Indicates an image encoder. d h The dimensions of the pathological images are used to obtain the image representations of all cases, denoted as . Where K represents the total number of medical records.

[0030] Furthermore, the gene characterization acquisition unit is used to divide all gene data into 50 groups based on the first gene data according to the Human Molecular Signatures Database (MSigDB), and encode them through an SNN network to obtain a genome feature matrix. ,in, N This indicates the number of groups; in this embodiment, it is 50. d g Representing feature dimension, G Gene modalities are represented and input to the gene encoder based on the genome feature matrix. Average pooling is performed on all genomic feature matrices to obtain gene characterization. , , d g = d h =d, based on this, the genetic characterization of all cases is obtained, denoted as d. .

[0031] Furthermore, the fusion representation acquisition unit is used to obtain a fusion representation based on the splicing of image representation and gene representation. Where concatenate represents the splicing operation, based on which the fused representation of all cases is obtained, denoted as .

[0032] Furthermore, the modal graph acquisition unit acquires a modal graph set, specifically including: Nearest neighbor clustering is performed on the image representation H to obtain the adjacency matrix of the image representation. A H ∈[0,1] K×K Based on image representation H as node features, and combined with the image representation adjacency matrix, the modality of case images is constructed, resulting in an image modality map. G H ={ H , A H}; Perform nearest neighbor clustering on gene representation G to obtain the gene representation adjacency matrix. A G ∈[0,1] K×K Based on gene representation G as node features, and combined with the gene representation adjacency matrix, the gene modality of the case is constructed, resulting in a gene modality graph. G G ={ G , A G}; Perform nearest neighbor clustering on the fused representation F to obtain the adjacency matrix of the fused representation. A F ∈[0,1] K×K Based on the fusion representation F as node features, and combined with the fusion representation adjacency matrix, the fusion modality of the case is constructed, resulting in the fusion modality graph. G F ={ F , A F}; Based on image modality G H Gene modality diagram G G and fusion modal diagram G F To form a modal atlas.

[0033] Furthermore, the fine-map acquisition unit acquires a fine-map atlas, specifically including: Based on image modality G H Input to image structure learner GSL H The adjacency matrix of the fine image modality map is obtained. The graph structure learning process only changes the adjacency matrix, without altering the node features, to obtain a refined graph of the image. , H}; Based on gene modality maps GG Input to gene structure learner GSL G The adjacency matrix of the refined gene modality map is obtained. The graph structure learning process only changes the adjacency matrix, without altering the node features, to obtain a refined gene graph. , G}; Based on fusion modality graph G F Input to fusion structure learner GSL F The adjacency matrix of the fine-grained fusion modality graph is obtained. The graph structure learning process only changes the adjacency matrix, without altering the node features, to obtain a fused refined graph. , F}; A fine atlas is composed of fine-grained images, fine-grained genes, and fused fine-grained images.

[0034] The pre-training module is used to input the fine graph set into the graph network encoder group, obtain the corresponding results to construct the first loss function, and train the encoder group, graph structure learner group and graph network encoder group to obtain the pre-trained components.

[0035] Furthermore, the pre-training module includes: an image enhancement unit, a representation extraction unit, a first function construction unit, and a pre-training unit; The image enhancement unit is used to remove and add random edges based on the adjacency matrix corresponding to the image modality map and the gene modality map to obtain the image enhancement map and the gene enhancement map; The characterization extraction unit is used to input image fine maps, gene fine maps, fusion fine maps, image enhancement maps and gene enhancement maps into the corresponding graph network encoder to obtain the corresponding characterizations; The first function construction unit is used to construct a pre-trained loss function based on the corresponding representation; The pre-training unit is used to pre-train the encoder group, graph structure learner group, and graph network encoder group based on the pre-training loss function, to obtain the pre-trained optimized encoder group, optimized graph structure learner group, and optimized graph network encoder group. The optimized encoder group includes: optimized image encoder and optimized gene encoder; the optimized graph structure learner group includes: optimized image structure learner, optimized gene structure learner, and optimized fusion structure learner; and the optimized graph network encoder group includes: optimized image graph network encoder, optimized fusion graph network encoder, and optimized gene graph network encoder.

[0036] Furthermore, the image enhancement unit acquires image enhancement maps and gene enhancement maps, specifically including: Based on image modality G HAdjacency matrix in A H By removing and adding random edges, the adjacency matrix of the enhanced image modality is obtained. Based on adjacency matrix The image enhancement map is obtained by combining the image representation H with the image representation H. , H}; Based on gene modality maps G G Adjacency matrix in A G By removing and adding random edges, the adjacency matrix of the enhanced gene modality graph is obtained. Based on adjacency matrix Gene enhancement diagram obtained from gene characterization G { , H}

[0037] Furthermore, the characterization extraction unit includes: a first extraction subunit, a second extraction subunit, and a third extraction subunit; The first extraction subunit is used to extract based on the image fine map { , H} and image enhancement map { , H Input to image graph network encoder GCN H This corresponds to the fine-scale image representation. and enhanced graph representation : ; ; in, d f This indicates the dimension of the graph representation.

[0038] The second extraction subunit is used for extraction based on the gene fine map { , G} and gene enhancement map { , H Input to the gene map network encoder GCN G This corresponds to the detailed gene map representation. and enhance gene representation The formula is the same as the formula for the above image fine-scale representation and enhanced image representation.

[0039] The third extraction subunit is used for extraction based on the fused fine map. , F Input to the fusion graph network encoder GCN F A fused fine-scale representation is obtained. Image fine-scale representation Enhanced graph representation Fine-map representation of genes Enhance gene representation and fusion of fine-scale graph representations Together they serve as corresponding representations.

[0040] Furthermore, the first function construction unit includes: a first constraint subunit, a second constraint subunit, a third constraint subunit, and a first loss construction subunit. The first constraint subunit is used to construct an in-modality graph structure learning loss function based on image fine graph representation, enhanced graph representation, gene fine graph representation, and enhanced gene representation as the first alignment constraint. : ; ; in, I lb and Both represent the InfoNCE loss function. sim Represents cosine similarity. τ Indicates the temperature coefficient. Indicates the first k Detailed graphical representation of each case, Indicates the first k Enhanced image representation of individual cases, Indicates the first i Enhanced graph representation of 1 case, with a total of K cases.

[0041] The second constraint subunit is used to align two modalities of information from the same case based on image fine-map representation and gene fine-map representation, thus constructing the second alignment constraint. : .

[0042] The third constraint subunit is used to add constraints to the fusion representation and each modality representation based on image fine-map representation, gene fine-map representation, and fusion fine-map representation, thus constructing the third alignment constraint. : .

[0043] The first loss construction subunit is used to obtain the pre-trained loss function by adding the first alignment constraint, the second alignment constraint, and the third alignment constraint. : .

[0044] The fine-tuning data acquisition module is used to acquire second pathological images with task labels and paired second gene data as fine-tuning data and build a cache area.

[0045] Furthermore, the fine-tuning data acquisition module includes: a fine-tuning data acquisition unit and a cache construction unit; Fine-tuning the data acquisition unit to acquire second pathological images with task labels. and paired second gene data As fine-tuning data , S This is represented as the number of cases in the fine-tuned data; The buffer construction unit is used to input the corresponding second pathological image and second gene data into the optimized image encoder. and optimize gene encoder The corresponding image coding features and gene coding features are obtained and concatenated to obtain the true fusion features to construct the cache area. : .

[0046] The branch building module is used to build online branch models and target branch models based on pre-trained components.

[0047] Furthermore, the branch building module includes: an online model building unit and a target model building unit; An online model building unit is used to build online branching models based on an optimized image encoder, an optimized image structure learner, an optimized image graph network encoder, and an inductor. The target model building unit is used to build target branch models based on optimized image encoders, optimized gene encoders, optimized image structure learners, and optimized image graph network encoders.

[0048] Furthermore, in this embodiment, the inductor is composed of an SNN network.

[0049] The fine-tuning module is used to input fine-tuning data into the online branch model and the target branch model, combined with the buffer, to obtain the output results, construct a second loss function, and train the online branch model to obtain the fine-tuned online model.

[0050] Furthermore, the fine-tuning module includes: a view enhancement unit, an online result output unit, a target result output unit, a second loss construction unit, and a fine-tuning unit; The view enhancement unit is used to perform data enhancement based on the second pathological image to obtain a first enhanced view. Second Enhanced View ; Online results output unit for use based on the first enhanced view The input is fed into the online branch model, and the first encoded feature is obtained by optimizing the image encoder processing. h = The first encoded feature is input into the inductor to obtain the gene suggestion feature. Prompt G and with the first encoded feature h Fusion, resulting in the first fusion feature f Based on the first fusion feature f With cache data Input to optimized image structure learner GSL The second fusion feature is obtained. A r Based on the first fusion feature f With the second fusion feature A r Input to optimized image graph network encoder GCN The third fusion feature is obtained. Z As an online result; Target result output unit, used based on the second enhanced view Second gene data X G The input is fed into the target branch model, and processed by optimizing the image encoder and the gene encoder respectively to obtain the corresponding second encoded features. and gene coding characteristics And the first splicing feature is obtained by fusion. Based on the first splicing feature and cache data Input to optimized image structure learner GSL The second splicing feature is obtained. Based on the first splicing feature With the second splicing feature Input to optimized image graph network encoder GCN The third splicing feature is obtained. As the target result; The second loss construction unit is used to construct a fine-tuning loss function based on the first fusion feature, the second fusion feature, the third fusion feature, the first concatenation feature, the second concatenation feature, and the third concatenation feature; The fine-tuning unit is used to train the online branch model based on the fine-tuning loss function to obtain the fine-tuned online model.

[0051] Furthermore, the first fusion feature f for: ; Second fusion feature A r for: ; in, Readout This represents a function that reads case representations from the buffer and concatenates them with the input fusion representation; Third fusion feature Z for: ; Third fusion feature Z The target result is used for specific task prediction.

[0052] Furthermore, the second loss construction unit includes: a first loss subunit, a second loss subunit, a third loss subunit, a fourth loss subunit, and a total loss subunit. The first loss subunit is used to construct the first loss function based on the third fused feature. : ; in, y Indicates the truth value label for the task. This represents the batch of data input to the previous network, which is a subset of the training set.

[0053] The second loss subunit is used to construct a second loss function based on the first fused feature and the second concatenated feature. : .

[0054] The third loss subunit is used to construct the third loss function based on the third fusion feature and the third concatenation feature. : ; in, This represents the node representation of the current batch of data input to the network. This indicates the corresponding output of the target branch.

[0055] The fourth loss subunit is used to introduce weighted balanced cross-entropy (BCE) based on the second fusion feature and the second splicing feature to align the structural predictions of the two branches, thus constructing the fourth loss function. : ; ; in, and They are respectively for Loss calculation for zero and non-zero elements. c 0 and c1 represents the number of zero and non-zero elements in the adjacency matrix of the target branch prediction structure, respectively. The purpose of the alignment constraint is to ensure that the online branch can learn the knowledge of the large-scale data in the first stage through the target branch during the training process, so as to obtain effective and robust case representation and stable structural association. Based on this, the association between cases and the existing real gene data in the database are used to guide the process of generating missing gene data.

[0056] The total loss subunit is used to construct a fine-tuning loss function based on the first loss function, the second loss function, the third loss function, and the fourth loss function. : .

[0057] Furthermore, the fine-tuning unit is used to train the online branch model based on the fine-tuning loss function, update the online branch parameters, freeze the target branch parameters, and use the exponential moving average (EMA) strategy to ensure stable updates of the target branch parameters, ultimately obtaining the fine-tuned online model.

[0058] The results output module is used to obtain pathological images without gene data and input them into the fine-tuning online model to obtain fusion results with fusion gene hints.

[0059] Furthermore, the results output module outputs the fusion results, specifically including: The pathological image without gene data is input into the fine-tuned optimized image encoder to obtain the encoded image representation. The image representation is then input into the inductor to generate a gene cue representation with paired gene information. The image representation and the gene cue representation are then concatenated to obtain the multimodal representation of the input case. Next, this representation and the buffer representation are fed into the fine-tuned optimized image graph structure learner to obtain the refined adjacency matrix. Finally, the adjacency matrix and the multimodal representation of the input case are fed into the fine-tuned optimized image graph network encoder to obtain the cure probability of the case.

[0060] Example 2 In the reasoning process, this invention introduces real genomic data as auxiliary supervision to make full use of genomic data with diagnostic information to alleviate the predicament of modality loss, and achieves superior performance, as shown in Table 1.

[0061] Table 1. c-Index indices for survival prognosis (mean ± standard deviation)

[0062] This patent was compared with existing technologies on three datasets, and its performance is significantly better than existing methods for solving gene modality loss, and it achieves performance comparable to multimodal fusion methods, demonstrating the effectiveness of this patent.

[0063] Example 3 like Figure 3 As shown, based on the same inventive concept, this invention also provides a multimodal structure learning method for pathological images and gene data, including: The first pathological image and the paired first gene data were used as training data. The training data is input into the initial encoder group to obtain the modality map set, and the modality map set is input into the graph structure learner group to obtain the fine map set; Based on the fine graph atlas input to the graph network encoder group, the corresponding results are used to construct the first loss function, and the encoder group, graph structure learner group and graph network encoder group are trained to obtain the pre-trained components; Acquire second pathological images with task tags and paired second gene data as fine-tuning data and construct a cache area; Online branching models and target branching models are constructed based on pre-trained components; Based on the fine-tuning data input to the online branch model and the target branch model combined with the buffer, the output results are used to construct a second loss function and train the online branch model to obtain the fine-tuned online model. Pathological images without gene data are obtained and input into a fine-tuned online model to obtain fusion results indicated by fusion gene suggestions.

[0064] Furthermore, in this embodiment, the implementation methods of each step correspond one-to-one with the above-mentioned system module function implementation process, and will not be described in detail here.

[0065] Example 4 Based on the same inventive concept, the present invention also provides an electronic device, which includes a processor and a memory, wherein the memory stores instructions, characterized in that the instructions are loaded and executed by the processor to implement a multimodal structure learning method for pathological images and gene data as in Example 3.

[0066] Based on the same inventive concept, the present invention also provides a computer device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When the processor executes a program stored in memory, it can implement a multimodal structure learning method for pathological images and gene data, as shown in Example 3.

[0067] The electronic device may include a processor, a communications interface, a memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions in the memory to execute a multimodal structure learning method for pathological images and gene data as described in Embodiment 3.

[0068] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0069] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0070] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal structure learning system for pathological images and gene data, characterized in that, include: The training data acquisition module is used to acquire the first pathological image and the paired first gene data as training data. The data processing module is used to input the training data into the initial encoder group to obtain a modality map set, and input the modality map set into the graph structure learner group to obtain a fine map set; The pre-training module is used to input the fine graph set into the graph network encoder group, obtain the corresponding result to construct a first loss function, and train the encoder group, the graph structure learner group and the graph network encoder group to obtain the pre-trained components; The fine-tuning data acquisition module is used to acquire second pathological images with task labels and paired second gene data as fine-tuning data and build a cache area; A branch construction module is used to construct an online branch model and a target branch model based on the pre-trained components, respectively. The fine-tuning module is used to input the fine-tuning data into the online branch model, the target branch model combined with the cache, obtain the output result, construct a second loss function, and train the online branch model to obtain the fine-tuned online model. The results output module is used to obtain pathological images without gene data and input them into the fine-tuning online model to obtain fusion results with fusion gene hints.

2. The multimodal structure learning system for pathological images and gene data according to claim 1, characterized in that, The data processing module includes: an image representation acquisition unit, a gene representation acquisition unit, a fusion representation acquisition unit, a modality map acquisition unit, and a fine map acquisition unit; The image representation acquisition unit is used to divide the first pathological image into multiple image blocks and map them into multiple feature vectors to obtain a feature matrix, and input the feature matrix into the image encoder to obtain the image representation; The gene characterization acquisition unit is used to obtain a genome feature matrix based on the first gene data preprocessing and input it into the gene encoder to obtain gene characterization. The fusion characterization acquisition unit is used to obtain a fusion characterization by splicing the image characterization and the gene characterization. The modality graph acquisition unit is used to perform nearest neighbor clustering based on the image representation, the gene representation, and the fusion representation respectively to obtain the corresponding adjacency matrix, and to construct the modality graph set based on the corresponding representations, which consists of the image modality graph, the gene modality graph, and the fusion modality graph. The fine-map acquisition unit is used to input the image modality map, the gene modality map, and the fusion modality map into the corresponding graph structure learner to obtain the fine-map set composed of the image fine-map, the gene fine-map, and the fusion fine-map.

3. The multimodal structure learning system for pathological images and gene data according to claim 1, characterized in that, The pre-training module includes: an image enhancement unit, a representation extraction unit, a first function construction unit, and a pre-training unit; The image enhancement unit is used to remove and add random edges based on the adjacency matrix corresponding to the image modality map and the gene modality map to obtain the image enhancement map and the gene enhancement map; The characterization extraction unit is used to input the image fine map, the gene fine map, the fusion fine map, the image enhancement map, and the gene enhancement map into the corresponding graph network encoder to obtain the corresponding characterization. The first function construction unit is used to construct a pre-trained loss function based on the corresponding representation; The pre-training unit is used to pre-train the encoder group, the graph structure learner group, and the graph network encoder group based on the pre-training loss function to obtain a pre-trained optimized encoder group, optimized graph structure learner group, and optimized graph network encoder group. The optimized encoder group includes an optimized image encoder and an optimized gene encoder. The optimized graph structure learner group includes an optimized image structure learner, an optimized gene structure learner, and an optimized fusion structure learner. The optimized graph network encoder group includes an optimized image graph network encoder, an optimized fusion graph network encoder, and an optimized gene graph network encoder.

4. The multimodal structure learning system for pathological images and gene data according to claim 3, characterized in that, The characterization extraction unit includes: a first extraction subunit, a second extraction subunit, and a third extraction subunit; The first extraction subunit is used to input the image fine map and the image enhancement map into the image graph network encoder to obtain the image fine map representation and the enhancement map representation respectively; The second extraction subunit is used to input the gene fine map and the gene enhancement map into the gene map network encoder to obtain the gene fine map representation and the enhanced gene representation accordingly. The third extraction subunit is used to obtain a fusion fine map representation based on the fusion fine map input to the fusion map network encoder. The image fine map representation, the enhancement map representation, the gene fine map representation, the enhancement gene representation, and the fusion fine map representation together serve as the corresponding representation.

5. A multimodal structure learning system for pathological images and gene data according to claim 4, characterized in that, The first function construction unit includes: a first constraint subunit, a second constraint subunit, a third constraint subunit, and a first loss construction subunit; The first constraint subunit is used to construct a first alignment constraint based on the image fine graph representation, the enhancement graph representation, the gene fine graph representation, and the enhancement gene representation; The second constraint subunit is used to construct a second alignment constraint based on the image fine graph representation and the gene fine graph representation; The third constraint subunit is used to construct a third alignment constraint based on the image fine graph representation, the gene fine graph representation, and the fusion fine graph representation; The first loss construction subunit is used to obtain the pre-trained loss function by adding the first alignment constraint, the second alignment constraint and the third alignment constraint.

6. The multimodal structure learning system for pathological images and gene data according to claim 5, characterized in that, The fine-tuning data acquisition module includes: a fine-tuning data acquisition unit and a cache construction unit; The fine-tuning data acquisition unit is used to acquire the second pathological image with task tags and the paired second gene data as the fine-tuning data; The buffer construction unit is used to construct the buffer by inputting the second pathological image and the second gene data into the optimized image encoder and the optimized gene encoder, obtaining corresponding image coding features and gene coding features, and splicing them together to obtain real fusion features.

7. A multimodal structure learning system for pathological images and gene data according to claim 6, characterized in that, The branch construction module includes: an online model construction unit and a target model construction unit; The online model building unit is used to build the online branch model based on the optimized image encoder, the optimized image structure learner, the optimized image graph network encoder, and the inductor. The target model construction unit is used to construct the target branch model based on the optimized image encoder, the optimized gene encoder, the optimized image structure learner, and the optimized image graph network encoder.

8. A multimodal structure learning system for pathological images and gene data according to claim 7, characterized in that, The fine-tuning module includes: a view enhancement unit, an online result output unit, a target result output unit, a second loss construction unit, and a fine-tuning unit; The view enhancement unit is used to perform data enhancement based on the second pathological image to obtain a first enhanced view and a second enhanced view. The online result output unit is used to input the first enhanced view into the online branch model, process it through the optimized image encoder to obtain a first encoded feature, input the first encoded feature into the inductor to obtain a gene cue feature and fuse it with the first encoded feature to obtain a first fused feature, input the first fused feature and the buffer data into the optimized image structure learner to obtain a second fused feature, and input the first fused feature and the second fused feature into the optimized image graph network encoder to obtain a third fused feature as the online result; The target result output unit is used to input the second enhanced view and the second gene data into the target branch model, process them through the optimized image encoder and the optimized gene encoder respectively, obtain the second coding feature and the gene coding feature, and fuse them to obtain the first splicing feature. The first splicing feature and the buffer data are input into the optimized image structure learner to obtain the second splicing feature. The first splicing feature and the second splicing feature are input into the optimized image graph network encoder to obtain the third splicing feature as the target result. The second loss construction unit is used to construct the fine-tuning loss function based on the first fusion feature, the second fusion feature, the third fusion feature, the first splicing feature, the second splicing feature, and the third splicing feature; The fine-tuning unit is used to train the online branch model based on the fine-tuning loss function to obtain the fine-tuned online model.

9. A multimodal structure learning system for pathological images and gene data according to claim 8, characterized in that, The second loss construction unit includes: a first loss subunit, a second loss subunit, a third loss subunit, a fourth loss subunit, and a total loss subunit; The first loss subunit is used to construct a first loss function based on the third fusion feature; The second loss subunit is used to construct a second loss function based on the first fusion feature and the second splicing feature; The third loss subunit is used to construct a third loss function based on the third fusion feature and the third splicing feature; The fourth loss subunit is used to construct a fourth loss function based on the second fusion feature and the second splicing feature. The total loss subunit is used to construct the fine-tuning loss function based on the first loss function, the second loss function, the third loss function, and the fourth loss function.

10. A multimodal structure learning method for pathological images and gene data, used to implement the multimodal structure learning system for pathological images and gene data as described in any one of claims 1-9, characterized in that, include: The first pathological image and the paired first gene data were used as training data. The training data is input into the initial encoder group to obtain the modality map set, and the modality map set is input into the graph structure learner group to obtain the fine map set; Based on the fine graph set input to the graph network encoder group, the corresponding result is used to construct the first loss function, and the encoder group, the graph structure learner group and the graph network encoder group are trained to obtain the pre-trained components; Acquire second pathological images with task tags and paired second gene data as fine-tuning data and construct a cache area; Based on the pre-trained components, an online branch model and a target branch model are constructed respectively; Based on the fine-tuning data input to the online branch model, the target branch model combined with the cache, the output result is used to construct a second loss function and train the online branch model to obtain the fine-tuned online model. Pathological images without gene data are obtained and input into the fine-tuned online model to obtain fusion results characterized by fusion gene suggestions.

Images