Gene mutation prediction method and system based on pathological report and full-slice cross-modal representation learning

Abstract

The invention belongs to the technical field of histopathology full-slice image analysis, and relates to a gene mutation prediction method and system based on a pathological report and full-slice cross-modal representation learning, and the method comprises the steps: obtaining a histopathology full-slice image and a data set of a corresponding pathological and molecular detection report, and carrying out the preprocessing and coding; performing word segmentation and feature extraction on the pathological diagnosis report and the molecular detection report; cross-modal representation learning is realized through reconstruction of pathological report features and comparative learning based on a category refinement queue; designing a loss function and training the model offline by using a sample data set with a label; and performing lung cancer gene mutation prediction on the histopathology full-slice image by using the trained model. According to the method, the defects that detailed annotations of histopathology full-slice image data are difficult to obtain and the single-modal information expression ability is insufficient can be overcome, and slice image level mutation tag classification is effectively completed by using a cross-modal information guide weak supervision method of a pathological report and a molecular detection report.

Description

Technical Field

[0001] The present invention belongs to the technical field of whole-slice image analysis in histopathology, and in particular relates to a gene mutation prediction method and system based on pathology report and whole-slice cross-modal representation learning. Background Art

[0002] Traditional gene mutation detection methods, such as next-generation sequencing and targeted sequencing, involve collecting biological samples from patients, such as blood and tumor tissue, and performing nucleic acid extraction, library construction, gene sequencing, etc. In recent years, with the development of slide scanner technology, computer hardware, and artificial intelligence algorithms, deep learning has shown high efficiency in image processing, especially in the classification of medical pathology images.

[0003] With the continuous development and improvement of slice scanner technology, computer hardware and artificial intelligence algorithms, deep learning has become more efficient in image processing, especially in recognition and classification, and its application based on multi-instance learning has become more and more extensive. At the same time, it has also shown strong competitiveness in pathological image classification because it solves the inherent problems of large amount of information in pathological images and difficulty in obtaining block image labels. Using multi-supervised deep learning methods based on multi-instance learning for medical pathological image classification can not only reduce the burden on medical workers, but also improve diagnostic efficiency and improve the medical health level of the general public.

[0004] However, current deep learning-related gene mutation prediction methods are all based on a single image modality, ignoring the prior guidance information in other modalities such as text modalities. This limits the model's ability to fully explore the rich information expression of the entire slice, resulting in traditional gene sequencing tests having problems such as high cost and long cycle, and it is difficult to fully and accurately predict the type of gene mutation. Summary of the invention

[0005] The purpose of the present invention is to provide a gene mutation prediction method and system based on pathology report and whole-slice cross-modal representation learning, so as to solve the problems of high cost and long cycle of traditional gene sequencing detection mentioned in the background technology.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0007] In a first aspect, the present invention proposes a gene mutation prediction method based on pathology report and full-slice cross-modal representation learning, the method comprising:

[0008] S1, obtaining a tissue pathology full-slice image with image-level labels, as well as a pathology diagnosis report and a molecular detection report corresponding to each full-slice image, and preprocessing the full-slice image to obtain a local block image;

[0009] S2, encoding each of the local block images into a feature vector and then performing pooling aggregation to obtain a feature vector of the full slice image;

[0010] S3, performing word segmentation and feature extraction on the pathology diagnosis report and the molecular detection report to obtain a report feature vector and a word segmentation feature vector of the pathology diagnosis report, and a report feature vector of the molecular detection report;

[0011] S4. Establish the connection between the pathology diagnosis report and the local area of ​​the whole slice image through the cross-attention mechanism to reconstruct the overall semantics of the pathology diagnosis report; implement contrastive learning based on category refinement queue, cross-modally fuse the reconstructed pathology report features with the full slice features, and obtain a cross-modal representation learning model;

[0012] S5. Design a loss function including classification, reconstruction and molecular semantic contrast loss, use the labeled data set to train the cross-modal representation learning model, and use the trained cross-modal representation learning model to predict the gene mutation type of the whole tissue pathology slice image to be predicted.

[0013] Further, step S1 includes:

[0014] S101, obtaining histopathological full-slice images with labels indicating lung cancer gene mutation types, as well as pathological diagnosis reports and molecular detection reports corresponding to each full-slice image, to form a data set G, G = {(H 1 ,P 1 ,Z 1 ),(H 2 ,P 2 ,Z 2 ),...,(H i ,P i ,Z i ),...,(H N ,P N ,Z N )}, where H i represents the i-th histopathological full-slice image, P i represents the pathological diagnosis report corresponding to the i-th tissue pathology full-slice image, Z i represents the molecular detection report corresponding to the i-th histopathological full-slice image, and N represents the number of samples in the dataset;

[0015] S102, dividing the full slice image in the data set G into fixed-size block images according to the non-overlapping sliding window strategy, and removing the background block images without tissue, to obtain the local block images H i , where p i,jrepresents the jth block image in the i-th tissue pathology full slide image, M i Represents the total number of block images divided from the i-th histopathology full-slice image.

[0016] Further, step S2 includes:

[0017] Training a block-wise image encoder E based on self-supervised learning image , the block image encoder E image Encode the local block image as d s dimensional feature vector, i.e. The feature vector of the whole slice image is represented as the aggregated

[0018] Further, step S3 includes:

[0019] S301. Use pre-trained text feature encoder E text The pathological diagnosis report i Encode to get the complete report feature vector and word segmentation feature vector Where N i Indicates report P i The length of the word segmentation feature vector in E text Molecular test report Z i Encode to get the complete report feature vector

[0020] S302, using pre-trained text feature encoder E text Molecular test report Z i Encode to get the complete report feature vector

[0021] Further, step S4 includes:

[0022] S401, establish the connection between the word segmentation feature vector in the pathology diagnosis report and the local block image feature vector in the full slice image through the cross attention mechanism, and obtain the word segmentation feature vector W′ i , as follows:

[0023]

[0024] Q t =W i A Q , K t =X i A K , V t =X i A V

[0025] Among them A Q , A K , A V are all learnable weight matrices, d is the dimension of the input feature 512; the reconstructed word segmentation feature vector Obtaining the complete text feature vector through attention pooling

[0026] S402. Establish a queue for each lung cancer gene mutation label to store report features; regard the originally extracted pathology report features and the reconstructed report features that match them as positive samples, and the reconstructed features in other categories of queues as negative samples; for the reconstructed pathology report features, the original report features that match them are regarded as positive samples, and the original features in other categories of queues are regarded as negative samples; through contrastive learning, align the semantics of the fusion features with the molecular detection supervision information, as shown in the following formula:

[0027] Q = {q 0 ,q 1 , ..., q C-1}

[0028] P = {T′ i , T i}

[0029] List r =Concat(q 0 ,q 1 ,...,q C-2 ,T′ i )

[0030] List p =Concat(q 0 ,q 1 ,...,q C-2 ,T i )

[0031] List r =List r / List r .norm()

[0032] List p =List p / List p .norm()

[0033] logits r =np.dot(List r , List p .T)*np.exp(T)

[0034] logitsp =logits r .T

[0035] L contrastive =CE(logits r [C-1, C-1)+CE(logits p [C-1],C-1)

[0036] Where Q is the queue for storing feature sample pairs, P is the positive sample pair in the current batch, t is a learnable temperature coefficient, CE(·) is the cross entropy loss function, and each training iteration process performs contrastive learning. At the end of the iteration, the batch of sample pairs is queued to update the queue. The reconstruction report loss is expressed as

[0037] S403, reconstructing the pathology report feature T′ i With full slice feature S i Fusion obtains cross-modal fusion features Obtain a cross-modal representation learning model, based on the contrastive learning alignment F in S402 i and O i , we get the molecular semantic contrast loss The fusion features are semantically aligned with the molecular detection supervision information.

[0038] Furthermore, in step S5, the design includes a loss function for classification, reconstruction, and molecular semantic contrast loss, including:

[0039] Setting the loss function to include classification prediction loss Rebuilding reported losses and molecular semantic contrast loss The sum of , where the loss function

[0040]

[0041] Where CE represents the cross entropy loss, and yi represents the label corresponding to the full slice;

[0042]

[0043] In a second aspect, the present invention proposes a gene mutation prediction system based on pathology report and full-slice cross-modal representation learning, which is applied to perform any of the above gene mutation prediction methods, and the system comprises:

[0044] A data acquisition module is used to acquire histopathological full-slice images with image-level labels, as well as pathological diagnosis reports and molecular detection reports corresponding to each full-slice image, to form a data set;

[0045] A preprocessing module is used to preprocess the full slice image to obtain a local block image;

[0046] An image encoding module is used to encode each local block image into a feature vector and then perform pooling aggregation to obtain a feature vector of the full slice image;

[0047] A text encoding module is used to perform word segmentation and feature extraction on the pathology diagnosis report and the molecular detection report to obtain a report feature vector and a word segmentation feature vector of the pathology diagnosis report, and a report feature vector of the molecular detection report;

[0048] The cross-modal learning module is used to establish the connection between the pathology diagnosis report and the local area of ​​the full-slice image through the cross-attention mechanism, reconstruct the overall semantics of the pathology diagnosis report, and implement contrastive learning based on the category refinement queue to cross-modally fuse the reconstructed pathology report features with the full-slice features to obtain a cross-modal representation learning model;

[0049] The training and prediction module is used to design a loss function including classification, reconstruction and molecular semantic contrast loss, use the labeled data set to train the cross-modal representation learning model, and use the trained cross-modal representation learning model to predict the gene mutation type of the whole tissue pathology slice image to be predicted.

[0050] The beneficial effects of the present invention are:

[0051] 1. Compared with traditional molecular detection methods, the present invention only uses whole-slice images and histopathology reports for inference, achieving good lung cancer gene prediction performance, which can reduce the turnaround time of clinical gene mutation detection and help pathologists develop targeted treatment plans more quickly.

[0052] 2. The present invention uses a cross-modal representation learning model to effectively model the correlation between histological pathology full-slice images and corresponding pathology reports, and combines the prior guidance of molecular detection reports to solve the problem of single modality in other full-slice analysis methods based on deep learning, which can better express the feature representation of the full slice and improve the accuracy of mutation classification;

[0053] 3. The present invention designs a comparative learning strategy based on category refinement queues, which can improve the learning ability of deep networks to distinguish the characteristic expressions of different mutation types.

[0054] 4. The present invention uses the cross attention mechanism to establish fine-grained associations between diagnostic terms in pathological diagnosis reports and block images in the whole slice image, aligns cross-modal information at a fine-grained level, and combines it with the self-attention mechanism used between block images to allow the network model to adaptively focus on potential lesion areas to further infer possible lung cancer gene mutation sites, thereby improving the value of clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of a process of the gene mutation prediction method of the present invention;

[0056] Figure 2 A technical roadmap for the gene mutation prediction method of the present invention;

[0057] Figure 3 A weakly supervised network model diagram for report-guided cross-modal representation learning in the present invention;

[0058] Figure 4 It is a training sample image of the whole slice of tissue pathology in the present invention;

[0059] Figure 5 A system architecture diagram of the gene mutation prediction system in the present invention. DETAILED DESCRIPTION

[0060] The present application is further described in detail below in conjunction with the accompanying drawings. It is necessary to point out here that the following specific implementation methods are only used to further illustrate the present application and cannot be understood as limiting the scope of protection of the present application. Technical personnel in this field can make some non-essential improvements and adjustments to the present application based on the above application content.

[0061] Example 1

[0062] like Figure 1-4 As shown, this embodiment proposes a gene mutation prediction method based on cross-modal representation learning of pathology reports and full slices, which mainly uses the information exchange between the two modalities of image and text to realize the full mining of full slice information by the model, and realizes the alignment of the diagnosis description in the pathology report with the lesion area in the full slice through cross attention, and uses the contrast learning strategy based on category refinement to enhance the model's recognition ability for full slices of different mutation types; more specifically, the method includes:

[0063] S1, obtaining a tissue pathology full-slice image with image-level labels, as well as a pathology diagnosis report and a molecular detection report corresponding to each full-slice image, and preprocessing the full-slice image to obtain a local block image;

[0064] Further preferably, step S1 comprises:

[0065] S101, obtaining histopathological full-slice images with labels indicating lung cancer gene mutation types, as well as pathological diagnosis reports and molecular detection reports corresponding to each full-slice image, to form a data set G, G = {(H 1 ,P 1 ,Z 1 ),(H 2 ,P 2 ,Z 2 ),...,(H i ,P i ,Z i ),...,(H N ,P N ,Z N )}, where H i represents the i-th histopathological full-slice image, P i represents the pathological diagnosis report corresponding to the i-th tissue pathology full-slice image, Z i represents the molecular detection report corresponding to the i-th histopathological full-slice image, and N represents the number of samples in the dataset;

[0066] S102, dividing the full slice image in the data set G into fixed-size block images according to the non-overlapping sliding window strategy, and removing the background block images without tissue, to obtain the local block images H i , where p i,j represents the jth block image in the i-th tissue pathology full slide image, M i Represents the total number of block images divided from the i-th histopathology full-slice image.

[0067] In some optional embodiments, the data used is for non-small cell carcinoma in lung cancer, and is divided into two data sets, namely, a public data set (TCGA-EGFR) from The Cancer Genome Atlas (TCGA) and an internal data set (USTC-EGFR) from the First Affiliated Hospital of University of Science and Technology of China (USTC). For the public data set (TCGA-EGFR), it contains two types of whole slice images: 589 wild type (Wild) and 106 mutant (Mutant); for the internal data set (USTC-EGFR), it includes 114 whole slices of exon 19 deletion mutation type (19del), 181 whole slices of missense mutation type (L858R) of exon 21, 129 whole slices of wild type (Wild) and 133 whole slices of other mutation types (Others), and the size of the block images divided by the whole slices is 256×256, as shown in FIG. Figure 3As shown, the labels of all data are confirmed by professional pathologists, 70% of all data are used as training sets, and 30% of the data are used for testing.

[0068] S2, encoding each of the local block images into a feature vector and then performing pooling aggregation to obtain a feature vector of the full slice image;

[0069] Further preferably, step S2 comprises:

[0070] Training a block-wise image encoder E based on self-supervised learning image , the block image encoder E image Encode the local block image as d s dimensional feature vector, i.e. The feature vector of the whole slice image is represented as the aggregated

[0071] In some optional embodiments, the self-supervised learning training framework used is DINO, the block image encoder f is ViT-S, the dimension of the encoded feature vector dim=384, and the training samples are block images sampled from the full slice image of the training set.

[0072] S3, performing word segmentation and feature extraction on the pathology diagnosis report and the molecular detection report to obtain a report feature vector and a word segmentation feature vector of the pathology diagnosis report, and a report feature vector of the molecular detection report;

[0073] Further preferably, step S3 comprises:

[0074] S301. Use pre-trained text feature encoder E text The pathological diagnosis report i Encode to get the complete report feature vector and word segmentation feature vector Where N i Indicates report P i The length of the word segmentation feature vector in E text Molecular test report Z i Encode to get the complete report feature vector

[0075] S302, using pre-trained text feature encoder E text Molecular test report Z i Encode to get the complete report feature vector

[0076] In some optional embodiments, the text pre-training framework used is BERT, and the dimension of the encoded feature vector dim=768.

[0077] S4. Establish the connection between the pathology diagnosis report and the local area of ​​the whole slice image through the cross-attention mechanism to reconstruct the overall semantics of the pathology diagnosis report; implement contrastive learning based on category refinement queue, cross-modally fuse the reconstructed pathology report features with the full slice features, and obtain a cross-modal representation learning model;

[0078] Further preferably, step S4 comprises:

[0079] S401, establish the connection between the word segmentation feature vector in the pathology diagnosis report and the local block image feature vector in the full slice image through the cross attention mechanism, and obtain the word segmentation feature vector W′ i , as follows:

[0080]

[0081] Q t =W i A Q , K t =X i A K , V t =X i A V

[0082] Among them A Q , A K , A V are all learnable weight matrices, d is the dimension of the input feature 512; the reconstructed word segmentation feature vector Obtaining the complete text feature vector through attention pooling The significance of text reconstruction is to encourage the model to learn the close connection between modalities and better understand the implicit information expression of the whole slice through the guidance of text information;

[0083] S402. Establish a queue for each lung cancer gene mutation label to store report features; regard the originally extracted pathology report features and the reconstructed report features that match them as positive samples, and the reconstructed features in other categories of queues as negative samples; for the reconstructed pathology report features, the original report features that match them are regarded as positive samples, and the original features in other categories of queues are regarded as negative samples; through contrastive learning, align the semantics of the fusion features with the molecular detection supervision information, as shown in the following formula:

[0084] Q = {q 0 ,q 1 , ..., q C-1}

[0085] P = {T′ i , T i}

[0086] Listr =Concat(q 0 ,q 1 , ..., q C-2 , T′ i )

[0087] List p =Concat(q 0 ,q 1 , ..., q C-2 , T i )

[0088] List r =List r / List r .norm()

[0089] List p =List p / List p .norm()

[0090] logits r =np.dot(List r , List p .T)*np.exp(T)

[0091] logits p =logits r .T

[0092] L contrastive =CE(logits r [C-1, C-1)+CE(logits p [C-1],C-1)

[0093] Where Q is the queue for storing feature sample pairs, P is the positive sample pair in the current batch, t is a learnable temperature coefficient, CE(·) is the cross entropy loss function, and each training iteration process performs contrastive learning. At the end of the iteration, the batch of sample pairs is queued to update the queue. The reconstruction report loss is expressed as

[0094] S403, reconstructing the pathology report feature T′ i With full slice feature S i Fusion obtains cross-modal fusion features Obtain a cross-modal representation learning model, based on the contrastive learning alignment F in S402 i and O i , we get the molecular semantic contrast loss The fusion features are semantically aligned with the molecular detection supervision information.

[0095] S5. Design a loss function including classification, reconstruction and molecular semantic contrast loss, use the labeled data set to train the cross-modal representation learning model, and use the trained cross-modal representation learning model to predict the gene mutation type of the whole tissue pathology slice image to be predicted.

[0096] Further preferably, in step S5, the design includes a loss function of classification, reconstruction and molecular semantic contrast loss, including:

[0097] Setting the loss function to include classification prediction loss Rebuilding reported losses and molecular semantic contrast loss The sum of , where the loss function

[0098]

[0099] Where CE represents the cross entropy loss, y i Indicates the label corresponding to the full slice;

[0100]

[0101] According to the above-mentioned embodiments of the present invention, the prediction method proposed in the present invention can overcome the difficulty in obtaining detailed annotations of full-slice image data of tissue pathology and the insufficient ability to express single-modal information, and use the cross-modal information of pathology reports and molecular detection reports to guide the weak supervision method to effectively complete the mutation label classification at the slice image level.

[0102] Combination Figure 5 As shown, based on the same inventive concept, the present invention also proposes a gene mutation prediction system based on pathology report and full-slice cross-modal representation learning, which is applied to execute the above gene mutation prediction method. The system includes

[0103] A data acquisition module is used to acquire histopathological full-slice images with image-level labels, as well as pathological diagnosis reports and molecular detection reports corresponding to each full-slice image, to form a data set;

[0104] A preprocessing module is used to preprocess the full slice image to obtain a local block image;

[0105] An image encoding module is used to encode each local block image into a feature vector and then perform pooling aggregation to obtain a feature vector of the full slice image;

[0106] A text encoding module is used to perform word segmentation and feature extraction on the pathology diagnosis report and the molecular detection report to obtain a report feature vector and a word segmentation feature vector of the pathology diagnosis report, and a report feature vector of the molecular detection report;

[0107] The cross-modal learning module is used to establish the connection between the pathology diagnosis report and the local area of ​​the full-slice image through the cross-attention mechanism, reconstruct the overall semantics of the pathology diagnosis report, and implement contrastive learning based on the category refinement queue to cross-modally fuse the reconstructed pathology report features with the full-slice features to obtain a cross-modal representation learning model;

[0108] The training and prediction module is used to design a loss function including classification, reconstruction and molecular semantic contrast loss, use the labeled data set to train the cross-modal representation learning model, and use the trained cross-modal representation learning model to predict the gene mutation type of the whole tissue pathology slice image to be predicted.

[0109] It should be noted here that each module in the above-mentioned mutation prediction system corresponds to steps S1 to S5 in implementing the above-mentioned optimization method, and the instances and application scenarios implemented by multiple modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment 1.

[0110] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned gene mutation prediction methods are implemented.

[0111] In another embodiment provided by the present invention, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of any gene mutation prediction method in the above embodiments.

[0112] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0113] In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0114] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A gene mutation prediction method based on pathology report and whole-slice cross-modal representation learning, characterized in that: The method comprises: S1, obtaining a tissue pathology full-slice image with image-level labels, as well as a pathology diagnosis report and a molecular detection report corresponding to each full-slice image, and preprocessing the full-slice image to obtain a local block image; S2, encoding each of the local block images into a feature vector and then performing pooling aggregation to obtain a feature vector of the full slice image; S3, performing word segmentation and feature extraction on the pathology diagnosis report and the molecular detection report to obtain a report feature vector and a word segmentation feature vector of the pathology diagnosis report, and a report feature vector of the molecular detection report; S4. Establish the connection between the pathology diagnosis report and the local area of ​​the whole slice image through the cross-attention mechanism to reconstruct the overall semantics of the pathology diagnosis report; implement contrastive learning based on category refinement queue, cross-modally fuse the reconstructed pathology report features with the full slice features, and obtain a cross-modal representation learning model; S5. Design a loss function including classification, reconstruction and molecular semantic contrast loss, use the labeled data set to train the cross-modal representation learning model, and use the trained cross-modal representation learning model to predict the gene mutation type of the whole tissue pathology slice image to be predicted.

2. According to claim 1, a gene mutation prediction method based on pathology report and whole-slice cross-modal representation learning is characterized in that: Step S1 includes: S101, obtaining histopathological full-slice images with labels indicating lung cancer gene mutation types, as well as pathological diagnosis reports and molecular detection reports corresponding to each full-slice image, to form a data set G, G = {(H1, P1, Z1), (H2, P2, Z2), ..., (H i ,P i ,Z i ),...,(H N ,P N ,Z N )}, where H i represents the i-th histopathological full-slice image, P i represents the pathological diagnosis report corresponding to the i-th tissue pathology full-slice image, Z i represents the molecular detection report corresponding to the i-th histopathological full-slice image, and N represents the number of samples in the dataset; S102, dividing the full slice image in the data set G into fixed-size block images according to the non-overlapping sliding window strategy, and removing the background block images without tissue, to obtain the local block images H i , where p i,j represents the jth block image in the i-th tissue pathology full slide image, M i Represents the total number of block images divided from the i-th histopathology full-slice image.

3. A gene mutation prediction method based on pathology report and whole-slice cross-modal representation learning according to claim 2, characterized in that: Step S2 includes: Training a block-wise image encoder E based on self-supervised learning image , the block image encoder E image Encode the local block image as d s dimensional feature vector, i.e. The feature vector of the whole slice image is represented as the aggregated 4. A gene mutation prediction method based on pathology report and whole-slice cross-modal representation learning according to claim 3, characterized in that: Step S3 includes: S301. Use pre-trained text feature encoder E text The pathological diagnosis report i Encode to get the complete report feature vector and word segmentation feature vector Where N i Indicates report P i The length of the word segmentation feature vector in E text Molecular test report Z i Encode to get the complete report feature vector S302, using the pre-trained text feature encoder E text Molecular test report Z i Encode to get the complete report feature vector 5. A gene mutation prediction method based on pathology report and whole-slice cross-modal representation learning according to claim 4, characterized in that: Step S4 includes: S401, establish the connection between the word segmentation feature vector in the pathology diagnosis report and the local block image feature vector in the full slice image through the cross attention mechanism, and obtain the word segmentation feature vector W' i , as follows: Q t =W i A Q ,K t =X i A K ,V t =X i A V Among them A Q , A K , A V are all learnable weight matrices, d is the dimension of the input feature 512; the reconstructed word segmentation feature vector Obtaining the complete text feature vector through attention pooling S402. Establish a queue for each lung cancer gene mutation label to store report features; regard the originally extracted pathology report features and the reconstructed report features that match them as positive samples, and the reconstructed features in other categories of queues as negative samples; for the reconstructed pathology report features, the original report features that match them are regarded as positive samples, and the original features in other categories of queues are regarded as negative samples; through contrastive learning, align the semantics of the fusion features with the molecular detection supervision information, as shown in the following formula: Q={q0,q1,…,q C-1 } P={T′ i ,T i } List r =Concat(q0,q1,...,q C-2 ,T′ i ) List p =Concat(q0,q1,...,q C-2 ,T i ) List r =List r / List r .norm() List p =List p / List p .norm() logits r =np.dot(List r ,List p .T)*np.exp(t) logits p =logits r .T L contrastive =CE(logitsr[C-1,C-1)+CE(logits p [C-1],C-1) Where Q is the queue for storing feature sample pairs, P is the positive sample pair in the current batch, t is a learnable temperature coefficient, CE(·) is the cross entropy loss function, and each training iteration process performs contrastive learning. At the end of the iteration, the batch of sample pairs is queued to update the queue. The reconstruction report loss is expressed as S403, reconstructing the pathology report feature T' i With full slice feature S i Fusion obtains cross-modal fusion features Obtain a cross-modal representation learning model, based on the contrastive learning alignment F in S402 i and O i , we get the molecular semantic contrast loss The fusion features are semantically aligned with the molecular detection supervision information.

6. A gene mutation prediction method based on pathology report and whole-slice cross-modal representation learning according to claim 5, characterized in that: In step S5, the design includes loss functions for classification, reconstruction and molecular semantic contrast loss, including: Setting the loss function to include classification prediction loss Rebuilding reported losses and molecular semantic contrast loss The sum of , where the loss function Where CE represents the cross entropy loss, y i Indicates the label corresponding to the full slice; 7. A gene mutation prediction system based on pathology report and whole-slice cross-modal representation learning, characterized in that: Applied to executing the gene mutation prediction method according to any one of claims 1 to 6, the system comprises: A data acquisition module is used to acquire histopathological full-slice images with image-level labels, as well as pathological diagnosis reports and molecular detection reports corresponding to each full-slice image, to form a data set; A preprocessing module is used to preprocess the full slice image to obtain a local block image; An image encoding module is used to encode each local block image into a feature vector and then perform pooling aggregation to obtain a feature vector of the full slice image; A text encoding module is used to perform word segmentation and feature extraction on the pathology diagnosis report and the molecular detection report to obtain a report feature vector and a word segmentation feature vector of the pathology diagnosis report, and a report feature vector of the molecular detection report; The cross-modal learning module is used to establish the connection between the pathology diagnosis report and the local area of ​​the full-slice image through the cross-attention mechanism, reconstruct the overall semantics of the pathology diagnosis report, and implement contrastive learning based on the category refinement queue to cross-modally fuse the reconstructed pathology report features with the full-slice features to obtain a cross-modal representation learning model; The training and prediction module is used to design a loss function including classification, reconstruction and molecular semantic contrast loss, use the labeled data set to train the cross-modal representation learning model, and use the trained cross-modal representation learning model to predict the gene mutation type of the whole tissue pathology slice image to be predicted.

Images