Multi-modal information fusion method, system and equipment for large language model of pathological full slice and medium

Abstract

The invention discloses a multi-modal information fusion method, system and device for a large language model oriented to pathological full slices, and a medium. The obtained coding vector of the local representation of the pathological image region block is input into a pathological image information mixed attention layer to be fused to obtain a representation vector of a full slice of the pathological image; performing multi-modal fusion on the obtained representation vector of the full slice of the pathological image and the text vector of the large language model through a cross attention mechanism; the system, the equipment and the medium are based on the multi-modal information fusion method of the pathological full-slice-oriented language large model, and multi-modal information fusion is carried out according to pathological image information and text information input by a user; according to the invention, doctors are helped to carry out quicker and more accurate pathological judgment.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal large models of pathological images at the intersection of artificial intelligence and medicine, and specifically relates to a multimodal information fusion method, system, equipment and medium for a language large model of pathological full slices. Background Art

[0002] In the past decade, deep learning and convolutional neural networks have shown outstanding advantages in the field of computer vision and have been widely used in medical image analysis. The use of deep learning and computer vision technology can help doctors quickly identify diseased cells, thereby greatly improving the efficiency of doctors' diagnosis and treatment, assisting doctors in various clinical diagnoses, and achieving good results in the classification and segmentation of pathological images. However, pathological images are large in size and high in resolution. When processing pathological whole slide images (WSI), the existing technology often uses a patch-by-patch processing method, and finally takes the average value of the output of each patch. This processing method does not fully utilize the information of the pathological image, and there is information loss between different pathological image blocks, which is not conducive to making a complete diagnosis or analysis of the pathological image.

[0003] In addition, the field of clinical pathology naturally has multimodal characteristics. In the process of diagnosis by doctors, it is often necessary to combine text information, pathological images, and various other test results for patients to make a comprehensive judgment. The existing methods of pathological image processing are mainly reflected in the application of artificial intelligence in the field of computer vision. They do not utilize the multimodal characteristics of the pathological field itself, which leads to the inability to fully utilize multimodal information such as language in clinical diagnosis and the lack of effective multimodal data fusion mechanism. This limits the application ability of the model in simulating actual diagnostic scenarios. With the development of artificial intelligence technology and hardware computing power, the number of parameters of artificial intelligence models continues to increase, showing enhanced capabilities. ChatGPT and Llama have shown strong capabilities in the field of natural language processing. The large language model has demonstrated capabilities that exceed the original model in a variety of natural language processing tasks, and the large language model has the potential to establish a general model architecture between different tasks. The main application scenario of the existing large model is natural language processing. Applying the capabilities of the existing language large model to the pathological scenario can combine multiple modalities to improve the analysis results of pathological data. However, the existing generative large models in the field of pathology mainly use methods based on a single pathological local area image to carry out predictions and dialogues. In order to expand the input of pathological images to the level of full pathological slices, for example, the paper "MED-FLAMINGO: AMULTIMODAL MEDICAL FEW SHOT LEARNER" and the paper "A multimodal generative AI copilot for human pathology" published in the journal Nature both used methods based on local area pre-training to construct large models in the field of pathology, and failed to integrate the global information of the entire pathological image. Summary of the invention

[0004] In order to overcome the defects of the above-mentioned existing technologies, the present invention proposes a multimodal information fusion method, system, device and medium for a large language model of a full pathological slice. By segmenting and processing the pathological image, a coding vector for a local representation of the pathological image region block is obtained: the coding vector for the local representation of the pathological image region block is input into the pathological image information mixed attention layer for fusion to obtain the representation vector of the full pathological slice; then the representation vector of the full pathological slice is obtained and the text vector of the large language model is used to realize multimodal fusion through a cross-attention mechanism; this helps to assist doctors in making faster and more accurate pathological judgments.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A multimodal information fusion method for a large language model of a pathological full slide includes the following steps:

[0007] Step 1: Segment and process the pathological image: Given a complete pathological slice, divide the pathological slice into fixed window sizes. After segmentation, encode each local area through a visual encoder to obtain the encoding vector of the local representation of the pathological image area block.

[0008] Step 2: Full slice information fusion of pathological images: The encoding vector of the local representation of the pathological image area block obtained in step 1 is converted into The input pathological image information is mixed with the attention layer to obtain the representation vector of the whole slice of the pathological image, and the information bottleneck potential representation module r is introduced to encode the local representation encoding vector of the pathological image area block The compressed dimension is used as a fixed-length representation vector of the full slice of the pathological image;

[0009] Step 3: Multimodal fusion of the large language model and the whole slice information: introduce the large language model, and the input of the large language model is text content. The text is transmitted between different levels of the large language model in the form of vectors; freeze the weights of each layer of the original large language model, and establish a trainable modal mixing layer between the layers of the large language model, and train the parameters of the unfrozen part, that is, the representation vector of the full slice of the pathological image obtained in step 2 and the text vector of the large language model are cross-attended through tanh gate to achieve multimodal fusion and obtain the modal fusion vector; use the modal fusion vector to generate the final answer of the large language model.

[0010] The specific method of step 1 is:

[0011] In step 1, a full pathological slice of a patient containing complete information is segmented according to a fixed step size and window size, and the center is cropped to obtain n pathological image area blocks, namely patches; the set representation of each pathological image area block is obtained, and after cropping and normalization, it is represented as X = {x1, x2, …, x n}Pathology image collection;

[0012] In step 2), the pathological image set obtained in step 1) is input into the pre-trained visual encoder to obtain the encoding vector of the local representation of each pathological image region block. The visual encoder adopts the ResNet model trained on a fixed data set, or the CLIP model obtained by contrastive learning; the encoding vector of the local representation of the pathological image region block obtained after the entire pathological image region block is processed by the visual encoder is recorded as Wherein, n is the number of pathological image region blocks cut from the pathological image.

[0013] The specific method of step 2 is:

[0014] The encoding vector of the local representation of the pathological image region block obtained in step 2) of step 1 is The pathological image information is input into the mixed attention layer to carry out the full slice information fusion process of the pathological image; the pathological image information mixed attention layer includes an information bottleneck potential representation module r and a pathological image region block local representation encoding vector Fusion cross attention layer; information bottleneck potential representation module r and pathological image area block local representation encoding vector Splice as the query vector query of the cross attention layer, and use the local representation encoding vector of the pathological image area block As the key and value in the cross-attention layer, the cross-attention layer is used to achieve the fusion effect of the entire pathological image, and the result after the cross-attention layer is summed with the original information bottleneck potential representation module r using the residual; the result is then passed through the forward propagation layer neural network (Feed-Forward Network, FFW) and the residual to obtain the final fusion vector as the representation vector of the full slice of the pathological image with a fixed length.

[0015] The information bottleneck potential representation module r is used to compress the local representation encoding vector of the pathological image area block processed by the visual encoder. The information in the pathological image area is reduced to the local representation encoding vector dimension; the information bottleneck potential representation module r is a dimension The learnable vector, the value of the information bottleneck potential representation vector is recorded as

[0016] Used for the full slice information fusion process.

[0017] The full slice information fusion of the pathological image is expressed by the formula:

[0018]

[0019] V=V'+FFW(V')

[0020] Among them, V is the representation vector of the whole slice of the pathological image, are the learnable projection parameters of the cross-attention layer in the process of building full slices of pathological images.

[0021] The specific method of step three is:

[0022] The representation vector V of the full slice of the pathological image finally obtained in step 2 is combined with the vector L = {l1,l2,…,l t-1} Modal fusion is achieved through tanh-gated cross-attention; the vector L of the text generated by the large language model itself before time t is used as the query vector query of the cross-attention layer, and the representation vector V of the full slice of the pathological image is used as the key and value in the cross-attention layer, and then the result is gated by tanh and the residual is summed; the result is then passed through the tanh-gated forward propagation layer neural network and the residual is summed to obtain a multimodal fusion vector; it can be expressed as:

[0023]

[0024] L merge =L'+tanh(ffw_para)·FFW(L)

[0025] Among them, attn_para and ffw_para are both trainable scalar parameters of tanh gate with initialization value 0, L merge is the vector transmitted in the large language model after multimodal fusion, are the learnable parameters of the attention mechanism in the modality fusion process;

[0026] The loss function in the process of multimodal fusion can be expressed as:

[0027]

[0028] Among them, l <t refers to the answer content generated by the large model before time t, p(l t |l <t ,V) refers to the probability of the real text output corresponding to position t under the condition of the full slice representation vector V of the given pathological image and the answer content generated by the large language model before time t.

[0029] The large language model adopts a Decoder-only generative large language model, including Llama and TeleChat.

[0030] A multimodal information fusion system for a large language model of pathological full slides, including:

[0031] The pathological image segmentation and processing module is used in step 1 to segment the pathological image into local area image blocks and encode them through a visual encoder to obtain a coding vector representing the local area block of the pathological image.

[0032] The pathological image full slice information fusion module is used in step 2 to convert the encoding vector of the local representation of the pathological image area block obtained in step 1 into The input pathological image information is mixed with the attention layer to obtain the representation vector of the whole slice of the pathological image. By introducing the information bottleneck potential representation module r, the local representation encoding vector of the pathological image area block is converted into The compressed dimension is used as a fixed-length representation vector of the full slice of the pathological image;

[0033] The multimodal fusion module of the large language model and the full slice information, in step three, uses the cross-attention mechanism to fuse the representation vector of the full slice of the pathological image obtained in step two with the text vector of the large language model, and obtains a modal fusion vector to generate the answer of the large language model.

[0034] A multimodal information fusion device for a language large model of a pathological full slice comprises: a memory and a processor, wherein the memory stores a computer program so that the processor executes the multimodal information fusion method for the language large model of a pathological full slice.

[0035] A storage medium for receiving user input program, wherein when the stored computer program is executed by a processor, it can perform multimodal information fusion based on the multimodal information fusion method of the language large model for pathological full slices according to pathological image information and text information input by the user.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The method of the present invention realizes the segmentation and processing of the pathological image and the fusion of the whole slice information through step one and step two. The designed pathological image segmentor and the pre-trained visual encoder can model the representation vectors of each area of ​​the pathological image. The designed pathological image information mixed attention layer can use the representation vectors of each area of ​​the pathological image and the potential representation module of the information bottleneck to establish the representation vector for the entire pathological image; this processing method creates conditions for the subsequent multimodal data fusion module, and has the advantages of aggregating the global information of the pathological slice and enhancing the vertical domain capability of the model.

[0038] 2. The method of the present invention fuses the representation vector of the entire pathological image in step 2 with the text vector of the large language model through step 3, and saves the content of the original large model vector in the initial state through gated residual cross-attention, and uses the cross-attention mechanism to establish a fusion vector of image information and text information, and then adjusts the parameters of the model gating unit during the optimization process of the model to achieve the fusion of text input information and image input information; after the model generates an answer, the loop dialogue can be completed by providing the model with the historical input of the model, thereby realizing a multimodal large model information mixing technology based on the full pathological slice, which has the advantage of injecting multimodal data processing capabilities into the large language model.

[0039] 3. The multimodal large model information mixing technology based on the whole pathology slice simulates the comprehensive diagnosis process of doctors using multiple modal information in real clinical scenarios during the analysis of pathology slices, and utilizes the capabilities of existing visual and language large models along with the improvement of model parameters; the visual encoding module based on the whole pathology slice can establish connections between different pathology slice areas, and integrate the information of different regions of the whole pathology image into the representation vector of the whole slice; after obtaining the pathology image information, the gating unit containing cross-attention is used to realize the multimodal fusion of the language model input containing the processing results of the whole pathology report and the image information; it has the advantages of comprehensively utilizing the multimodal characteristics of pathology and conforming to the actual clinical judgment situation.

[0040] In summary, the present invention divides the pathological image into local areas and encodes them using a visual encoder, then uses the information bottleneck to fuse the local representations of each pathological slice into a global representation of the pathological slice, and then fuses them with the original vectors in the large language model through gated cross-attention. In this way, a multimodal large model for all pathological slices can be realized, which has the advantage of aggregating global information of pathological slices, enhancing the vertical domain capability of the model, injecting multimodal data processing capabilities into the large language model, and utilizing the multimodal features in pathological medical scenarios, which helps to assist doctors in making faster and more accurate pathological judgments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the pathological full-slice processing module in steps one and two of the present invention.

[0042] Figure 2 It is a schematic diagram of the multimodal fusion module of the three major language models and the full slice information in the steps of the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0044] A multimodal information fusion method for a large language model of a pathological full slide includes the following steps:

[0045] See also Figure 1 ,Step 1: Segment and process the pathological image: Given a complete pathological full slice, the pathological full slice is segmented according to a fixed window size. After the segmentation is completed, the segmented local areas are encoded through a visual encoder to obtain the encoding vector of the local representation of the pathological image area block

[0046] In step 1), a full pathology slice of a patient containing complete information is segmented according to a fixed step size and window size, which can be adjusted according to the usage scenario. Usually 512 or 1024 pixels are used, and the center is cropped to obtain n pathology image area blocks, i.e., patches; the set representation of each pathology image area block is obtained, and after cropping and normalization, it is represented as X = {x1, x2, …, x n}Pathology image collection;

[0047] In step 2), the pathological image set obtained in step 1) is input into the pre-trained visual encoder to obtain the encoding vector of the local representation of each pathological image region block. The visual encoder adopts the ResNet model trained on a fixed data set, or the CLIP model obtained by contrastive learning; the encoding vector of the local representation of the pathological image region block obtained after the entire pathological image region block is processed by the visual encoder is recorded as Wherein, n is the number of pathological image region blocks cut from the pathological image.

[0048] Step 2: Full slice information fusion of pathological images: The encoding vector of the local representation of the pathological image area block obtained in step 1 is converted into The input pathological image information is mixed with the attention layer to obtain the representation vector of the whole slice of the pathological image for subsequent model training, and the information bottleneck potential representation module r is used to encode the local representation encoding vector of the pathological image area block The compressed dimension is used as a fixed-length representation vector of the full slice of the pathological image;

[0049] The encoding vector of the local representation of the pathological image region block obtained in step 2) of step 1 is The pathological image information is input into the mixed attention layer to carry out the full slice information fusion process of the pathological image; the pathological image information mixed attention layer includes an information bottleneck potential representation module r and a pathological image region block local representation encoding vector Fusion cross attention layer; information bottleneck potential representation module r and pathological image area block local representation encoding vector Splice as the query vector query of the cross attention layer, and use the local representation encoding vector of the pathological image area block As the key and value in the cross-attention layer, the cross-attention layer is used to achieve the fusion effect of the entire pathological image, and the result after the cross-attention layer is summed with the original information bottleneck potential representation module r using the residual; the result is then passed through the forward propagation layer neural network (Feed-Forward Network, FFW) and the residual to obtain the final fusion vector as the representation vector of the full slice of the pathological image with a fixed length.

[0050] The information bottleneck potential representation module r is used to compress the local representation encoding vector of the pathological image area block processed by the visual encoder. The information in the pathological image area is reduced to the local representation encoding vector dimension; the information bottleneck potential representation module r is a dimension The learnable vector is denoted by Used for the full slice information fusion process.

[0051] The full slice information fusion of the pathological image is expressed by the formula:

[0052]

[0053] V=V'+FFW(V')

[0054] Among them, V is the representation vector of the whole slice of the pathological image, are the learnable projection parameters of the cross-attention layer in the process of building full slices of pathological images.

[0055] See also Figure 2 , Step 3: Multimodal fusion of large language model and full slice information: introduce the decoder-only large language model (Llama, TeleChat), input as text content, and the text is transmitted between different levels of the large language model in the form of vectors; freeze the weights of each layer of the original decoder-only large language model, establish a trainable modal mixing layer between the layers of the decoder-only large language model, and train the parameters of the unfrozen part, that is, the representation vector of the full slice of the pathological image obtained in step 2 and the text vector of the decoder-only large language model are cross-attended by tanh gate to achieve modal fusion, and obtain the modal fusion vector; use the modal fusion vector to generate the final answer of the decoder-only large language model;

[0056] The representation vector V of the full slice of the pathological image finally obtained in step 2 is combined with the vector L = {l1,l2,…,l t-1} Modal fusion is achieved through tanh-gated cross attention; the vector L of the text generated by the decoder-only large language model before time t is used as the query, and the representation vector V of the full slice of the pathological image is used as the key and value, and then the result is gated by tanh and the residual is summed; the result is then passed through the tanh-gated forward propagation layer neural network and the residual is summed to obtain a multimodal fusion vector; it can be expressed as:

[0057]

[0058] L merge =L'+tanh(ffw_para)·FFW(L)

[0059] Among them, attn_para and ffw_para are both trainable scalar parameters of tanh gate with initialization value 0, L merge is the vector transmitted in the Decoder-only large language model after multimodal fusion. are the learnable parameters of the attention mechanism in the modality fusion process;

[0060] The loss function in the process of multimodal fusion can be expressed as:

[0061]

[0062] Among them, l <t refers to the answer content generated by the Decoder-only large language model before time t, p(l t |l <t ,V) refers to the probability of the real text output corresponding to the position t under the condition of the full slice representation vector V of the given pathological image and the answer content generated by the large model before time t.

[0063] The present invention is further illustrated by specific examples.

[0064] Hardware equipment: a server equipped with a high-performance GPU for running deep learning models. In this example, NVIDIA A100 is used; a pathology image scanner for acquiring pathology full-slice images. In this example, Olympus VS-M1R is used.

[0065] Model parameters: pre-trained visual encoder model, clip-vit-base-patch16 is used in this embodiment; large language model, Llama-7b is used in this embodiment.

[0066] Software environment: Python programming language; PyTorch deep learning framework; corresponding software packages that support Llama and clip-vit-base-patch16; relevant software packages that support reading pathological images are configured according to actual needs.

[0067] Data: A dataset containing multiple pathological full-slice images and their corresponding text descriptions. This example uses the TCGA pathological image dataset.

[0068] Implementation steps:

[0069] Pathological image acquisition: According to actual usage, the patient's pathological full-slice image is acquired from a pathological image scanner, or pathological images are acquired from the Cancer Genome Atlas Program (TCGA) database. In the implementation process of this embodiment, 12,560 cancer pathological slices from the TCGA dataset and 25,713 non-pathological tissue slices from the Genotype-Tissue Expression dataset are used as training content.

[0070] Step 1: 1) Pathological image segmentation and processing: For each pathological slice, the acquired pathological full slice image is segmented according to a window size of 512x512 pixels, and the step size in the segmentation process is 256, and a series of pathological image patches are obtained. After cropping and normalization, the pathological image patch set representation is obtained, which is represented by X = {x1, x2, …, x n}.

[0071] 2) Input the pathological image set obtained in step 1) of step 1 into the pre-trained visual encoder to obtain the encoding vector of the local representation of each pathological image region block, input the segmented pathological image region block into the pre-trained visual encoder clip-vit-base-patch16, and record the encoding vector of the local representation of the pathological image region block obtained after the entire pathological image region block is processed by the visual encoder as Wherein, n is the number of pathological image region blocks into which the pathological image is segmented.

[0072] Step 2: Full slice information fusion of pathological images: The encoding vector of the local representation of the pathological image area block obtained in step 1 is converted into The input pathological image information is integrated into the hybrid attention layer to obtain the representation vector of the full slice of the pathological image for subsequent model training, and the information bottleneck potential representation module r is used to encode the local representation encoding vector of the pathological image region block The compressed dimension is used as the representation vector of the full slice of the pathological image with a fixed length. The potential module of the information bottleneck r is set to a learnable parameter with a dimension of 32*768. The torch.nn.MultiheadAttention function included in the PyTorch deep learning framework is used to implement the cross-attention process of the potential representation of the information bottleneck and the local representation encoding vector of the pathological image region block. The result obtained by the cross-attention is passed through the residual structure and the forward neural network propagation process as the representation vector of the full slice of the pathological image. The output dimension of the forward propagation layer is set to 4096 dimensions that conform to the Llama internal vector transfer dimension. The full slice information fusion of the pathological image is expressed by the formula:

[0073]

[0074] V=V'+FFW(V')

[0075] Among them, V is the representation vector of the whole slice of the pathological image, are the learnable projection parameters of the cross-attention layer in the process of building full slices of pathological images.

[0076] Step 3: Multimodal fusion of large language model and full slice information: The obtained representation vector V of the full slice of the pathological image is combined with the vector L = {l1,l2,…,l t-1} Modal fusion is achieved through tanh-gated cross-attention; and a modal fusion vector is obtained. Llama-7b consists of 32 processing layers. In this embodiment, a modal mixing module is inserted between the corresponding Llama-7b processing layers every 8 layers to introduce the information of the pathological image into the large language model to obtain a modal fusion vector. The vector L of the text generated by the Decoder-only large language model itself before time t is used as the query, and the representation vector V of the full slice of the pathological image is used as the key and value, and then the result is gated by tanh and the residual is summed; the result is then passed through the tanh-gated forward propagation layer neural network and the residual is summed to obtain a multimodal fusion vector; the mixing process of each modal mixing module is expressed by the formula:

[0077]

[0078] L merge =L'+tanh(ffw_para)·FFW(L)

[0079] Among them, attn_para and ffw_para are both trainable scalar parameters of tanh gate with initialization value 0, L merge is the vector transmitted in the Decoder-only large language model after multimodal fusion. is the learnable parameter of the attention mechanism in the modality fusion process

[0080] The initial value of the tanh gating parameter of each modality mixing module is 0 to ensure that the process starts with the original transfer value of the large language model. The text vector generated by the large language model and the representation vector of the full pathological slice are obtained through cross attention, forward neural network and tanh gating to obtain the modality fused vector, and the modality fusion vector is continued to be propagated layer by layer in the remaining layers of the large language model to generate the final text answer. The loss function in the process of multimodal fusion can be expressed as:

[0081]

[0082] Among them, l <t refers to the answer content generated by the Decoder-only large language model before time t, p(l t |l <t ,V) refers to the probability of the real text output corresponding to the position t under the condition of the full slice representation vector V of the given pathological image and the answer content generated by the large model before time t.

[0083] According to the detailed steps of the above embodiment, the pathological section "TCGA-06-0178-01Z-00-DX4" contained in the TCGA pathological section data set is used as the pathological image input, and the question "What is the main diagnosis in this pathological section?" is input in the initial text input of the large language model. After three steps of pathological image segmentation and processing, full-slice information fusion of the pathological image, and multimodal fusion of the large language model and the full-slice information, the output of the model is: "Based on the results shown in the pathological image, the preliminary diagnosis of this pathological image is glioblastoma." The multimodal information fusion method of the language large model of the pathological full section proposed in the present invention can realize the input and analysis of the pathological full section and the fusion of information of multiple modalities.

[0084] Clinical application potential: The embodiments of the present invention demonstrate the potential of this technology in clinical diagnosis, which helps to assist doctors in making faster and more accurate pathological diagnoses.

[0085] It can be seen that this embodiment demonstrates the specific application process of the present invention in the field of pathological image analysis. The innovation of the present invention is mainly reflected in the proposal of a processing and information fusion method for the whole pathological slice; an information fusion method using tanh gated cross attention to realize the multimodal fusion of a large language model and the whole slice information. Compared with the existing methods for the analysis of local pathological regions, the present invention proposes a new processing flow and information fusion method for the whole pathological slice. The present invention divides the pathological image into local regions and encodes them using a visual encoder, and then uses the information bottleneck to fuse the local representation of each pathological slice into a global representation of the pathological slice, and then fuses the original vector in the large language model through gated cross attention, thereby realizing a multimodal large model for the whole pathological slice, which has the ability to aggregate the global information of the pathological slice, enhance the vertical domain capability of the model, inject multimodal data processing capabilities into the large language model, and utilize the innovativeness of the multimodal features in the pathological medical scene, which helps to assist doctors in making faster and more accurate pathological judgments.

Claims

1. A multimodal information fusion method for a large language model of pathological full sections, characterized by: The specific steps include the following: Step 1: Segment and process the pathological image: Given a complete pathological slice, divide the pathological slice into fixed window sizes. After segmentation, encode each local area through a visual encoder to obtain the encoding vector of the local representation of the pathological image area block. Step 2: Full slice information fusion of pathological images: The encoding vector of the local representation of the pathological image area block obtained in step 1 is converted into The input pathological image information is mixed with the attention layer to obtain the representation vector of the whole slice of the pathological image, and the information bottleneck potential representation module r is introduced to encode the local representation encoding vector of the pathological image area block The compressed dimension is used as a fixed-length representation vector of the full slice of the pathological image; Step 3: Multimodal fusion of the large language model and the whole slice information: introduce the large language model, and the input of the large language model is text content. The text is transmitted between different levels of the large language model in the form of vectors; freeze the weights of each layer of the original large language model, and establish a trainable modal mixing layer between the layers of the large language model, and train the parameters of the unfrozen part, that is, the representation vector of the full slice of the pathological image obtained in step 2 and the text vector of the large language model are cross-attended through tanh gate to achieve multimodal fusion and obtain the modal fusion vector; use the modal fusion vector to generate the final answer of the large language model.

2. According to claim 1, a multimodal information fusion method for a language large model of a pathological full slide is characterized in that: The specific method of step 1 is: In step 1, a full pathological slice of a patient containing complete information is segmented according to a fixed step size and window size, and the center is cropped to obtain n pathological image area blocks, namely patches; the set representation of each pathological image area block is obtained, and after cropping and normalization, it is represented as X = {x1, x2, …, x n }Pathology image collection; In step 2), the pathological image set obtained in step 1) is input into the pre-trained visual encoder to obtain the encoding vector of the local representation of each pathological image region block. The visual encoder adopts the ResNet model trained on a fixed data set, or the CLIP model obtained by contrastive learning; the encoding vector of the local representation of the pathological image region block obtained after the entire pathological image region block is processed by the visual encoder is recorded as Wherein, n is the number of pathological image region blocks cut from the pathological image.

3. The multimodal information fusion method for a language large model of a pathological full slide according to claim 1 is characterized in that: The specific method of step 2 is: The encoding vector of the local representation of the pathological image region block obtained in step 2) of step 1 is The pathological image information is input into the mixed attention layer to carry out the full slice information fusion process of the pathological image; the pathological image information mixed attention layer includes an information bottleneck potential representation module r and a pathological image region block local representation encoding vector Fusion cross attention layer; information bottleneck potential representation module r and pathological image area block local representation encoding vector Splice as the query vector query of the cross attention layer, and use the local representation encoding vector of the pathological image area block As the key and value in the cross-attention layer, the cross-attention layer is used to achieve the fusion effect of the entire pathological image, and the result after the cross-attention layer is summed with the original information bottleneck potential representation module r using the residual; the result is then passed through the forward propagation layer neural network (Feed-Forward Network, FFW) and the residual to obtain the final fusion vector as the representation vector of the full slice of the pathological image with a fixed length.

4. A multimodal information fusion method for a language large model of a pathological full slide according to claim 1 or 3, characterized in that: The information bottleneck potential representation module r is used to compress the local representation encoding vector of the pathological image area block processed by the visual encoder. The information in the pathological image area is reduced to the local representation encoding vector dimension; the information bottleneck potential representation module r has a dimension of The learnable vector, the value of the information bottleneck potential representation vector is recorded as Used for the full slice information fusion process.

5. A multimodal information fusion method for a language large model of a pathological full slide according to claim 1 or 3, characterized in that: The full slice information fusion of the pathological image is expressed by the formula: V=V'+FFW(V') Among them, V is the representation vector of the whole slice of the pathological image, are the learnable projection parameters of the cross-attention layer in the process of building full slices of pathological images.

6. The multimodal information fusion method for a language large model of a pathological full slide according to claim 1 is characterized in that: The specific method of step three is: The representation vector V of the full slice of the pathological image finally obtained in step 2 is combined with the vector L = {l1,l2,…,l t-1 } Modal fusion is achieved through tanh-gated cross-attention; the vector L of the text generated by the large language model itself before time t is used as the query vector query of the cross-attention layer, and the representation vector V of the full slice of the pathological image is used as the key and value in the cross-attention layer, and then the result is gated by tanh and the residual is summed; the result is then passed through the tanh-gated forward propagation layer neural network and the residual is summed to obtain a multimodal fusion vector; it can be expressed as: L merge =L'+tanh(ffw_para)·FFW(L) Among them, attn_para and ffw_para are both trainable scalar parameters of tanh gate with initialization value 0, L merge is the vector transmitted in the large language model after multimodal fusion, are the learnable parameters of the attention mechanism in the modality fusion process; The loss function in the process of multimodal fusion can be expressed as: Among them, l <t refers to the answer content generated by the large model before time t, p(l t |l <t ,V) refers to the probability of the real text output corresponding to position t under the condition of the full slice representation vector V of the given pathological image and the answer content generated by the large language model before time t.

7. A multimodal information fusion method for a language large model of a pathological full slide according to claim 1 or 6, characterized in that: The large language model adopts a Decoder-only generative large language model, including Llama and TeleChat.

8. A multimodal information fusion system for a large language model of pathological full sections, characterized by: include, The pathological image segmentation and processing module is used in step 1 to segment the pathological image into local area image blocks and encode them through a visual encoder to obtain a coding vector representing the local area block of the pathological image. The pathological image full slice information fusion module is used in step 2 to convert the encoding vector of the local representation of the pathological image area block obtained in step 1 into The input pathological image information is mixed with the attention layer to obtain the representation vector of the whole slice of the pathological image. By introducing the information bottleneck potential representation module r, the local representation encoding vector of the pathological image area block is converted into The compressed dimension is used as a fixed-length representation vector of the full slice of the pathological image; The multimodal fusion module of the large language model and the full slice information, in step three, uses the cross-attention mechanism to fuse the representation vector of the full slice of the pathological image obtained in step two with the text vector of the large language model, and obtains a modal fusion vector to generate the answer of the large language model.

9. A multimodal information fusion device for a large language model of a pathological full slide, characterized in that: include: A memory and a processor, wherein the memory stores a computer program so that the processor executes the multimodal information fusion method for the language large model of the whole pathological section.

10. A program storage medium for receiving user input, characterized in that: When the stored computer program is executed by the processor, it can perform multimodal information fusion based on the multimodal information fusion method of the language large model for the whole pathological slice according to the pathological image information and the text information input by the user.

Images