Pathology prediction method based on pathology image-text large model

Abstract

The invention provides a pathology prediction method based on a pathology image-text large model. The method comprises the following steps: acquiring a WSI image and corresponding diagnosis text data; the method comprises the following steps: reading a multi-magnification WSI image, extracting image features through a vim4path model, screening information which can be represented through the image in a diagnosis text by utilizing a GPT model, and constructing a user-defined vocabulary library; inputting the image features and the screened text into a pathological image-text large model, completing training based on token word segmentation of a user-defined vocabulary library, and synchronously generating a thermodynamic diagram corresponding to tokens; and taking tokens of a to-be-detected pathological image output by the large pathological image-text model as a retrieval key, and matching a report with the highest similarity in the pathological report RAG retrieval library as final output. Training can be completed only through a small number of limited WSI images, the video memory requirement is low, high efficiency, low resource consumption and high interpretability are achieved, and the technical problems that an existing pathological image-text model is high in data dependence and high in video memory requirement are solved.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of artificial intelligence, medical image processing, and natural language processing, and more specifically, to a pathological prediction method based on a large pathological image and text model. Background Technology

[0002] The existing training process for large-scale pathological image models has the following technical shortcomings: 1) It is highly dependent on data, requiring tens of thousands of WSI images to ensure training results, while high-quality pathological slide data is costly and time-consuming to acquire; 2) Extremely high GPU memory requirements; the training process typically requires over 100GB of GPU memory, which limits the widespread application of the model. 3) Poor interpretability; the correlation between model output and image features is unclear, making it difficult to meet the rigorous requirements of medical diagnosis. 4) The diagnostic text output lacks the standardization of clinical reports and requires secondary revision by doctors. Summary of the Invention

[0003] This invention addresses the technical problems existing in the prior art by providing a pathological prediction method based on a large pathological image and text model. This method overcomes the problem that the training process of existing large pathological image and text models requires a large amount of data and high video memory, resulting in high data costs.

[0004] According to a first aspect of the present invention, a pathological prediction method based on a large pathological image-text model is provided, comprising: Acquire full scan images of multiple pathological slides and corresponding diagnostic text data; Read the first multi-magnification image of each full scan image of the pathological slide, and extract the first image feature vector of the first multi-magnification image based on the vim4path model, wherein the vim4path model is a feature extraction network pre-trained on the pathological image dataset; Based on the GPT model, information that can be represented by images is filtered from each of the diagnostic text data to obtain the filtered diagnostic text data. A training set is constructed based on the first image feature vector corresponding to each full scan image of the pathological slide and the filtered diagnostic text data, and the pathological image and text large model is trained based on the training set. The second multi-magnification image of the full scan image of the pathological slide to be tested is read, and the second image feature vector of the second multi-magnification image is extracted. The second image feature vector is input into the trained pathological image-text model, and the predicted text sequence and the corresponding image region heat map are output. The predicted text sequence is matched against each pathology report in the RAG pathology report retrieval database, and the pathology report with the highest similarity to the predicted text sequence is output. Returns the pathology report and corresponding image region heatmap for the full scan image of the pathological slide to be tested.

[0005] This invention provides a pathological prediction method based on a large-scale pathological image-text model. It achieves efficient feature extraction with low data volume by utilizing a feature extraction model with dedicated pre-training capabilities for pathological images. Simultaneously, it leverages the GPT model to filter highly relevant text, improving data quality and training efficiency while avoiding invalid data diluting model performance. Furthermore, it constructs a custom vocabulary as a tokenizer, eliminating the need for separate training and significantly lowering the data source threshold for model training. Multi-magnification image feature extraction reduces the amount of original image data (16 times less than 40x image data), lowering the hardware requirements and GPU memory demands during the training of the large-scale pathological image-text model. Attached Figure Description

[0006] Figure 1 A flowchart of a pathological prediction method based on a large pathological image and text model is provided as an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a pathological prediction system based on a large pathological image and text model provided in one embodiment of the present invention; Figure 3 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention; Figure 4 This is a schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. Detailed Implementation

[0007] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined with each other to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0008] Figure 1 The following is a flowchart illustrating a pathological prediction method based on a large pathological image-text model according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps: Step 1: Obtain full scan images of multiple pathological slides and corresponding diagnostic text data.

[0009] Understandably, the purpose of this invention is to detect the pathological condition of the full scan image of the pathological slide to be tested. Therefore, it is necessary to train a large pathological image model and detect the pathological condition of the full scan image of the pathological slide to be tested based on the large pathological image model.

[0010] Before training the large-scale pathological image model, data collection was performed to obtain a dataset of whole scan images (WSI) of pathological slides. and corresponding diagnostic text dataset Among them, the data volume of the dataset This is a relatively small dataset. The WSI images must include 10x magnification scans, and the diagnostic text data must cover core information such as lesion type, morphological characteristics, and pathological grading. The dataset size meets the requirements. 0, meaning that only 1k-level WSI images are needed to train a large-scale pathological image model.

[0011] Step 2: Read the first multi-magnification image of each full scan image of the pathological slide, and extract the first image feature vector of the first multi-magnification image based on the vim4path model, wherein the vim4path model is a feature extraction network pre-trained on the pathological image dataset.

[0012] Understandably, image preprocessing is performed on each full scan image of a pathological slide in the collected dataset. Specifically, the full scan images of each pathological slide are read. 10x magnification image (Reduce image resolution to reduce computational cost) Extract the first image feature vector of each full scan image of a pathological slide using the vim4path model. ,in , For feature dimensions.

[0013] Specifically, for each WSI image, extract its 10x sub-image. (Reduce image resolution to reduce computational load), and extract features using the vim4path model to obtain fixed-dimensional feature vectors. The vim4path model, pre-trained on a pathological image dataset, possesses the ability to efficiently extract key features such as lesion regions and tissue morphology. Its feature extraction process can be represented as follows:

[0014] in, Used for downsampling and local feature extraction Capturing long-distance dependencies Output a fixed-dimensional feature vector.

[0015] In this embodiment of the invention, the vim4path model is a Transformer-based pathological image feature extraction model, which outputs feature dimensions. The value is 384.

[0016] In this embodiment of the invention, the feature extraction network is a vim4path model pre-trained on a pathological image dataset, rather than a general image encoder trained from scratch. The core reason why traditional image-text models require a massive amount of WSI images is that general feature extractors need to learn specific features of pathological images, such as lesion regions, tissue morphology, and cell structures, from scratch. Insufficient data makes it impossible to accurately capture this key information.

[0017] The vim4path model has learned a large number of common features of pathological images during the pre-training stage (such as typical textures of different lesions, distinguishing features between normal and abnormal tissues, and representation rules of common pathological structures). When using these features for image feature extraction in this model, it can complete the association learning between image features and pathological text without relying on a large amount of data to relearn basic pathological visual features, which greatly reduces the need for the amount of target training data.

[0018] Step 3: Based on the GPT model, filter the information in each of the diagnostic text data that can be represented by images to obtain the filtered diagnostic text data.

[0019] Understandably, the diagnostic text data corresponding to each full scan image of a pathological slide also needs to be preprocessed. This includes constructing a filtering function using the GPT model. , obtain the filtered text Based on all Build a custom vocabulary .

[0020] Specifically, the GPT model is used to analyze the original diagnostic text. The process involves filtering out information that cannot be represented by images (such as patient history, laboratory test results, etc.). The filtering logic satisfies the following:

[0021] in Calculate the score function for the text-image association in the GPT model. Preset threshold (empirical value) Based on all filtered results Build a custom vocabulary , The update mechanism is as follows:

[0022] That is, only semantic units that appear more than 3 times are retained to ensure the effectiveness and conciseness of the vocabulary.

[0023] Among them, custom vocabulary library Includes pathological diagnostic terminology, lesion description vocabulary, and anatomical terms, and uses a word frequency threshold to filter out low-frequency semantic units. (Words with a frequency of ≤3 are removed).

[0024] It should be noted that another reason for the large data requirements of traditional pathological image-text models is that the original pathological text contains a large amount of redundant information unrelated to the images (such as patient history, laboratory test results, doctor's notes, etc.). This invalid information dilutes the value of effective data, causing the model to require more data to filter out useful correlations. This invention constructs a filtering function S(T) using a GPT model to accurately remove information in the original diagnostic text that cannot be represented by images, retaining only the core diagnostic content directly related to the WSI image (such as lesion type, morphological characteristics, pathological grade, glandular formation rate, etc.), resulting in a high-purity filtered text C. i Highly relevant text data allows the model to focus directly on the mapping relationship between "image features and core pathological diagnostic information" during training, avoiding interference from invalid data and significantly improving the training value of a single data point.

[0025] In addition, the training of the pathology image-text model uses a tokenizer based on a custom vocabulary V for word segmentation. The vocabulary V consists of all filtered texts C. i The pathology-specific semantic units in the model include core vocabulary such as pathological diagnostic terms, lesion descriptions, and anatomical terms, and low-frequency redundant words are filtered out using a word frequency threshold. Traditional models use a general vocabulary, which requires processing a large number of words unrelated to pathology, and the model needs more data to learn the specific semantic logic of pathological texts. However, the custom vocabulary V simplifies the semantic space, so that the token sequence obtained after tokenizer segmentation only contains core pathological semantic units, reducing the semantic learning burden of the pathology image-text model. Therefore, the large-scale pathology image-text model in this embodiment of the invention does not need to learn general semantics through massive amounts of data, but only requires a small amount of data to accurately grasp the semantic expression rules of pathological texts, further reducing the demand for training data.

[0026] Step 4: Based on the first image feature vector corresponding to each full scan image of the pathological slide and the filtered diagnostic text data, construct a training set, and train the pathological image-text large model based on the training set.

[0027] Understandably, the first image feature vector is extracted from each full scan image of a pathological slide. and filtered diagnostic text data , constitute the training set .

[0028] training set The pathology image-text large model is input into the training process. This model employs a dual-tower structure of "image feature encoder - text decoder," where the image feature encoder receives... And perform dimension mapping; the text decoder is based on the Transformer architecture, to... Autoregressive training is performed on the target sequence.

[0029] The training process for the large-scale pathological image and text model is as follows: Step 41, Input a large-scale pathological image model, which is based on the tokenizer tool. Perform text segmentation to obtain a sequence of segmented tokens. ,in, , This represents the sequence of the j-th token. express The first in One token, The sequence length is given.

[0030] Understandably, Input pathology image and text large model, pathology image and text large model based on custom vocabulary. Initialize the tokenizer to implement a bidirectional mapping between text and token sequences: To avoid semantic biases caused by a general vocabulary; tokens are automatically generated after word segmentation. mask item The token to be filled is marked as The valid semantic token is marked as .

[0031] Step 42, Output of large-scale pathological image and text model Each of them The predicted probability and Corresponding image region heatmap .

[0032] Understandably, Input the pathology image and text large model, and the pathology image and text large model output each word after word segmentation. Predicted probabilities and each token sequence Corresponding image region heatmap The pixel values ​​in the image region heatmap represent the correlation strength between the region and the corresponding token sequence, intuitively demonstrating the image basis for the model's decisions and improving interpretability.

[0033] Step 43, based on Each of them The predicted probability and Corresponding image region heatmap Calculate the loss function value.

[0034] Among them, the loss function design of the pathological image and text large model is: combining masked cross-entropy loss (to optimize the prediction accuracy of effective tokens and eliminate interference from filling positions) and heatmap constraint loss (to ensure interpretability).

[0035] (1) Masked cross-entropy loss component: through The loss contribution of padding positions is masked, and only the average log loss of valid tokens is calculated to ensure that training focuses on meaningful semantic units. The expression for cross-entropy loss is:

[0036] in the formula ( =1) Avoid loss bias caused by differences in sequence length; (2) Heat map constraint loss Ensure that each valid token ( The alignment between the image region and the image area is expressed as:

[0037] in The height and width of the image at 10x magnification. For the first One valid token The corresponding image region is labeled with a heatmap. Ensure that alignment loss is calculated only for valid tokens.

[0038] Therefore, the loss function of the large-scale pathological image model can be expressed as a weighted sum of the masked cross-entropy loss and the heatmap constraint loss, as follows:

[0039] in, The large model representing pathological images and texts represents the first... Real tags at each time step The predicted probability; yes The mask item, valid token position Fill position ; It is the total number of valid positions, satisfying , ; for The corresponding image region heatmap, This is the balance coefficient.

[0040] Step 44: Adjust the model parameters of the pathological image and text model based on the loss function value until the loss function value is minimized, and obtain the trained pathological image and text model.

[0041] Understandably, the model parameters of the pathological image and text model are adjusted based on the calculated loss function value, and the pathological image and text model is iteratively trained until the loss function value is minimized, thus obtaining the trained pathological image and text model.

[0042] It should be noted that during the training of the large pathological image model, the dimensionality of the input data is reduced by extracting image features at a 10x ratio, the token sequence length is shortened by combining a custom vocabulary, and the masked loss function is used to reduce the amount of invalid computation. This makes the memory usage of the model training process ≤60G, and the training can be completed on a single high-end GPU server (such as NVIDIA A10080G).

[0043] Step 5: Read the second multi-magnification image of the full scan image of the pathological slide to be tested, and extract the second image feature vector of the second multi-magnification image. Input the second image feature vector into the trained pathological image-text model, and output the predicted text sequence and the corresponding image region heat map.

[0044] Understandably, after acquiring the trained pathological image-text model, the pathological condition represented by the pathological slide image to be tested is predicted. That is, during the inference process of the pathological image-text model, the full scan image of the pathological slide to be tested is read. Obtain the corresponding 10x magnification image and extract the second image feature vector from the 10x magnification image. The second image feature vector The trained pathological image-text model outputs a predicted token sequence. and corresponding image region heatmap .

[0045] Step 6: Match the predicted text sequence with each pathology report in the RAG pathology report retrieval database, and output the pathology report with the highest similarity to the predicted text sequence.

[0046] Understandably, this involves constructing a RAG search database of 100,000 pathology reports. , This represents the w-th pathology report in the search database.

[0047] In one embodiment of the present invention, the construction of the pathology report RAG retrieval database includes: collecting 100,000 pathology reports. For each pathology report Structured parsing is performed to extract core semantic units, which include diagnostic conclusions, lesion characteristics, and pathological grading. The extracted core semantic units are then compared with... Save after formatting alignment.

[0048] Output of the large pathological image and text model (Only valid tokens are retained, padding items are removed) Convert to semantic embedding vectors, and then calculate With each Semantic similarity:

[0049] in For semantic embedding functions, It is a cosine function.

[0050] Calculated After comparing the similarity with each pathology report in the search database, the output is... The most similar pathology report and .

[0051] Step 7: Return the pathology report and the corresponding image region heatmap corresponding to the full scan image of the pathological slide to be tested.

[0052] Understandably, the return and The most similar pathology report and image region heatmap This refers to the pathological results represented by the full scan image of the pathological section to be tested.

[0053] See Figure 2 This illustrates an embodiment of the pathological prediction system based on a large pathological image-text model according to the present invention. The system includes: The acquisition module 201 is used to acquire multiple full scan images of pathological slides and corresponding diagnostic text data; Extraction module 202 is used to read the first multi-magnification image of each full scan image of the pathological slide and extract the first image feature vector of the first multi-magnification image; The filtering module 203 is used to filter information that can be represented by images in each of the diagnostic text data based on the GPT model, so as to obtain the filtered diagnostic text data. Training module 204 is used to construct a training set based on the first image feature vector corresponding to each full scan image of the pathological slide and the filtered diagnostic text data, and to train the pathological image and text large model based on the training set; The output module 205 is used to read the second multi-magnification image of the full scan image of the pathological slide to be tested, extract the second image feature vector of the second multi-magnification image, input the second image feature vector into the trained pathological image-text model, and output the predicted text sequence and the corresponding image region heat map. The matching module 206 is used to match the predicted text sequence with each pathology report in the pathology report RAG retrieval library, output the pathology report with the highest similarity to the predicted text sequence, and return the pathology report corresponding to the full scan image of the pathology slide to be tested and the corresponding image region heatmap.

[0054] It is understood that the pathological prediction system based on a large pathological image and text model provided by the present invention corresponds to the pathological prediction method based on a large pathological image and text model provided in the foregoing embodiments. The relevant technical features of the pathological prediction system based on a large pathological image and text model can be referred to the relevant technical features of the pathological prediction method based on a large pathological image and text model, and will not be repeated here.

[0055] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 3 As shown, an embodiment of the present invention provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, it implements the steps of a pathological prediction method based on a large pathological image and text model.

[0056] See Figure 4 , Figure 4 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 4 As shown, this embodiment provides a computer-readable storage medium 400 on which a computer program 411 is stored. When the computer program 411 is executed by a processor, it implements the steps of a pathological prediction method based on a large pathological image and text model.

[0057] The pathological prediction method and system based on a large pathological image and text model provided in this invention have the following beneficial effects: (1) Low data requirement: Training can be completed with only 1k level WSI images, which reduces the data source dependence by an order of magnitude compared with existing models (which require tens of thousands of data). The core advantages are: through the pathological image-specific pre-training capability of the vim4path model, efficient feature extraction is achieved with low data volume; at the same time, the GPT model is used to filter highly relevant texts, which improves data quality and training efficiency and avoids invalid data diluting model performance; and a custom vocabulary is built as a tokenizer, which does not require separate training and greatly reduces the data source threshold for model training.

[0058] (2) Low GPU memory requirements: The training process requires ≤60G of GPU memory, compatible with mainstream high-end GPU devices (such as NVIDIA A10080G), significantly lowering the hardware threshold compared to existing models (which require over 100G of GPU memory). Technical support includes: 10x image feature extraction to reduce the amount of original image data (16 times less than 40x image data); and a masked loss function to avoid unnecessary computation. This advantage breaks the dependence of existing models on ultra-high-end computing power clusters, reduces the hardware cost of model training and deployment, and promotes the widespread application of pathological AI technology in primary healthcare institutions.

[0059] (3) Strong interpretability: By outputting heatmaps corresponding to tokens, a visual association between text output and image features is established, meeting the rigorous requirements of "traceability and verifiability" in medical diagnosis. Specifically, each diagnosis-related token (such as "adenocarcinoma" or "moderately differentiated") corresponds to a heatmap of a specific region in the WSI image. Doctors can intuitively verify whether the model's decision is based on lesion features in the image, rather than a false association. The pixel intensity of the heatmap quantifies the degree of association between the token and the image region, providing doctors with decision-making support, reducing concerns about trust in "black box models," and improving clinical acceptance.

[0060] (4) High practicality: Combined with the RAG search library of 100,000 pathology reports, the output results conform to clinical report standards and can directly assist doctors in diagnostic decision-making. The advantages are reflected in: The RAG search library extracts core semantic units such as diagnostic conclusions, lesion characteristics, and pathological grading through structured parsing of clinical reports, ensuring that the output format is consistent with clinical practice and reducing the workload of doctors in secondary editing; The 100,000 reports cover common and rare pathological cases, improving the model's adaptability to complex cases, especially providing authoritative references for primary care physicians or young doctors, reducing the risk of misdiagnosis and missed diagnosis; The semantic similarity matching mechanism ensures the relevance and accuracy of the output reports and avoids generalized diagnostic conclusions.

[0061] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0062] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0064] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0065] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0066] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0067] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A pathological prediction method based on a large-scale pathological image and text model, characterized in that, include: Acquire full scan images of multiple pathological slides and corresponding diagnostic text data; Read the first multi-magnification image of each full scan image of the pathological slide, and extract the first image feature vector of the first multi-magnification image based on the vim4path model, wherein the vim4path model is a feature extraction network pre-trained on the pathological image dataset; Based on the GPT model, information that can be represented by images is filtered from each of the diagnostic text data to obtain the filtered diagnostic text data. A training set is constructed based on the first image feature vector corresponding to each full scan image of the pathological slide and the filtered diagnostic text data, and the pathological image and text large model is trained based on the training set. The second multi-magnification image of the full scan image of the pathological slide to be tested is read, and the second image feature vector of the second multi-magnification image is extracted. The second image feature vector is input into the trained pathological image-text model, and the predicted text sequence and the corresponding image region heat map are output. The predicted text sequence is matched with each pathology report in the pathology report RAG retrieval database, and the pathology report with the highest similarity to the predicted text sequence is output. Returns the pathology report and corresponding image region heatmap for the full scan image of the pathological slide to be tested.

2. The pathological prediction method based on a large-scale pathological image and text model according to claim 1, characterized in that, The acquisition of multiple full-scan images of pathological slides and corresponding diagnostic text data includes: Obtain a dataset of full scan images of pathological slides and corresponding diagnostic text dataset ,in, Indicates the first Full scan images of pathological slides, Indicates the first Diagnostic text data corresponding to full scan images of pathological slides, M≤10000; Read the first multi-magnification image of each of the pathological slide full scan images, and extract the first image feature vector of the first multi-magnification image based on the vim4path model, including: Read the full scan image of each of the pathological slides 10x magnification image The first image feature vector of the 10x magnification image is extracted using the vim4path model. ,in , For feature dimensions.

3. The pathological prediction method based on a large-scale pathological image and text model according to claim 1, characterized in that, The step of filtering information that can be represented by images from each diagnostic text data based on the GPT model to obtain the filtered diagnostic text data includes: Constructing a filtering function based on the GPT model Based on the filtering function, each of the diagnostic text data The data is filtered to obtain the filtered diagnostic text data. ; Based on all filtered diagnostic text data Build a custom vocabulary .

4. The pathological prediction method based on a large-scale pathological image and text model according to claim 1, characterized in that, The constructed training set is represented as , Indicates the first Image feature vectors of full scan images of pathological slides For the first The filtered diagnostic text data, M represents the number of training samples in the training set; The training of the large-scale pathological image and text model based on the training set includes: Will Input a large-scale pathological image model, which is based on the tokenizer tool. Perform text segmentation to obtain a sequence of segmented tokens. ,in, , Indicates the first A sequence of tokens, express The first in One token, The sequence length; Output Each of them The predicted probability and Corresponding image region heatmap ; based on Each of them The predicted probability and Corresponding image region heatmap Calculate the loss function value; The model parameters of the pathological image and text model are adjusted based on the loss function value until the loss function value is minimized, thereby obtaining the trained pathological image and text model.

5. The pathological prediction method based on a large-scale pathological image and text model according to claim 4, characterized in that, The loss function of the large-scale pathological image model is a weighted sum of the masked cross-entropy loss and the heatmap constraint term, and its expression is: in, The large model representing pathological images and texts represents the first... Real tags at each time step The predicted probability; yes The mask item, valid token location Fill position ; It is the total number of valid positions, satisfying , ; for The corresponding image region heatmap, This is the balance coefficient.

6. The pathological prediction method based on a large pathological image-text model according to claim 1, characterized in that, The process involves reading a second multi-magnification image of the full-scan image of the pathological slide to be tested, extracting a second image feature vector from the second multi-magnification image, inputting the second image feature vector into the trained pathological image-text model, and outputting a predicted text sequence and a corresponding image region heatmap, including: Read the full scan image of the pathological slide to be tested Obtain the corresponding 10x magnification image and extract the second image feature vector from the 10x magnification image. ; The second image feature vector The trained pathological image-text model outputs a predicted token sequence. and corresponding image region heatmap ; The predicted text sequence is matched against each pathology report in the RAG pathology report retrieval database, and the pathology report with the highest similarity to the predicted text sequence is output, including: Constructing a RAG search database of 100,000 pathology reports ,calculate With each Semantic similarity: in For semantic embedding functions, It is a cosine function; Output and The most similar pathology report and .

7. The pathological prediction method based on a large-scale pathological image and text model according to claim 2, characterized in that, The vim4path model is a Transformer-based pathological image feature extraction model, which outputs feature dimensions. The value is 384.

8. The pathological prediction method based on a large pathological image-text model according to claim 3, characterized in that, The custom vocabulary library This includes pathological diagnostic terminology, lesion description vocabulary, and anatomical terms, using a word frequency threshold to filter out low-frequency semantic units. .

9. The pathological prediction method based on a large pathological image-text model according to claim 4, characterized in that, The image region heatmap Expressed as: in, The pixel coordinates of the 10x magnification image. ) is the feature vector of a 10x magnification image. For the first tokens The weighting coefficients, The attention weight calculation function, for The sequence length.

10. The pathological prediction method based on a large pathological image-text model according to claim 6, characterized in that, The construction of the pathology report RAG search database includes: Collecting 100,000-level pathology reports For each pathology report Structured parsing is performed to extract core semantic units, which include diagnostic conclusions, lesion characteristics, and pathological grading. The extracted core semantic units are then compared with... Save after formatting alignment.

Images