A pathological image processing model training method and system

Abstract

This invention mainly relates to a training method for a pathological image processing model, comprising the following steps: acquiring multiple image patches obtained from the division of a whole-slice image; generating a joint score by fusing the neural network attention score and morphological quality score of each image patch, and selecting a subset of high-value image patches from the multiple image patches based on the joint score; and jointly training the pathological basic model and the task network based on the high-value image patch subset to obtain an optimized pathological image processing model. The joint training includes: freezing the main parameters of the pathological basic model and updating only the parameters introduced by the parameter-efficient adaptation structure in the pathological basic model and the parameters of the task network. Through the above design, this application effectively solves the problem that large-scale pathological basic models cannot be trained end-to-end under limited computing power conditions in whole-slice image-level tasks due to the huge computational and storage resource requirements.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a training method and system for a pathological image processing model. Background Technology

[0002] With the popularization of digital pathology technology, whole-slide images (WSI) have become a core data carrier for clinical diagnosis and artificial intelligence analysis. WSIs are characterized by ultra-high resolution, with a single image typically containing billions of pixels, making them unsuitable for direct input into deep learning models. Therefore, WSIs need to be segmented into a large number of small image patches for analysis.

[0003] In recent years, pathology foundation models (PFMs) trained on massive amounts of data have developed rapidly. These models often employ advanced architectures such as the visual Transformer and are pre-trained on large-scale pathological images and text data, demonstrating powerful general feature representation capabilities. In practical applications, to accomplish WSI-level prediction tasks (such as tumor classification, grading, and prognosis prediction), the industry commonly introduces multi-instance learning (MIL) frameworks or other feature aggregation networks into the backend of PFMs.

[0004] However, WSI contains a huge number of patches (typically thousands to tens of thousands), while PFM has an extremely large parameter scale, which leads to severe computational challenges during training. These challenges manifest as extremely high memory usage, enormous computational costs, and excessively long training cycles, making it difficult for researchers to perform end-to-end optimization of PFM and severely limiting its performance potential in downstream tasks.

[0005] Currently, mainstream technical solutions attempt to alleviate the above problems, but still have significant limitations: The first approach is based on feature freezing. This approach inputs all patches into a pre-trained PFM with frozen parameters to extract features, and only trains the subsequent MIL aggregation network. Although this method reduces computational overhead, because PFM cannot be adapted to specific task data, its feature representation ability is fixed, resulting in poor generalization performance when facing rare diseases or specific data distributions, and failing to fully utilize the adaptability of large-scale base models.

[0006] The second approach is based on attention-based selection. This approach calculates the attention weights of each patch using a MIL network and selects a subset of patches for computation based on these weights, aiming to reduce the input size. However, this type of method relies solely on semantic-level attention mechanisms, making it highly susceptible to noise interference such as image artifacts, uneven coloring, and high-contrast backgrounds. This can lead to a large number of low-quality or diagnostically meaningless regions being mistakenly selected as high-value regions, introducing noise and impairing the model's robustness and prediction accuracy.

[0007] Furthermore, in the field of natural image processing, although parameter-efficient fine-tuning techniques (such as LoRA) can adapt to large models by introducing a small number of trainable parameters, these techniques are not combined with the input size control of WSI. If fine-tuning is still performed on all patches, even with a small number of parameter updates, the computational and memory overhead during forward and backward propagation is still unbearable, failing to fundamentally solve the computational bottleneck caused by WSI.

[0008] In summary, existing technologies lack a systematic solution capable of addressing the dual challenges of large input scale and numerous model parameters posed by WSI. Their main shortcomings can be summarized as follows:

[0009] (1) High computational complexity makes it difficult to achieve end-to-end training of PFM; (2) The basic model cannot fully adapt to downstream tasks, resulting in performance limitations; (3) The patch selection mechanism is too simple and lacks reliability; (4) Failed to optimize parameter efficiency and input efficiency in a coordinated manner.

[0010] These shortcomings collectively restrict the deployment and application of large-scale basic pathology models in real clinical scenarios.

[0011] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0012] The technical problem this application aims to solve is "how to address the issue that large-scale pathological basic models cannot be effectively trained end-to-end under limited computing power conditions due to the huge computational and storage resource requirements in whole-slice image-level tasks, for example: how to collaboratively compress the input scale and parameter scale of model training to overcome the computing power bottleneck."

[0013] The technical solution adopted in this application to solve the above-mentioned technical problems is as follows.

[0014] This application provides a training method for a pathological image processing model, comprising the following steps: Obtain multiple image patches obtained from the division of a full-slice image; By fusing the neural network attention scores and morphological quality scores of each image patch, a joint score is generated, and a subset of high-value image patches is selected from multiple image patches based on the joint score. Based on a subset of high-value image patches, the pathological baseline model and the task network are jointly trained to obtain an optimized pathological image processing model. The joint training includes freezing the main parameters of the pathological baseline model and updating only the parameters introduced by the parameter efficient adaptation structure in the pathological baseline model and the parameters of the task network.

[0015] In some embodiments, the neural network attention score is obtained by processing each image patch through a pre-trained multi-instance learning network.

[0016] In some embodiments, the morphological quality score is obtained by analyzing the pixel data of each image patch and calculating at least one morphological evaluation factor.

[0017] In some embodiments, morphological evaluation factors include: tissue proportion of image patches, rationality of staining distribution of image patches, color saturation and brightness variability of image patches, information content of image patches, nucleocytoplasmic ratio or cell density of image patches, and background noise suppression factor of image patches.

[0018] In some embodiments, calculating the background noise suppression factor includes: calculating the proportion of black pixels and / or the proportion of white pixels in an image patch; if the proportion of black pixels is higher than a first threshold, or the proportion of white pixels is higher than a second threshold, then reducing the morphological quality score of the image patch.

[0019] In some embodiments, fusing the neural network attention score and morphological quality score of each image patch includes multiplying the normalized neural network attention score by the normalized morphological quality score.

[0020] In some embodiments, the pathological baseline model is a pre-trained model.

[0021] In some embodiments, the parameter efficient adaptation structure includes a low-rank adaptation LoRA module, a weight decomposition low-rank adaptation DoRA module, or an adapter module.

[0022] In some embodiments, a pathological image processing system is also provided for implementing the training method of this application, the system comprising: The image patch acquisition module is configured to acquire multiple image patches obtained by dividing a full-slice image. The high-value subset selection module is configured to generate a joint score by fusing the neural network attention score and morphological quality score of each image patch, and select a high-value image patch subset from multiple image patches based on the joint score. The joint training module is configured to jointly train the pathological baseline model and the task network based on a subset of high-value image patches to obtain an optimized pathological image processing model. The joint training module is further configured to freeze the main parameters of the pathological baseline model and only update the parameters introduced by the parameter efficient adaptation structure in the pathological baseline model and the parameters of the task network. The memory is used to store parameters of the basic pathological model and the task network; The processor is used to call and execute the image patch acquisition module, the high-value subset screening module, and the joint training module.

[0023] In some embodiments, a computer-readable storage medium is also provided, on which computer program instructions are stored, which, when executed by a processor, implement the steps of the training method of this application.

[0024] The present invention has the following beneficial effects: This application effectively solves the problem that large-scale pathological basic models cannot be trained end-to-end under limited computing power in whole-slice image-level tasks due to the huge computational and storage resource requirements. Specifically, it addresses this issue by collaboratively optimizing the input data scale and model parameter update frequency. This application employs a technique of "generating a joint score by fusing neural network attention scoring and morphological quality scoring, and selecting a subset of high-value image patches based on this joint score." This eliminates the need to process all image patches during model training, significantly reducing the input data size and thus greatly alleviating the computational and memory pressure during forward and backward propagation. It also solves the computational bottleneck problem caused by excessively large input sizes. By adopting a joint training strategy of "freezing the main parameters of the pathological basic model and updating only the parameters introduced by the parameter efficient adaptation structure and the parameters of the task network," the application significantly reduces the number of trainable parameters while still enabling the pathological basic model to adapt to downstream tasks. This controls the scale of parameter updates while maintaining the model's expressive power and generalization performance, overcoming the shortcomings of traditional feature freezing methods where the model cannot adapt to specific task data.

[0025] Furthermore, the neural network attention score of this application is obtained by processing each image patch through a multi-instance learning network, which can identify high-value regions related to the task from a semantic level; while the morphological quality score is obtained by analyzing the pixel data of the image patch and calculating a variety of morphological evaluation factors, which can effectively identify and filter low-quality regions caused by uneven staining, background noise, tissue tearing, etc., so as to take into account both semantic relevance and image quality in the screening process, significantly improving the reliability and representativeness of the high-value subset, and avoiding the problem that a single attention mechanism is susceptible to noise interference.

[0026] Furthermore, this application integrates the normalized neural network attention score with the morphological quality score by multiplying them, which mathematically achieves the coupling and balance of the two types of scores. This fully utilizes the semantic perception capability of the pre-trained model and introduces image quality constraints, making the screening results more robust and reasonable.

[0027] Furthermore, the parameter-efficient adaptation structure (such as LoRA, DoRA, or Adapter modules) adopted in this application further compresses the parameter update scale, achieving adaptation of the basic model with very few additional parameters. This, in conjunction with the input filtering mechanism, reduces the computational power requirement of the training process.

[0028] In summary, the aforementioned technical features of this application work together to form a systematic input and parameter collaborative compression scheme: on the one hand, the feedforward scale is controlled by high-quality input screening, and on the other hand, the amount of backpropagation parameters is controlled by efficient parameter fine-tuning. This fundamentally breaks through the dual computational bottleneck in whole-slice image processing, making end-to-end training of large-scale pathological basic models possible under limited computing power conditions. This significantly improves the performance and generalization ability of the model in downstream tasks, providing a technical foundation for the efficient and reliable deployment of pathological image analysis in real clinical scenarios.

[0029] Other beneficial effects of the present invention will be further described below. Attached Figure Description

[0030] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating an embodiment of this application; Figure 2 This is a schematic diagram illustrating melanin determination in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the determination of white pigment in an embodiment of this application. Detailed Implementation

[0031] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope and application of the present invention.

[0032] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0033] To address the limitations of existing technologies in whole-slice image (WSI) level AI modeling, such as limited computational resources, difficulty in involving basic models in end-to-end training, and insufficient stability in image patch selection, this invention aims to solve the following technical problems: First, given that WSI typically contains thousands to tens of thousands of image patches and the pathological baseline model has a large number of parameters, how can the baseline model participate in end-to-end joint optimization of downstream tasks under limited computational resources? Existing technologies, due to excessive computational complexity, usually only freeze the baseline model parameters, limiting the model's adaptability to data specific to certain tasks. This invention reduces the original input size from M image patches to K significantly relevant image patches by constructing a subset of high-value image patches. This significantly reduces computational complexity while providing feasible conditions for the baseline model to participate in backpropagation, thereby realizing an end-to-end optimization path.

[0034] Secondly, in the process of image patch selection using WSI, how can we simultaneously ensure that the selection results are both task-relevant and morphologically reliable? Selection methods based solely on attention weights are easily affected by artifacts, abnormal staining, or background interference, potentially leading to the misselection of low-quality regions. This invention constructs a joint scoring mechanism of neural attention and morphological quality, enabling the selection process to consider both task semantic contribution and histological structure and image quality, thereby improving the stability and reliability of the input subset.

[0035] Furthermore, given the participation of the base model in training, how can we further reduce the parameter size and memory usage to make the model training process feasible under actual computing resource conditions? This invention, based on the subset input mechanism, introduces an efficient parameter adaptation structure. While freezing the main weights of the base model, it only updates lightweight adaptation parameters, achieving coordinated optimization of input size control and parameter size control.

[0036] The present invention will be further described below.

[0037] In some embodiments, such as Figure 1 As shown, this invention provides a training method for a pathological image processing model, which may involve a two-stage mechanism, including: Phase 1: High-value image patches are selected through a joint screening mechanism of neural network attention and morphological quality.

[0038] Phase 2: Efficient end-to-end parameter fine-tuning of high-value image patch subsets based on WSI.

[0039] The two-stage sequential execution, along with the collaborative reduction of computational complexity in both input and parameter size dimensions, enables efficient end-to-end training of models with a large number of parameters at the WSI level.

[0040] In some embodiments, the first stage involves selecting high-value image patches using a joint screening mechanism of neural network attention and morphological quality, as follows: For a WSI (Wide Spot Image), data preprocessing is first required. First, the tissue region is segmented from the entire WSI slice, and irrelevant background is removed. Then, the tissue region is divided into a large set of M non-overlapping image patches. In other words, the entire tissue region is cut into non-overlapping image patches of the same size, similar in size to common natural images. For example, a WSI tissue region at 20x magnification is cut into M 224x224 image patches. Then, a joint selection mechanism is used to select the top K high-quality image patches from these M patches, where K is much smaller than M.

[0041] In some embodiments, the goal of the first phase is to reduce the number of elements in the original WSI to [amount missing]. Selecting Top from a set of image patches (default A subset of high-value image blocks (which can be adjusted according to your needs).

[0042] In some embodiments, this stage has three key components: Attention scoring in neural network pathways; Quality scoring of morphological pathways; A joint screening mechanism based on attention scoring of neural network pathways and quality scoring of morphological pathways.

[0043] (a) Attention scoring of neural network pathways.

[0044] To measure the semantic contribution of each image patch to the downstream diagnostic task, this invention introduces a neural network attention scoring mechanism based on Multiple Instance Learning (MIL).

[0045] (1) Feature extraction is performed using a pre-trained pathological baseline model at the frozen image patch level.

[0046] Suppose a WSI is divided into M image patches:

[0047] First, each image patch is input into the frozen pre-trained pathological baseline model for feature encoding to obtain the corresponding feature representation:

[0048] in: The visual encoder for the Basic Pathology Model (PFM) can be selected manually; MUSK is used by default. The output dimension is... This is determined by the visual encoder of the chosen pathological basis model (PFM), as different PFMs encode image patches into feature vectors with different dimensions.

[0049] The feature matrix will then be obtained.

[0050] Encoders are optional, but not limited to: CONCH [1] ; UNI [2] ; MUSK [3] ; Or other pathological pre-trained basic models.

[0051] It should be noted that this invention does not limit the specific encoding model. The visual encoder used in the first stage and the encoder to be fine-tuned in the second stage can adopt the same or different model architectures. In some embodiments, the visual encoder in the first stage adopts a pre-trained pathological baseline model (PFM) with frozen parameters. Because the model is large and has been pre-trained on large-scale medical datasets (such as UNI, CONCH, MUSK, Virchow, etc.), it has strong feature extraction capabilities. At the same time, the freezing method can greatly reduce the computational cost of the first stage.

[0052] (2) Calculation of attention score based on MIL network.

[0053] feature matrix Input the pre-trained MIL attention network to obtain the attention weights for each image patch.

[0054] In some embodiments, an Attention-based MIL (ABMIL) structure is used, and its attention calculation form is as follows:

[0055] These are the parameters for the pre-trained network. and These are the hyperbolic tangent and the Sigmoid function, respectively. This represents element-wise multiplication; The attention weights are then obtained through Softmax normalization:

[0056] satisfy:

[0057] In some embodiments, the MIL network may employ Feather-24K. [8] Pre-trained ABMIL [9] Model.

[0058] In some embodiments, both the vision encoder and the MIL network in the first stage remain in a parameter-frozen state.

[0059] It is understood that, in order to minimize computational overhead and ensure the reliability of the screening, the multi-instance learning network preferably uses a model pre-trained on a large-scale medical pathology slide dataset (such as the MIL network in the Feather model). The parameters of this pre-trained network are frozen and do not need to be updated, thereby providing high-quality semantic attention scores for image patches at extremely low computational cost.

[0060] Of course, this invention remains open to specific implementations of multi-instance learning networks. Theoretically, networks trained from scratch using specific task data could also be used, but this would introduce additional training costs. The core of this invention lies in utilizing a multi-instance learning mechanism (regardless of the source of its parameters) to obtain semantic scores and fusing them with morphological scores, rather than being limited to specific network parameter states. If a pre-trained multi-instance learning network with superior performance becomes available in the future, it should be understood that this invention can be naturally extended and applied within its scope.

[0061] Replacement description: This invention is not limited to using the ABMIL structure, attention weights It can be obtained through any network architecture that can output an importance score for each image patch, such as CLAM. [4] TransMIL [5] And other multi-instance learning models that can output image block-level weights.

[0062] (ii) Quality score of morphological pathways.

[0063] The quality score of morphological pathways assesses the importance of an image patch solely from an image perspective.

[0064] Let's assume a WSI is divided into M image blocks:

[0065] Then, the importance of the image patch was evaluated from a morphological image perspective.

[0066] (1) Organizational proportion estimation.

[0067] Normalize the RGB image to the range: Then convert it to HSV space.

[0068] The default thresholds are as follows (the following three default thresholds can be set to other numbers): SATURATION_THRESHOLD = 0.2; LOW_VALUE_THRESHOLD = 0.6; HIGH_VALUE_THRESHOLD = 0.92; Organization pixel is defined as:

[0069] in: S Indicates the saturation channel. V Indicates the value channel; For logical AND; Logical OR; Organizational proportion (in percentage form): ; It's understandable that an image patch contains many pixels. Based on the previous definition of an organized pixel, we determine the number of pixels in an image patch that are identified as organized pixels, and the total number of pixels identified as organized pixels is N. tissue The total number of pixels in an image patch is N. total N tissue Divide by N total This can be used to measure the proportion of tissue regions in an image patch relative to the entire image patch.

[0070] (2) Organizational quantity factor.

[0071] Default threshold: TISSUE_HIGH_THRESH = 80; TISSUE_LOW_THRESH = 10; The organizational quantity factor is defined as:

[0072] (3) Nucleus / Cytoplasm factor, which measures the nucleus-to-cytoplasm ratio, i.e., the area of ​​the nucleus / the area of ​​the cytoplasm.

[0073] Analyzing from the perspective of HSV: Hue angle calculation:

[0074] Scope of retention:

[0075] Default Reference Center: HSV_PURPLE = 270; HSV_PINK = 330; calculate:

[0076]

[0077]

[0078] (4) Color factor:

[0079] In some embodiments, the proportion of pixel regions that may be cell nuclei is analyzed from the perspective of hue in the HSV color space through "suspected nucleus pixel proportion" evaluation. The density of cell nuclei in a region is approximated by identifying pixels in a specific hue range (typically corresponding to cell nucleus staining, such as blue-violet in H&E staining).

[0080] The method assesses the proportion of pixels in an image patch that may be cell nuclei by evaluating the "proportion of pixels representing suspected nuclei." This method approximates the nucleus region using color features, and its output differs from the precise definition of cell morphology-based segmentation in pathology, but it effectively serves the image quality screening purpose of this invention. As an extended embodiment of this invention, a method based on a pre-trained segmentation model (such as U-Net) can be used to more accurately calculate the actual proportion of tissue and nucleus regions, thereby making this quality assessment indicator more pathologically significant.

[0081] It should be noted that the above method is a fast approximate estimation based on color statistics. Its output differs from the precisely defined "nucleocytoplasmic ratio" based on cell morphology segmentation in pathology, but it effectively serves the image patch selection purpose of this invention. Its advantage lies in its extremely high computational efficiency, perfectly matching the core objective of this invention to reduce global computational complexity. However, the method based on a pre-trained segmentation model (such as U-Net), which is an extended implementation of this invention, would significantly increase the computational load by again utilizing another neural network model for cell segmentation.

[0082] (5) Saturation and brightness variance factors.

[0083] Still analyzing HSV Calculate the standard deviation:

[0084]

[0085] The default threshold is set to 0.05, but you can adjust it yourself. The value of factor is determined based on the following conditions: like and If the factor is less than 0.05, then the factor = 0.4. If only one item is below the threshold (i.e.) <0.05 or If the condition is less than 0.05 (but not simultaneously satisfied), then factor = 0.7; Otherwise (i.e.) ≥ 0.05 and If the value is greater than or equal to 0.05, then the factor = 1.0; Then, squaring yields the final variance factor:

[0086] (6) Background rejection mechanism.

[0087] Default threshold (can be adjusted): BLACK_V = 0.12; WHITE_V = 0.95; WHITE_S = 0.10; MAX_BLACK_FRAC = 0.30; MAX_WHITE_FRAC = 0.60; Then determine the proportion of black and white regions in the entire image block; We will still analyze the image from the perspective of HSV.

[0088] like Figure 2 It calculates the proportion of the area occupied by black regions in an image patch. If the value of a pixel is less than the set BLACK_V (the default value is 0.12), then the pixel is determined to belong to the black region. black_frac represents the proportion of the area occupied by black regions in an image patch.

[0089] Next, we calculate the proportion of the area occupied by the white region in an image patch.

[0090] like Figure 3It calculates the proportion of the area occupied by white regions in an image patch. If the V value of a pixel is greater than the set WHITE_V (default value is 0.95) and the S value is less than the set WHITE_S (default value is 0.10), then the pixel is determined to belong to the white region. white_frac represents the proportion of the area occupied by white regions in an image patch.

[0091] Next is the background rejection mechanism: If a white or black background area occupies a large area of ​​an image block, the image block is considered unqualified and will no longer be considered.

[0092] If any of the following conditions are met: black_frac > 0.30; white_frac > 0.60; The score for that image patch is then set to zero.

[0093] In some embodiments, in addition to the hard rejection mechanism described above, a more lenient processing scheme is provided as an alternative embodiment.

[0094] For example, the background rejection mechanism can combine the background percentage as a negative factor with scores from other morphological scoring factors. For instance, this can be achieved using the following formula.

[0095]

[0096] in, and It is the threshold for setting the background ratio, which can be a suitable threshold obtained by sampling from a large medical dataset and performing statistical analysis.

[0097] In this way, while suppressing areas with a high background ratio, important tissue areas containing a small amount of background can be preserved, achieving a gentle processing of abnormal areas.

[0098] After obtaining the various morphological assessment factors, they need to be integrated into a single quality score. In a preferred embodiment, multiple morphological scoring factors are fused using a product to enhance the suppression effect on low-quality regions. When any key factor has a low value, the overall score decreases accordingly, thereby achieving gated screening of abnormal regions. In other embodiments, the scoring factors can also be integrated using a weighted linear combination, weighted product, or other nonlinear fusion functions. For example, the following weighted form can be constructed:

[0099]

[0100] Among them, the weighting coefficient Based on domain knowledge, the weight ratio can be adjusted to achieve a balance between suppressing abnormal regions and maintaining scoring stability.

[0101] Unnormalized score:

[0102] Mapping function:

[0103] If background rejection is triggered:

[0104] Furthermore, as another simplified implementation of the present invention, the morphological quality score can also be calculated based solely on the tissue proportion of the image patch and the background noise suppression factor, for example, by directly multiplying the two; the neural network attention score can also be obtained by processing the basic features of the image patch (such as features extracted by a lightweight pre-trained model) with a simple linear classifier and then normalizing it using the Softmax function. Although this simplified method is not as accurate as the aforementioned preferred method, it still achieves the basic concept of the present invention, significantly reduces the computational load, and selects a subset of image patches with higher value.

[0105] (iii) The attention score of the neural network pathway and the quality score of the morphological pathway are combined to obtain the final score of each image patch. Then, the images can be sorted according to the final score to select the TOP-K high-quality image patches.

[0106] For an image patch, the attention score of its neural network pathway is: The quality score of the morphological pathway was The final score is:

[0107] For all image patches in a WIS, we can obtain their respective final scores. Then, based on this final score, select the top-K high-value image patch subsets of the highest quality.

[0108] In some embodiments, the second stage: an efficient end-to-end parameter fine-tuning mechanism based on high-value image patch subsets is as follows: Using the high-value image patch subset obtained in the first stage as input, we then train the pathological basic model and downstream network end-to-end. Although the high-value image patch subset on the input side is already small, the pathological basic model is still very large. If we train all the parameters of the pathological basic model and the downstream network, it is still a huge challenge. Therefore, we need to introduce an efficient fine-tuning strategy for the pathological basic model and perform full training on the downstream network.

[0109] For efficient fine-tuning strategies of the pathological baseline model, the common LORA method can be used by default, freezing all parameters of the original pathological baseline model and adding a small number of trainable parameters to the linear layers.

[0110]

[0111] During end-to-end training, only the pathological baseline model is updated. And all parameters of the downstream network.

[0112] In some embodiments, the second stage trains a visual encoder and a downstream network, which also includes a MIL network. After the model is trained, when it is put into use, the MIL network trained in the second stage can be applied to the MIL network in the first stage to calculate the neural network-side attention score.

[0113] In some embodiments, a pathological image processing system is also provided for training and inference of a large-scale pathological basic model. The system includes an image patch acquisition module, a high-value subset screening module, a joint training module, a prediction output module, a memory, and a processor. The modules are connected via data interfaces to form a complete end-to-end training and inference process. The processor is coupled to the memory and is used to call and execute the various modules.

[0114] Image patch acquisition module: The image patch acquisition module receives raw WSI image data and performs preprocessing and segmentation operations on it. This module segments the input WSI into M image patches and performs standardization processing on each image patch, including but not limited to size normalization, color normalization, and background region filtering.

[0115] This module outputs a set of structured image patches for subsequent feature analysis.

[0116] High-value subset filtering module: The high-value subset selection module is used to evaluate the quality and semantic value of M image patches and select a subset of high-value image patches.

[0117] The module is configured to generate a joint score by fusing the neural network attention score and morphological quality score of each image patch, and then select K high-value image patches (K is much smaller than M) from multiple image patches based on the joint score.

[0118] This module significantly reduces the computational scale of subsequent processing while ensuring semantic relevance and morphological credibility, avoiding misselection problems caused by single neural attention, thus affecting the quality of training data and model stability.

[0119] Joint training module: The joint training module is used to jointly train the pathological baseline model and the task network based on a subset of high-value image patches to obtain an optimized pathological image processing model.

[0120] The joint training module is further configured to: freeze the main parameters of the pathological baseline model and only update the parameters introduced by the parameter-efficient adaptation structure in the pathological baseline model and the parameters of the task network.

[0121] This configuration achieves synergistic optimization of input size compression and parameter size compression, significantly reducing the computational resources required for model training. This makes it possible to perform end-to-end training with the basic model involved under limited computing power and improves the model's adaptability to specific downstream tasks.

[0122] Prediction output module: During the inference phase, the system performs forward computation based on a subset of high-value image patches. The prediction output module is configured to output diagnostic prediction results, including but not limited to: pathological classification results, risk scores, prognostic prediction results, and key region location information.

[0123] This output module can interface with hospital information systems or scientific research analysis platforms, and is suitable for clinical pathology diagnosis and artificial intelligence-assisted analysis, demonstrating the system's industrial applicability.

[0124] Memory: The memory is used to store computer program instructions, parameters of the basic pathological model and task network, as well as intermediate data generated during training.

[0125] processor: The processor is used to call and execute the image patch acquisition module, the high-value subset screening module, the joint training module, and the prediction output module, and coordinate the overall workflow of the system.

[0126] In some embodiments, a computer-readable storage medium is also provided, on which computer program instructions are stored, which, when executed by a processor, implement the steps of the training method of this application.

[0127] In some embodiments, a pathological analysis model is also provided, which is trained using the training method of this application. The pathological analysis model includes a pathological basic model and a task network that have been jointly trained.

[0128] In some embodiments, a pathological diagnosis auxiliary method is also provided, comprising the following steps: Obtain the whole-slice image of the tissue to be diagnosed; The pathological analysis model of this application is used to process the whole slide image to generate a diagnostic prediction result; Output diagnostic prediction results.

[0129] In some embodiments, the diagnostic prediction results include at least one of the following: tumor subtyping results, tumor grading results, prognostic prediction results, and molecular marker prediction results.

[0130] In summary, the contributions of this application to the prior art include, but are not limited to: I. A dual-path scoring-based image patch selection mechanism is proposed, which is based on neural network path attention scoring, morphological path quality scoring, and joint scoring mechanism. It simultaneously ensures semantic relevance and morphological credibility, and avoids the mis-selection problem caused by single neural attention.

[0131] 1. Neural Network Attention Scoring: Coarse-grained feature encoding is performed on all image patches in WSI using a frozen, pre-trained lightweight multi-instance learning network. The attention weights for each image patch are calculated using the MIL network, which serve as the attention score for each image patch. The neural network attention score can reflect the contribution of image patches to the downstream network.

[0132] 2. Morphological pathway quality score: To address the issue that relying solely on attention scoring may result in the selection of artifacts or low-quality tissue regions, this invention introduces an independent image morphological quality assessment mechanism.

[0133] Quality rating includes, but is not limited to, the following aspects: Organizational proportion assessment; assessment of the rationality of staining distribution; assessment of color saturation and brightness variability; Information content assessment; nucleocytoplasmic ratio or cell density assessment; background noise suppression factor.

[0134] This score is used to measure the score of an image patch at the histological image level.

[0135] Optionally, the threshold parameters involved in the morphological quality scoring in this invention are initialized using a calibration strategy based on the statistical characteristics of publicly available pathological images. Specifically, a representative sampling set is constructed from publicly available WSI / pathological images. The sampled image blocks are uniformly preprocessed and converted to the HSV color space. The distribution characteristics of the luminance channel V and the saturation channel S are statistically analyzed. The thresholds for black regions (low quantile of V), white backgrounds (high quantile of V and low quantile of S), and tissue candidate pixel screening thresholds (the tissue distribution quantile intervals of S and V) are determined using quantiles or robust statistics. Simultaneously, the purple / pink center value and hue range are determined based on the main peak position of the tissue pixel hue distribution in publicly available H&E data. The proportion of black / white pixels is calculated for each image block, and its distribution is statistically analyzed. The high quantile is taken as the initial value for the background rejection ratio threshold. The above thresholds are preferred initial values ​​and can be appropriately adjusted under different data sources or staining schemes, but this does not constitute a limitation on the scope of protection of this invention.

[0136] 3. Joint scoring mechanism: This invention combines the attention score of the neural network pathway and the quality score of the morphological pathway to obtain the final score of each image patch, thus combining the advantages of both pathways.

[0137] The default combination method for the two scores is a product. Before fusion, the morphological scores are also normalized to a similar range (e.g., the 0-1 interval) so that the two scores can be effectively fused on a comparable scale. Finally, all image patches are sorted according to the joint score, and the Top K image patches are selected as candidate regions.

[0138] Attention mechanism scores obtained through neural networks are black-box scores. Because a neural network is a black box, the scores obtained cannot be explicitly linked to our final medical questions. For example, a high attention score obtained by a neural network does not necessarily mean that it is strongly related to a positive biomarker for cancer. It is very likely that the image patch with a high attention score contains a large part of non-tissue areas. This is why it is necessary to add explicit morphological scores.

[0139] II. A parameter-efficient end-to-end training mechanism based on high-value image patch subsets is proposed. After completing the screening of high-value image patches, this invention proposes an efficient end-to-end training mechanism based on the parameter input of the subset of high-value image patches, which is used to achieve joint optimization of the pathological basic model and the downstream network under the condition of limited computing resources.

[0140] Its key innovations include: an end-to-end optimization mechanism based on high-quality image patch subset input, and the introduction of an efficient parameter adaptation mechanism.

[0141] 1. End-to-end optimization mechanism based on high-value image patch subsets: Unlike existing technologies that freeze the parameters of the basic model and only train the downstream network, this invention allows the basic model to participate in the joint optimization of the downstream task; After completing the screening of high-value image patches, this invention constructs an end-to-end optimization mechanism based on subset input to solve the problem that the base model is difficult to participate in joint training in whole slice image (WSI) level tasks.

[0142] Traditional methods typically freeze the basic pathological model and train only the downstream network. This is because the WSI contains a large number of image patches, and training all of them into the basic model would lead to excessive computational complexity. This invention, through the aforementioned filtering mechanism, reduces the original WSI containing M image patches to an input subset containing only K high-value image patches, where K is much smaller than M.

[0143] During the training phase, only a subset of K high-value image patches is input into the base model for feature encoding, and the encoding results are then fed into the downstream task network. Backpropagation is performed based on the downstream task loss function, allowing the gradient signal to simultaneously affect the trainable parameters of the base model and the parameters of the downstream network, thereby achieving end-to-end joint optimization of the base model and the downstream network.

[0144] By controlling the size of the subset, this invention significantly reduces the computational resources required for model training, making it possible to perform end-to-end training involving the base model under limited computing power.

[0145] Experimental results show that this two-stage training strategy can achieve end-to-end training based on ultra-large parameter scale pathological basic models and ultra-high resolution WSI images. The experiment can be completed on consumer-grade graphics cards, which greatly reduces the technical threshold and computing power requirements for computing equipment.

[0146] 2. Introduce an efficient parameter adaptation mechanism: To further reduce the parameter scale and memory overhead caused by the participation of the base model in training, this invention introduces an efficient parameter adaptation mechanism within the base model.

[0147] The mechanism inserts a lightweight trainable adaptation module into the model's feature transformation structure while keeping the pre-trained weights of the base model unchanged. During training, the main parameters of the base model are frozen, and only the parameters of the adaptation module and downstream networks are updated.

[0148] The adaptation module can adopt a low-rank matrix factorization structure, a learnable mapping structure in residual form, a local hierarchical parameter update structure, or other efficient fine-tuning structures that do not significantly increase the overall parameter size of the model.

[0149] This approach significantly reduces the proportion of trainable parameters while maintaining the model's expressive power, thereby reducing memory usage and computational burden, improving training stability, and enhancing the adaptability of large-scale pathological models for WSI-level tasks.

[0150] Furthermore, in some embodiments, this application has other alternatives, including but not limited to: (1) Other encoders can be used for the image feature extractor for the image patch in both stages; (2) Other encoders can be used for text feature extraction of pathology reports in both stages; (3) The specific feature dimensions of the image and text, as well as the corresponding dimension change matrix, can be changed; (4) The MIL network in the first stage is used to calculate the attention score of the neural network pathway for each image patch. This MIL network is not limited to ABMIL. [9] Networks can also use CLAM. [4] TransMIL [5] wait; (5) In the second stage, in addition to LORA, other efficient parameter tuning methods for pathological basic models include [6] Other efficient parameter tuning methods can also be used, such as DoRA. [7] .

[0151] Compared with existing technologies, this application has the following advantages: In the case of scarce computing resources, it enables end-to-end training of ultra-high resolution WSI images and ultra-large-scale pathological basic models. For the problem of huge WSI resolution, the first stage can be used to select a high-quality subset of image blocks. For the problem of huge number of parameters in the pathological basic model, the second stage can be used for efficient fine-tuning and adaptation.

[0152] It should be noted that the specific explanations of the terms used in the embodiments of this specification can be found in the publicly available references listed below. It should be specifically stated that these references are used only to illustrate optional implementation methods and are not intended to limit the scope of protection of this invention. Those skilled in the art should understand that any other model or method capable of achieving the same or similar functions can be considered as an optional implementation of this invention.

[0153] Related terms: CONCH [1] UNI [2] ;MUSK [3] CLAM [4] TransMIL [5] LoRA [6] DoRA [7] Feather-24K[8] ; ABMIL [9] .

[0154] References: [1] Lu, M. Y., Chen, B., Williamson, D. F., Chen, R. J., Liang, I., Ding, T.,...&Mahmood, F. (2024). A visual - language foundation model for computational pathology. Nature medicine , 30 (3), 863 - 874.

[0155] [2] Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F., Jaume, G., Song, A. H.,...&Mahmood, F. (2024). Towards a general - purpose foundation model for computational pathology. Nature medicine , 30 (3), 850 - 862.

[0156] [3] Xiang, J., Wang, X., Zhang, X., Xi, Y., Eweje, F., Chen, Y.,...&Li, R. (2025). A vision–language foundation model for precision oncology. Nature , 638 (8051), 769 - 778.

[0157] [4] Lu, M. Y., Williamson, D. F., Chen, T. Y., Chen, R. J., Barbieri, M.,&Mahmood, F. (2021). Data - efficient and weakly supervised computational pathology on whole - slide images. Nature biomedical engineering , 5 (6), 555 - 570.

[0158] [5] Shao, Z., Bian, H., Chen, Y., Wang, Y., Zhang, J.,&Ji, X. (2021).Transmil: Transformer based correlated multiple instance learning for wholeslide image classification. Advances in neural information processing systems , 34 , 2136-2147。

[0159] [6] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S.,...&Chen, W. (2022). Lora: Low-rank adaptation of large language models. Iclr , 1 (2), 3。

[0160] [7] Liu, S. Y., Wang, C. Y., Yin, H., Molchanov, P., Wang, Y. C. F.,Cheng, K. T.,&Chen, M. H. (2024, July). Dora: Weight-decomposed low-rankadaptation. In Forty-first International Conference on Machine Learning 。

[0161] [8] Shao, D., Chen, R. J., Song, A. H., Runevic, J., Lu, M. Y., Ding,T.,&Mahmood, F. (2025). Do Multiple Instance Learning Models Transfer?. arXiv preprint arXiv:2506.09022 。

[0162] [9] Ilse, M., Tomczak, J.,&Welling, M. (2018, July). Attention-baseddeep multiple instance learning. In international conference on machine learning (pp. 2127-2136). PMLR。

[0163] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0165] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0167] The background section of this invention may include background information about the problems or environment in which the invention is being developed, and is not necessarily a description of prior art. Therefore, the content included in the background section does not constitute an admission of prior art by the applicant.

[0168] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A training method for a pathological image processing model, characterized in that, Includes the following steps: Obtain multiple image patches obtained from the division of a full-slice image; By fusing the neural network attention score and morphological quality score of each image patch, a joint score is generated, and a subset of high-value image patches is selected from the multiple image patches based on the joint score; Based on the high-value image patch subset, the pathological basic model and the task network are jointly trained to obtain an optimized pathological image processing model. The joint training includes: freezing the main parameters of the pathological basic model and updating only the parameters introduced by the parameter efficient adaptation structure in the pathological basic model and the parameters of the task network.

2. The training method according to claim 1, characterized in that, The neural network attention score is obtained by processing each image patch through a pre-trained multi-instance learning network.

3. The training method according to claim 1, characterized in that, The morphological quality score is obtained by analyzing the pixel data of each image block and calculating at least one morphological evaluation factor.

4. The training method according to claim 3, characterized in that, The morphological evaluation factors include: tissue proportion of the image patch, rationality of staining distribution of the image patch, color saturation and brightness variability of the image patch, information content of the image patch, nucleocytoplasmic ratio or cell density of the image patch, and background noise suppression factor of the image patch.

5. The training method according to claim 4, characterized in that, Calculating the background noise suppression factor includes: calculating the proportion of black pixels and / or white pixels in the image patch; if the proportion of black pixels is higher than a first threshold, or the proportion of white pixels is higher than a second threshold, then the morphological quality score of the image patch is reduced.

6. The training method according to claim 1, characterized in that, The process of fusing the neural network attention score and morphological quality score of each image patch includes multiplying the normalized neural network attention score by the normalized morphological quality score.

7. The training method according to claim 1, characterized in that, The pathological model is a pre-trained model.

8. The training method according to claim 1, characterized in that, The parameter efficient adaptation structure includes a low-rank adaptation LoRA module, a weight decomposition low-rank adaptation DoRA module, or an adapter module.

9. A pathological image processing system for implementing the training method according to any one of claims 1 to 8, characterized in that, The system includes: The image patch acquisition module is configured to acquire multiple image patches obtained by dividing a full-slice image. The high-value subset screening module is configured to generate a joint score by fusing the neural network attention score and morphological quality score of each image patch, and to screen a high-value image patch subset from the plurality of image patches based on the joint score; A joint training module is configured to jointly train the pathological baseline model and the task network based on the high-value image patch subset to obtain an optimized pathological image processing model; wherein, the joint training module is further configured to: freeze the main parameters of the pathological baseline model and only update the parameters introduced by the parameter efficient adaptation structure in the pathological baseline model and the parameters of the task network. A memory for storing the parameters of the pathological baseline model and the task network; The processor is used to call and execute the image patch acquisition module, the high-value subset screening module, and the joint training module.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the steps of the training method as described in any one of claims 1 to 8.

Images