Method and application method, device and medium for training pathological image processing model

Abstract

This application provides a training method, application method, device, and medium for a pathological image processing model. The training method includes acquiring a whole-slice image of a pathological sample and segmenting it to obtain an initial image block sequence comprising at least two initial image blocks; inputting each initial image block into a first main network to obtain a first type of activation map feature; performing a first split on the initial image block sequence to obtain at least two non-overlapping first image block sub-sequences; inputting each first image block in each first image block sub-sequence into a second main network to obtain a class activation map sub-feature for each first image block sub-sequence; concatenating all class activation map sub-features to obtain a second type of activation map feature; and updating the parameters of the original pathological image processing model based on the first and second type of activation map features to obtain a target pathological image processing model. This application can generate class activation map features with higher accuracy.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to training methods and application methods, equipment, and media for pathological image processing models. Background Technology

[0002] Histopathological examination, as the gold standard for disease diagnosis, requires experienced pathologists to perform microscopic analysis of tissue sections to provide a pathological diagnosis for suspicious lesions. Specifically, pathologists first visually examine stained tissue sections to assess their quality, then use a low-power microscope to initially identify lesion areas. Once a suspicious area is located, the pathologist will reposition it to the center of the field of view and switch to a high-power microscope to observe its fine structure in detail. Therefore, histopathological examination faces challenges such as high diagnostic subjectivity, a large workload, and the potential to overlook critical areas.

[0003] With advancements in digital scanning technology, physical slides can be converted into high-resolution whole-slide images (WSI), which can accurately capture the details of the entire pathological tissue. However, because WSI is obtained through high-resolution imaging and whole-slide scanning, it can reach a massive size of tens of thousands of pixels. This means that most algorithms for whole-slide images can only achieve disease diagnosis and struggle with automatic segmentation of lesion areas; even if they can, their accuracy is insufficient for application requirements. Summary of the Invention

[0004] The main objective of this application is to propose a training method, application method, device, and medium for a pathological image processing model, which aims to enhance the accuracy of pathological images in both disease diagnosis and automatic segmentation of lesion areas.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for training a pathological image processing model, the method comprising:

[0006] The original pathological image processing model includes a segmentation module, a first main network, and a second main network. The method includes:

[0007] Obtain whole-section images of the sample for pathological examination;

[0008] The segmentation module segments the whole pathological slide image of the sample to obtain an initial image block sequence, which includes at least two initial image blocks.

[0009] Each of the initial image blocks is input into the first main network for feature recognition to obtain the first type of activation map features of the whole pathological slice image of the sample;

[0010] The initial image block sequence is first split to obtain at least two first image block subsequences, and the first image blocks of any two first image block subsequences do not overlap.

[0011] Each first image block in each first image block subsequence is input into the second main network for feature recognition to obtain class activation map sub-features for each first image block subsequence;

[0012] The class activation map features of each of the first image block sub-sequences are concatenated to obtain the second class activation map features of the whole pathological slice image of the sample.

[0013] The original pathological image processing model is updated with parameters based on the first type of activation map features and the second type of activation map features to obtain the target pathological image processing model.

[0014] In some embodiments, before updating the parameters of the original pathological image processing model based on the first type of activation map features and the second type of activation map features to obtain the target pathological image processing model, the method further includes updating the second type of activation map features, specifically including:

[0015] The initial image block sequence is split a second time to obtain at least two second image block subsequences; wherein at least one first image block subsequence differs from one of the second image block sequences;

[0016] The class activation map sub-features of each second image block sub-sequence are concatenated to obtain auxiliary class activation map features;

[0017] The second type of activation graph features are updated based on the auxiliary type activation graph features.

[0018] In some embodiments, the first principal network includes a raw feature extractor, a raw slice-level encoder, and a raw classifier. The step of inputting each of the initial image patches into the first principal network for feature recognition to obtain the first type of activation map features of the sample pathological whole-slice image includes:

[0019] For each initial image patch, the original feature extractor is input sequentially to obtain a low-dimensional feature representation sequence;

[0020] The positions of each initial image block in the whole pathological slice image of the sample are obtained to obtain the initial image block position sequence;

[0021] The low-dimensional feature representation sequence and the initial image block position sequence are input into the original slice-level encoder for encoding to obtain hidden state features;

[0022] The hidden state features are input into the original classifier for classification processing to obtain the first type of activation map features of the whole pathological slice image of the sample.

[0023] In some embodiments, updating the parameters of the original pathological image processing model based on the first type of activation map features and the second type of activation map features to obtain the target pathological image processing model includes:

[0024] Feature mapping is performed based on the features of the first type of activation map to obtain the first mapping data;

[0025] Based on the features of the second type of activation map, feature mapping is performed to obtain the second mapping data;

[0026] The first loss is calculated based on the first mapping data and the sample labels of the pre-acquired whole-slice pathological images to obtain the first loss data;

[0027] A second loss is calculated based on the second mapping data and the sample labels to obtain the second loss data;

[0028] The third loss is calculated based on the first type of activation map features and the second type of activation map features to obtain the third loss data;

[0029] The original pathological image processing model is updated with parameters based on the first loss data, the second loss data, and the third loss data to obtain the target pathological image processing model.

[0030] In some embodiments, the original slice-level encoder includes a first encoding module and a second encoding module. The step of inputting the low-dimensional feature representation sequence and the initial image patch position sequence into the original slice-level encoder for encoding processing to obtain hidden state features includes:

[0031] The first encoding module projects the low-dimensional feature representation sequence onto an encoding space of a preset dimension to obtain a first matrix;

[0032] Perform two-dimensional sine and cosine position embedding based on the initial image block position sequence to obtain a second matrix with the same dimension as the first matrix;

[0033] Add the first matrix and the second matrix element by element to obtain the third matrix;

[0034] The hidden state features are obtained by encoding and transforming the third matrix through the second encoding module.

[0035] In some embodiments, the original slice-level encoder is obtained in the following way:

[0036] The training image block sequence is obtained by segmenting unlabeled slice-level sample pathological images through the segmentation module.

[0037] Obtain the training image patch feature sequence generated by the original feature extractor based on the training image patch sequence;

[0038] The training image block sequence is subjected to random masking to obtain an unmasked image block sequence;

[0039] The sequence of unmasked image blocks is input into the encoder to obtain the feature sequence of unmasked image blocks;

[0040] The unmasked image patch feature sequence is filled using a learnable mask vector to obtain a filled image patch feature sequence that is consistent with the order of the training image patch sequence.

[0041] The feature sequence of the filled image patch is input into the decoder to obtain the mask reconstruction feature sequence;

[0042] The similarity between the training image patch feature sequence and the mask reconstruction feature sequence is calculated, and the encoder parameters are updated based on the similarity to obtain the original slice-level encoder.

[0043] To achieve the above objectives, a second aspect of this application provides a pathological image processing method, the method comprising:

[0044] Obtain the target pathological whole-section image of the target object;

[0045] The target pathological whole-slice image is processed by the target pathological image processing model obtained by the training method proposed in the first aspect to obtain target class activation map features and multiple target image block positions; wherein, the target image block is the position of the target image block obtained by segmenting the target pathological whole-slice image by the segmentation module;

[0046] Image reconstruction is performed based on the target class activation map features and the target image block positions to obtain the target class activation map;

[0047] The target class activation map is masked according to a preset segmentation threshold to obtain a target pathological mask map.

[0048] In some embodiments, the target class activation map features include candidate class activation map scores for at least two candidate categories, and the image reconstruction process based on the target class activation map features and the target image patch location to obtain the target class activation map includes:

[0049] The target category of the target pathological whole slice image is determined based on the candidate class activation map score of each candidate category;

[0050] The target channel of the channel classifier is determined based on the target category;

[0051] The target class activation map features are classified by channel weights of the target channels to obtain the target class activation map score of the target category;

[0052] The target class activation graph score is activated to obtain target class activation data;

[0053] The target class activation data is standardized to obtain target class standardized data;

[0054] The target class normalized data and the target image block positions are reconstructed to obtain the target class activation map.

[0055] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first or second aspect.

[0056] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first or second aspect.

[0057] The pathological image processing model training method, electronic device, and storage medium proposed in this application acquire whole-slice images of pathological samples; segment the whole-slice images of the pathological samples using a segmentation module to obtain an initial image block sequence, which includes at least two initial image blocks; input each initial image block into a first main network for feature recognition to obtain the first class activation map features of the whole-slice images of the pathological samples. By segmenting large-sized whole-slice images of pathological samples into small-sized image blocks, only one image block needs to be loaded and processed each time, significantly reducing memory and computational requirements. It also avoids feature dilution or neglect of details caused by excessively large overall image sizes, thereby improving feature extraction accuracy. Furthermore, by introducing class activation maps that can generate category-specific heatmaps into pathological image processing, features used for classification diagnosis can also be directly used for automatic segmentation of lesion areas. Furthermore, the initial image block sequence is first split to obtain at least two first image block sub-sequences, where the first image blocks of any two first image block sub-sequences do not overlap. Each first image block in each first image block sub-sequence is input into a second main network for feature recognition, obtaining class activation map sub-features for each first image block sub-sequence. The class activation map sub-features of each first image block sub-sequence are then concatenated to obtain the second class activation map features of the sample pathological whole-slice image. By splitting the image into sub-sequences before feature recognition, the independence and resolution of local features are enhanced. Furthermore, the parameters of the original pathological image processing model are updated based on the first and second class activation map features to obtain the target pathological image processing model. The target pathological image processing model trained using the global features obtained by the first main network and the local features obtained by the second main network considers both global and local aspects, resulting in the generation of more accurate class activation map features. This enhances the accuracy of classification diagnosis and automatic lesion region segmentation of pathological whole-slice images when using class activation map features for classification and segmentation.

[0058] The pathological image processing method, electronic device, and storage medium proposed in this application acquire a target pathological whole-slice image of a target object; perform image processing on the target pathological whole-slice image using a target pathological image processing model obtained through the training method proposed in the first aspect, to obtain target class activation map features and multiple target image block positions; wherein, the target image block is the position of the target image block obtained by segmenting the target pathological whole-slice image by the segmentation module; perform image reconstruction processing based on the target class activation map features and the target image block positions to obtain a target class activation map; and perform masking processing on the target class activation map according to a preset segmentation threshold to obtain a target pathological mask map. This method obtains more accurate class activation map features through the target pathological image processing model, enabling both classification diagnosis and automatic lesion region segmentation based on class activation map features, and enhancing the accuracy of both classification diagnosis and automatic lesion region segmentation of the pathological whole-slice image, thereby improving the accuracy in both classification diagnosis and automatic lesion region segmentation. Attached Figure Description

[0059] Figure 1 This is a flowchart of the training method for the pathological image processing model provided in the embodiments of this application;

[0060] Figure 2 yes Figure 1 Flowchart of step S103;

[0061] Figure 3 yes Figure 2 Flowchart of step S203;

[0062] Figure 4 This is another flowchart of the training method for the pathological image processing model provided in the embodiments of this application;

[0063] Figure 5 yes Figure 1 Flowchart of step S107;

[0064] Figure 6 This is a flowchart of the pathological image processing method provided in the embodiments of this application;

[0065] Figure 7 yes Figure 6 Flowchart of step S603;

[0066] Figure 8 This is a flowchart illustrating the training method and application of the pathological image processing model provided in the embodiments of this application;

[0067] Figure 9 This is a design flowchart provided in an embodiment of this application;

[0068] Figure 10This is a visual comparison chart of the image processing results obtained by the method provided in this application embodiment and the processing results of other existing methods;

[0069] Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0071] It should be noted that although functional modules are divided in the system diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0073] First, let's analyze some of the terms used in this application:

[0074] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0075] Histopathological examination, as the gold standard for disease diagnosis, requires experienced pathologists to perform microscopic analysis of tissue sections to provide a pathological diagnosis for suspicious lesions. Specifically, pathologists first visually examine stained tissue sections to assess their quality, then use a low-power microscope to initially identify lesion areas. Once a suspicious area is located, the pathologist will reposition it to the center of the field of view and switch to a high-power microscope to observe its fine structure in detail. Therefore, histopathological examination faces challenges such as high diagnostic subjectivity, a large workload, and the potential to overlook critical areas.

[0076] With advancements in digital scanning technology, physical slides can be transformed into high-resolution whole-slice images, which can accurately capture the details of the entire pathological tissue. However, because whole-slice images (WSI) are obtained through high-resolution imaging and whole-slice scanning, they can reach enormous sizes of tens of thousands of pixels. This means that most algorithms for whole-slice images can only achieve disease diagnosis and struggle with automatic segmentation of lesion areas; even if they can, their accuracy is insufficient for application requirements.

[0077] Based on this, embodiments of this application provide training methods, application methods, devices, and media for pathological image processing models, aiming to enhance the accuracy of pathological images in both disease diagnosis and automatic segmentation of lesion areas.

[0078] The training method and application method of the pathological image processing model, electronic device and storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the training method of the pathological image processing model in the embodiments of this application is described.

[0079] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0080] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0081] The training method for the pathological image processing model and the pathological image processing method provided in this application both relate to the field of image processing. The training method for the pathological image processing model and the pathological image processing method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the training method for the pathological image processing model and the pathological image processing method, but is not limited to the above forms.

[0082] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0083] It should be noted that in various specific embodiments of this application, when processing data related to the identity or characteristics of an object, such as object information, object behavior data, object historical data, and object location information, the object's permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining sensitive personal information of an object, separate permission or consent from the object is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the object's separate permission or consent is the necessary object-related data required for the proper functioning of the embodiments of this application obtained.

[0084] Please see Figure 1 , Figure 1This is an optional flowchart of a training method for a pathological image processing model provided in this application embodiment. The original pathological image processing model includes a segmentation module, a first main network, and a second main network. Figure 1 The method may include, but is not limited to, steps S101 to S107:

[0085] Step S101: Obtain whole pathological slide images of the sample;

[0086] Step S102: The whole pathological slide image of the sample is segmented by the segmentation module to obtain an initial image block sequence, which includes at least two initial image blocks.

[0087] Step S103: Input each initial image block into the first main network for feature recognition to obtain the first type of activation map features of the whole pathological slice image of the sample;

[0088] Step S104: Perform a first split on the initial image block sequence to obtain at least two first image block subsequences, wherein the first image blocks of any two first image block subsequences do not overlap;

[0089] Step S105: Input each first image block in each first image block sub-sequence into the second main network for feature recognition to obtain the class activation map sub-features of each first image block sub-sequence;

[0090] Step S106: The class activation map features of each first image block sub-sequence are spliced ​​together to obtain the second class activation map features of the whole pathological slice image of the sample.

[0091] Step S107: Update the parameters of the original pathological image processing model based on the first type of activation map features and the second type of activation map features to obtain the target pathological image processing model.

[0092] Steps S101 to S107, as illustrated in this embodiment, involve acquiring a whole-slice image of a pathological sample; segmenting the whole-slice image using a segmentation module to obtain an initial image block sequence, which includes at least two initial image blocks; inputting each initial image block into a first main network for feature recognition to obtain a first-class activation map feature of the whole-slice image. By segmenting a large-sized whole-slice image into small-sized image blocks, only one image block needs to be loaded and processed each time, significantly reducing memory and computational requirements. This also avoids feature dilution or neglect of details caused by an excessively large overall image size, thereby improving feature extraction accuracy. Furthermore, by introducing class activation maps capable of generating category-specific heatmaps into pathological image processing, features used for classification diagnosis can also be directly used for automatic segmentation of lesion areas. Furthermore, the initial image block sequence is first split to obtain at least two first image block sub-sequences, where the first image blocks of any two first image block sub-sequences do not overlap. Each first image block in each first image block sub-sequence is input into a second main network for feature recognition, obtaining class activation map sub-features for each first image block sub-sequence. The class activation map sub-features of each first image block sub-sequence are then concatenated to obtain the second class activation map features of the sample pathological whole-slice image. By splitting the image into sub-sequences before feature recognition, the independence and resolution of local features are enhanced. Furthermore, the parameters of the original pathological image processing model are updated based on the first and second class activation map features to obtain the target pathological image processing model. The target pathological image processing model trained using the global features obtained by the first main network and the local features obtained by the second main network considers both global and local aspects, resulting in the generation of more accurate class activation map features. This enhances the accuracy of classification diagnosis and automatic lesion region segmentation of pathological whole-slice images when using class activation map features for classification and segmentation.

[0093] In step S101 of some embodiments, the whole-section pathology image is a sample image used to fine-tune the model and can be obtained from a publicly available multicenter pathology WSI dataset. The WSI datasets include CAMELYON and PANDA. Specifically, the CAMELYON dataset includes CAMELYON16 and CAMELYON17, which contain lymph node tissue sections for detecting breast cancer metastasis, with a maximum resolution of 40× and an average image size of approximately 10. 5 ×10 5Pixel-level. CAMELYON16 includes 270 slices for training and 129 slices from the official test set. All tissue regions in metastatic slices are provided with pixel-level tumor / normal labels. CAMELYON17 includes 1000 slices. Since the slice-level labels for the 500 slices in the official CAMELYON17 test set are not publicly available, only the training portion containing 500 slices with slice-level diagnostic labels was used, of which only 50 provided pixel-level region annotations. Merging CAMELYON16 and CAMELYON17 resulted in a training set of 770 slices (512 negative and 258 positive) and a test set of 129 slices (81 negative and 48 positive). PANDA is the most comprehensive multicenter dataset on Gleason grading of prostate tissue, with a maximum resolution of 20× and an average image size of approximately 10. 4 ×10 4 Pixels. After obtaining Gleason scores, biopsy slides are converted into ISUP grades from 1 to 5, where ISUP grade 0 indicates benign tissue. In this embodiment, slides with ISUP scores of 0 and 5 were screened, and slides with low quality or insufficient resolution to support segmentation were removed, ultimately yielding 692 slides (360 negative and 332 positive).

[0094] In some embodiments, prior to step S102, the slices of the CAMELYON dataset are adjusted to a standard resolution of 0.5 MPP, or a magnification of 20×, to ensure that all slices are at a uniform scale.

[0095] In step S102 of some embodiments, the segmentation module is used to segment the whole-slice image of the sample pathology to divide it into multiple initial image blocks. Specifically, firstly, a 64x thumbnail of the whole-slice image of the sample pathology is obtained and converted from the RGB color space to the HSV color space. Next, the saturation channels of the image are thresholded to obtain a binary mask for the foreground tissue region. The specific process is as follows: firstly, medium blur is applied to smooth the edges, and small gaps and holes are filled by morphological closing operation; secondly, the approximate contours of the detected foreground objects are filtered by an optional area threshold, and contours with an area smaller than the threshold are discarded. The initial value of the area threshold can be set to 8, which can be manually adjusted according to the filtering effect, but using a higher area threshold will result in less foreground and more background detection. Then, the whole-slice image of the sample pathology is adjusted to a standard resolution of 0.5μm per pixel (MPP), and N non-overlapping image blocks x∈R of size 256x256 are cropped from the segmented foreground contours. N×256×256 .

[0096] Prior to step S103 in some embodiments, the training method for the pathological image processing model further includes pre-training a first principal network, which is used to obtain class activation map features based on each initial image patch of the sample pathological whole-slice image. The first principal network includes a raw feature extractor, a raw slice-level encoder, and a raw classifier. Class activation map features are a feature representation in the form of a heatmap, used to display the model's response intensity to each location in the input sample pathological whole-slice image; that is, to highlight the regions that the model considers most relevant to the target category, and also to provide semantically important feature representations for subsequent prediction and segmentation tasks.

[0097] The original feature extractor is used to extract features from the input image patch to obtain a low-dimensional feature representation. This original feature extractor can be a model pre-trained from a ViT model, which takes a 256×256 image patch as input and is pre-trained using the self-supervised learning method DINOv2. DINOv2 learns high-quality image representations through a student-teacher architecture and a multi-cropping enhancement method. DINOv2 employs a discriminative self-supervised learning objective, which can be viewed as a combination of image-level objectives, image patch-level objectives, and KoLeo regularization. In its implementation, DINOv2 crops the same 256×256 image patch into a set of 96×96 local views and a 224×224 global views. The student model receives all views, while the teacher model only receives the global view. In this way, the image-level objective prompts the model to learn the "local to global" relationship. Furthermore, DINOv2 performs image patch occlusion on the image patches passed to the student model, while the teacher model still receives the complete image patches. In this way, the patch-level objective, known as the iBOT loss, minimizes the difference in occluded image patch features between the student and teacher models. Simultaneously, to ensure a uniform distribution of learned image representations within the same batch, the KoLeo regularization term maximizes the minimum distance between representation vectors within the batch. Moreover, to better prevent model collapse, DINOv2 employs an improved method that advances teacher softmax centering to Sinkhorn-Knopp centering normalization.

[0098] The original slice-level encoder is used to encode the low-dimensional feature representation sequence and the positional information of the initial image patch corresponding to each low-dimensional feature representation in the whole slice image of the sample pathology, thus obtaining hidden state features. The hidden state features are representation vectors that integrate image patch features and spatial positional relationships, containing deeper semantic information about the image. This original slice-level encoder can be a model pre-trained as follows: First, a training image patch sequence is obtained after the unlabeled slice-level sample pathology image is segmented by a segmentation module. This training image patch sequence consists of multiple training image patches arranged in the positional order of the image patches on the slice-level sample pathology image (referred to as the first order). Next, a training image patch feature sequence is obtained by the original feature extractor based on the training image patch sequence. This training image patch feature sequence consists of training image patch features corresponding to the training image patches arranged in the first order. Then, the training image patch sequence is subjected to random masking to obtain an unmasked image patch sequence. This random masking can mask a preset percentage of training image patches, such as 75%, 60%, or 80%, without limitation. The image patches in the unmasked image patch sequence are also arranged in a first order. Next, the unmasked image patch sequence is input into the encoder to obtain an unmasked image patch feature sequence, which consists of the unmasked image patch features corresponding to the unmasked image patches arranged in a first order. Subsequently, the unmasked image patch feature sequence is padded using a learnable mask vector to obtain a padded image patch feature sequence with the same order as the training image patch sequence. The learnable mask vector is a vector matching the size of the input training image patch feature sequence and learns how to recover occluded / masked features during training. The padded process completes the unmasked image patch feature sequence according to the first order. Finally, the padded image patch feature sequence is input into the decoder to obtain a mask reconstruction feature sequence. Finally, the similarity between the training image patch feature sequence and the mask reconstruction feature sequence is calculated, and the encoder parameters are updated based on the similarity to obtain the original slice-level encoder; where the similarity can be cosine similarity or other similarity. In one specific implementation, assuming the training image patch sequence is [a1,a2,a3,a4,a5], the training image patch feature sequence is [b1,b2,b3,b4,b5]. After masking 80% of the image patches, the unmasked image patch sequence [b2] is obtained, which is input into the encoder for encoding processing to obtain the unmasked image patch feature sequence [c2]. This encoder can be constructed using Longnet. Then, the learnable mask vectors are used to fill [c2] to obtain the filled image patch feature sequence [d1,c2,d3,d4,d5].Next, [d1,c2,d3,d4,d5] is input into the decoder for decoding to obtain the mask reconstruction feature sequence [e1,e2,e3,e4,e5]. This decoder can also be constructed using Longnet. Finally, the similarity between the corresponding features in the training image patch feature sequence [b1,b2,b3,b4,b5] and the mask reconstruction feature sequence [e1,e2,e3,e4,e5] is calculated, and the encoder parameters are updated based on this similarity. In this specific embodiment, the same numbers in the sequence represent the same image patch; for example, b2, c2, d2, and e2 all correspond to the training image patch a2.

[0099] The original classifier can be a one-dimensional convolutional layer with a kernel size of 1. This original classifier is used to classify the hidden state features to obtain class activation map features.

[0100] Please see Figure 2 In step S103 of some embodiments, the training method for the pathological image processing model may include, but is not limited to, steps S201 to S204:

[0101] Step S201: For each initial image patch, input it sequentially into the original feature extractor to obtain a low-dimensional feature representation sequence;

[0102] Step S202: The position of each initial image block in the whole pathological slice image of the sample is obtained to obtain the initial image block position sequence;

[0103] Step S203: Input the low-dimensional feature representation sequence and the initial image block position sequence into the original slice-level encoder for encoding processing to obtain the hidden state features;

[0104] Step S204: Input the hidden state features into the original classifier for classification processing to obtain the first type of activation map features of the whole slice image of the sample pathology.

[0105] In step S201 of some embodiments, the low-dimensional feature representation sequence includes multiple low-dimensional feature representations, each of which describes basic information such as texture, color, and spatial information of the corresponding initial image patch. The spatial information includes shape features, symmetry, and the relative distribution characteristics of spatial positions. The order of the low-dimensional feature representation sequence is consistent with the order of the initial image patch sequence. Specifically, following the order of the initial image patch sequence, the initial image patches are sequentially input into the original feature extractor to obtain the low-dimensional feature representation corresponding to each initial image patch. These multiple low-dimensional feature representations are then arranged sequentially into a low-dimensional feature representation sequence. The low-dimensional feature representation sequence can be represented as G∈R. N×M M is the dimension of the low-dimensional feature representation.

[0106] In step S202 of some embodiments, the initial image block position sequence includes multiple initial image block positions. Each initial image block position represents the position information of the initial image block in the whole slide image of the sample pathology. The position information can be the image block coordinates including horizontal and vertical positions. The order of the initial image block position sequence is consistent with the initial image block sequence. Specifically, the origin of the coordinate system can be the upper left corner of the whole slide image of the sample pathology, the horizontal axis can be the direction extending from the origin to the image length direction, and the vertical axis can be the direction extending from the origin to the image width direction. Each initial image block position can be the coordinates p∈R of the upper left corner of the initial image block on the whole slide image of the sample pathology. 2 .

[0107] Please see Figure 3 The original slice-level encoder includes a first encoding module and a second encoding module. In step S203 of some embodiments, the training method of the pathological image processing model may include, but is not limited to, steps S301 to S304:

[0108] Step S301: The low-dimensional feature representation sequence is projected onto a coding space of a preset dimension through the first coding module to obtain the first matrix;

[0109] Step S302: Perform two-dimensional sine and cosine position embedding based on the initial image block position sequence to obtain a second matrix with the same dimension as the first matrix;

[0110] Step S303: Add the first matrix and the second matrix element by element to obtain the third matrix;

[0111] Step S304: The third matrix is ​​encoded and transformed by the second encoding module to obtain the hidden state features.

[0112] In step S301 of some embodiments, the first encoding module can be a fully connected layer used to project the low-dimensional feature representation sequence into a matrix of a preset dimension. Assuming the low-dimensional feature representation sequence consists of 2000 low-dimensional feature representations of 1536 dimensions, if the preset dimension encoding space is 768 dimensions, then the resulting first matrix is ​​a 2000×768 matrix. This approach makes the dimensions of the output first matrix more compact, reducing computational complexity and adapting to the input requirements of subsequent modules.

[0113] In step S302 of some embodiments, two-dimensional sine and cosine position embedding is a position encoding method that converts the spatial position (coordinates) of each image patch into a vector using sine and cosine functions, embedding it into a fixed-dimensional space, thereby obtaining a second matrix based on the embedding vectors corresponding to all image patches. Its purpose is to incorporate positional information into the model's input, enabling the model to utilize spatial relationships.

[0114] Specifically, the one-dimensional sine and cosine position embedding vector PE is calculated as follows:

[0115] PE (pos,2i) =sin(pos / 10000) 2i / E ),

[0116] PE (pos,2i+1) =cos(pos / 10000) 2i / E ),

[0117] Among them, PE (pos,2i) PE represents the 2i-th element in the one-dimensional sine and cosine position embedding vector. (pos,2i+1) Let pos represent the (2i+1)th element in the one-dimensional sine and cosine position embedding vector, where pos represents the one-dimensional position information, and i satisfies 2i≤E, 2i+1≤E. If the preset coding space is 768-dimensional, then E=768. Assume the coordinates (pos) of an image patch... 1 ,pos 2 ), where pos 1 pos represents the x-coordinate of the image patch on the whole pathological slide image of the sample. 2 Let PE be the ordinate of the image patch on the whole pathological slice image of the sample, and its two-dimensional sine and cosine position embedding vector. 2D The calculation method is as follows:

[0118]

[0119] Among them, PE 2D The first matrix is ​​an E-dimensional vector generated based on the coordinates of the image patches. After obtaining the vectors corresponding to all image patches, they are arranged in the order of the initial image patch sequence to obtain the second matrix. Thus, assuming there are 2000 initial image patches and E = 768, the first matrix is ​​a 2000×768 matrix, and the second matrix, composed of vectors calculated from the two-dimensional sine and cosine position embedding, is also a 2000×768 matrix.

[0120] In step S303 of some embodiments, element-wise addition refers to directly adding the elements at each corresponding position of the first and second matrices to form the third matrix. Therefore, the third matrix simultaneously contains both feature information and positional information of the image patch, and the third matrix has the same dimension as the first and second matrices.

[0121] In step S304 of some embodiments, the second encoding module, trained by LongNet, is used to encode the third matrix as hidden state features. The hidden state features can be represented as F∈R N×E .

[0122] It's worth noting that LongNet introduces Dilated Attention, enabling the model architecture to support inputs of billions of bytes in length, thus allowing for the computation of self-attention scores on extremely long sequences of pathological image patches. Dilated Attention significantly reduces computational complexity, meaning it lowers computational costs while efficiently capturing both long-range and short-range information.

[0123] Steps S301 to S304 above can effectively combine local features and location information to generate hidden state features that can capture the global and local relationships of the image, providing rich semantic information for subsequent classification and tasks.

[0124] In step S204 of some embodiments, the first type of activation map features are obtained based on the hidden state features, and the first type of activation map features can be represented as A∈R N×c This means that the class activation map scores simultaneously include C categories. Each class activation map score represents the strength of the correlation between an image patch and its category. All class activation map scores sum to 1. For example, C can have two categories: one negative and one positive. In this case, the class activation map scores include both negative and positive class activation map scores, and the sum of the negative and positive scores is 1. The original classifier, as shown above, can be a one-dimensional convolutional layer θ with a kernel size of 1, used to classify F∈R... N×E Generate the first type of activation graph features A∈R N×C .

[0125] Steps S201 to S204 above first extract low-dimensional feature representations of each initial image patch and combine them with the spatial location information of each image patch in the whole pathological slice image. These are then input into the original slice-level encoder for encoding, generating hidden state features. Next, the hidden state features are classified by a classifier to generate first-class activation map features. This process effectively fuses local features and spatial location information to generate activation map features that reflect key lesion areas in the image, thereby improving the classification accuracy of pathological images and providing a reliable foundation for subsequent lesion area detection and segmentation tasks.

[0126] In step S104 of some embodiments, the first splitting involves determining at least one splitting point, or at least two splitting points, in the initial image block sequence, with at least one image block separating each splitting point. Based on these splitting points, the initial image block sequence is divided into at least two first image block subsequences. Each first image block subsequence may be of equal or unequal length, each first image block subsequence includes at least one first image block, and the first image blocks of any two first image block subsequences do not overlap.

[0127] In other embodiments, the whole pathological slice image of the sample can be directly split, and multiple image block subsequences of equal or unequal length can be divided from the initial image block sequence according to the position information of the initial image block.

[0128] In step S105 of some embodiments, the second main network is composed of at least two first main networks in parallel. Each first main network performs feature recognition on a corresponding first image block sub-sequence. Therefore, the number of first main networks in the second main network is the same as the number of first image block sub-sequences obtained by splitting. The meaning of the class activation map sub-features of the first image block sub-sequence is essentially the same as the first class activation map features. Specifically, each first image block in each first image block sub-sequence is input into the corresponding first main network in the second main network for feature recognition to obtain the class activation map sub-features of each first image block sub-sequence.

[0129] In step S106 of some embodiments, the class activation map sub-features of each first image block sub-sequence are spliced ​​together according to the image block order of the initial image block sequence to obtain the second class activation map features of the whole slice image of the sample pathology. The meaning of the second class activation map features is essentially the same as that of the first class activation map.

[0130] In a specific implementation scenario of steps S104 to S106, from the initial image patch sequence X∈R N×256×256 The image is divided into four subsequences of unequal length, {X}. 1 ,X 2 ,X 3 ,X 4}, where in subsequence X 1 The image patch in the image is located in the upper left region of four non-overlapping regions, in subsequence X. 2 The image patch in the image is located in the lower left region of four non-overlapping regions, in subsequence X. 3 The image patch in the image is located in the upper right region of four non-overlapping regions, in subsequence X. 4 The image patch in the image is located in the lower right region of four non-overlapping regions. Subsequently, each subsequence X... k Each subsequence is processed by an extractor, encoder, and classifier to obtain its respective subsequence class activation map features {A}. 1 A 2 A 3 A 4},in This corresponds to the features of the i-th image patch in the first subsequence (located in the upper left region of the four non-overlapping regions). Finally, the subsequence class activation map features {A} are applied. 1 A 2 A 3 A 4The features A of the second type of activation map are merged into the first type of activation map A, which has the same dimension as the first type of activation map A. s The specific implementation process is as follows: For example, according to Find the coordinates of the corresponding image patches in the first image patch subsequence, locate the positions of the extracted features of these image patches in the first type of activation map feature A, and then... The i-th feature in the second activation map is inserted into feature A. s The corresponding positions, i.e., A and A s The order of the corresponding image patches for the features in the image remains consistent.

[0131] Please see Figure 4 In some embodiments, prior to step S107, the training method for the pathological image processing model further includes updating the features of the second type of activation map, which may include, but is not limited to, steps S401 to S403:

[0132] Step S401: Perform a second split on the initial image block sequence to obtain at least two second image block sub-sequences; wherein at least one first image block sub-sequence differs from one second image block sub-sequence;

[0133] Step S402: Concatenate the class activation map sub-features of each second image block sub-sequence to obtain auxiliary class activation map features;

[0134] Step S403: Update the features of the second type of activation map based on the features of the auxiliary type activation map.

[0135] In step S401 of some embodiments, the implementation of the second split is similar to that of the first split, except that at least one of the at least two second image block sub-sequences obtained by the second split is different from both the first image block sub-sequence and vice versa.

[0136] In step S402 of some embodiments, the class activation map sub-features of each second image block sub-sequence are spliced ​​together according to the image block order of the initial image block sequence to obtain auxiliary class activation map features of the whole slice image of the sample pathology. The meaning of the auxiliary class activation map features is essentially the same as that of the first class activation map features.

[0137] In step S403 of some embodiments, the update process may be to perform a weighted summation or convolutional fusion of the auxiliary class activation map features and the second class activation map features to obtain the fused class activation map features, which are used to replace the original second class activation map features.

[0138] Steps S401 to S403 above combine the first and second segmentation methods to obtain features from different segmentation perspectives, avoiding insufficient information or bias caused by a single perspective. Specifically, the auxiliary activation map features provide supplementary information for the second activation map features, particularly the feature enhancements brought by fine-grained semantics or different segmentation methods. After fusing features from different segmentation perspectives, the model can more comprehensively capture the global and local relationships of the whole-slice image, improving the accuracy of recognition and classification.

[0139] Please see Figure 5 In step S107 of some embodiments, the training method for the pathological image processing model may include, but is not limited to, steps S501 to S506:

[0140] Step S501: Perform feature mapping based on the features of the first type of activation map to obtain the first mapping data;

[0141] Step S502: Perform feature mapping based on the features of the second type of activation map to obtain the second mapping data;

[0142] Step S503: Calculate the first loss based on the first mapping data and the sample labels of the pre-acquired whole-slice images of the pathological tissue samples to obtain the first loss data;

[0143] Step S504: Calculate the second loss based on the second mapping data and sample labels to obtain the second loss data;

[0144] Step S505: Calculate the third loss based on the features of the first type of activation map and the features of the second type of activation map to obtain the third loss data;

[0145] Step S506: Update the parameters of the original pathological image processing model based on the first loss data, the second loss data, and the third loss data to obtain the target pathological image processing model.

[0146] In step S501 of some embodiments, the first type of activation map features are feature-mapped by the first global average pooling module to obtain the predicted first logarithmic value. The first logarithmic value includes the predicted value corresponding to each category, where the category can refer to whether the pathological whole slide image of the sample is positive or negative. The first logarithmic value is the first mapping data, which can be represented as follows:

[0147] In step S502 of some embodiments, the second type of activation map features are feature-mapped by the second global average pooling module to obtain the predicted second logarithmic value. The first logarithmic value includes the predicted values ​​corresponding to each category. This second logarithmic value is the second mapping data, which can be represented as follows:

[0148] In step S503 of some embodiments, the first loss data represents the difference between the first mapping data and the sample label, and the first loss data can be calculated using the cross-entropy loss function. The sample label Y of the pre-acquired whole-section pathological image is either positive or negative. Specifically, it can be calculated using the first loss function. For the first mapping data The first loss data is obtained by calculating the loss with the sample label Y.

[0149] In step S504 of some embodiments, the second loss data represents the difference between the second mapping data and the sample labels. The second loss data can also be calculated using the cross-entropy loss function. Specifically, it can be calculated using the second loss function... For the first mapping data The second loss data is obtained by calculating the loss with the sample label Y.

[0150] In step S505 of some embodiments, the third loss data represents the matching relationship between the first type of activation map features and the second type of activation map features. The third loss data can be calculated through reconstruction regularization. Specifically, it can be calculated through reconstruction regularization. For the first type of activation map feature A and the second type of activation map feature A s Matching is performed to obtain the third loss data.

[0151] In step S506 of some embodiments, the network structure of the target pathological image processing model is the same as that of the original pathological image processing model, which will not be described in detail here. Specifically, by adjusting the overall loss function... Minimizing the computational result allows for backpropagation of the original pathological image processing model, updating the parameters and ultimately obtaining the target pathological image processing model. Here, α is the weight parameter.

[0152] Steps S501 to S506 above improve the stability of classification by using the first loss data and the second loss data for backpropagation; and enhance the automatic delineation of key regions and boundaries by using the third loss data (i.e., highlighting the lesion areas that should be of most concern in the whole slice image of the sample pathology by minimizing the reconstruction loss of local and global class activation map features), and also enhance the local details of the model.

[0153] Please see Figure 6 , Figure 6 This is an optional flowchart of a pathological image processing method provided in the embodiments of this application. The method may include, but is not limited to, steps S601 to S604:

[0154] Step S601: Obtain the target pathological whole slice image of the target object;

[0155] Step S602: The target pathological whole slice image is processed by the target pathological image processing model obtained by the training method described above to obtain the target class activation map features and multiple target image block positions; wherein, the target image block positions are the positions of the target image blocks obtained by segmenting the target pathological whole slice image by the segmentation module.

[0156] Step S603: Perform image reconstruction processing based on the target class activation map features and the target image block position to obtain the target class activation map;

[0157] Step S604: Mask the target class activation map according to the preset segmentation threshold to obtain the target pathology mask map.

[0158] In step S601 of some embodiments, a high-resolution scan of the entire stained pathological tissue section of the target object is performed using a whole-section scanner to obtain a whole-section image of the target pathology.

[0159] In step S602 of some embodiments, the acquisition of the target pathological image processing model has been described in detail above and will not be repeated here. Specifically, the target pathological whole-slice image is input into the target pathological image processing model for image processing to obtain target class activation map features and multiple target image block positions. The target image block positions are obtained by segmenting the target pathological whole-slice image using the segmentation module in the target pathological image processing model. The meaning of the target class activation map features is essentially the same as that of the first type of activation map features. Multiple target image block positions can also be understood as forming a target image block position sequence.

[0160] Please see Figure 7 The target class activation map features include candidate class activation map scores for at least two candidate categories. In step S603 of some embodiments, the pathological image processing method may include, but is not limited to, steps S701 to S706:

[0161] Step S701: Determine the target category of the target pathological whole slice image based on the candidate class activation map scores of each candidate category;

[0162] Step S702: Determine the target channel of the channel classifier based on the target category;

[0163] Step S703: Classify the target class activation map features by channel weights to obtain the target class activation map score for the target category;

[0164] Step S704: Activate the target class activation map score to obtain target class activation data;

[0165] Step S705: Standardize the target class activation data to obtain target class standardized data;

[0166] Step S706: Reconstruct the target class standardized data and target image block positions to obtain the target class activation map.

[0167] It should be noted that the meaning of the target class activation map feature is essentially the same as that of the first type activation map score, and the target class activation map feature A... ′ ∈R N×C It also includes the candidate class activation graph scores of C candidate categories, and the sum of all candidate class activation graph scores is 1. For example, the number of C can be 2, one candidate category is negative and the other candidate category is positive. In this case, the candidate class activation graph scores include the negative class activation graph score and the positive class activation graph score, and the sum of the negative class activation graph score and the positive class activation graph score is 1.

[0168] In step S701 of some embodiments, the target category of the target pathological whole slide image indicates which category the target pathological whole slide image belongs to. The target category is the candidate category corresponding to the candidate class activation map score with the highest score among all candidate class activation map scores. Specifically, the maximum value is filtered among all candidate class activation map scores to obtain the largest candidate class activation map score, and then the candidate category corresponding to the largest candidate class activation map score is taken as the target category of the target pathological whole slide image.

[0169] In step S702 of some embodiments, a channel classifier is used to process and analyze multi-channel information from class activation map features. Each channel in the channel classifier corresponds to a candidate category or a component of a specific feature. The goal of the channel classifier is to weight or select these channels to ultimately extract the features corresponding to the target category. Therefore, the target channel is the channel corresponding to the target category.

[0170] In step S703 of some embodiments, the weight of the target channel can be represented as the weight of other channels being 0, while the weight of the target channel is 1. Specifically, multiplying the weight of the target channel by the target class activation map feature yields the target class activation map score A for the target category. ′ c .

[0171] In steps S704 and S705 of some embodiments, the activation process is used to perform a nonlinear transformation on the target class activation map scores to enhance feature saliency. This activation process can specifically be implemented using the ReLU activation function. The normalization process is used to normalize the target class activation data, adjusting it to a uniform scale. This normalization process can specifically be implemented using the Normalization technique. The mathematical expression for this activation and normalization processes is A″.c =(A′) c -Min(A′ c )) / (Max(A′ c )-Min(A′ c ), where A″ c Standardize the data for the target class.

[0172] In step S706 of some embodiments, the reconstruction process is used to map the target normalized data back to the spatial distribution of the target pathological whole slide image according to the image patch positions. The target class activation map obtained by the reconstruction process represents the response intensity of each image patch in the target pathological whole slide image to the target class, and can show the activation distribution of the target class to intuitively reflect the classification region. Furthermore, the size of the target class activation map is the same as that of the target pathological whole slide image. Specifically, through the target class normalized data A″... x ∈R N and the corresponding coordinates P′∈R N×2 Generate a class activation map I with the same size W×H as the target pathological whole slide image, and its mathematical representation is as follows:

[0173]

[0174] Where i is the i-th image patch.

[0175] Steps S701 to S706 described above can identify the target category based on the activation map score and accurately select the feature channels associated with that category. The activation map is then further processed and standardized to obtain an accurate target category activation map. This activation map not only reflects important regions in the image related to the target category but also provides high-quality support for subsequent lesion segmentation and localization.

[0176] In step S604 of some embodiments, a preset segmentation threshold is used to distinguish between high-response image patches (target category image patches) and low-response image patches (non-target category image patches) in the target class activation map. The segmentation threshold is typically in the range of [0,1]. In this embodiment, the preset segmentation threshold is 0.5. Masking processing is used to compare the response value of the target class activation map with the segmentation threshold to generate a binary mask, thereby marking which image patches belong to the target category. The target pathology mask map represents the position of the target category image patches in the target pathology whole-slice image in binary image form.

[0177] By introducing class activation maps, steps S601 to S604 above can automatically identify and segment lesion areas during disease diagnosis, providing accurate pathological image analysis results.

[0178] Please refer to Figure 8 , Figure 8This is a flowchart illustrating the training method and application of the pathological image processing model provided in the embodiments of this application.

[0179] Figure 8 Part (A) describes the training method WsiSSM for the pathological image processing model. Firstly, a full-slice image (e.g., with a resolution of 70000×80000) is obtained from the slice. Then, an initial image patch sequence is generated based on the segmentation module. The specific implementation process is similar to steps S101 to S102, and will not be elaborated here. Simultaneously, during this process, the coordinates of each image patch (e.g., with a resolution of 256×256) on the full-slice image are recorded. Then, the initial image patch sequence is input into the first backbone network, which outputs the first class activation map feature (CAM feature). The specific implementation process is similar to that of step S103. Specifically, the first backbone network includes an original feature extractor, an original slide-level encoder, and an original classifier. The image patch sequence is input into the original feature extractor, which outputs image patch features. The image patch features are input into the original slide-level encoder, which outputs hidden state features. The hidden state features are input into the original classifier, which outputs class activation map features. The specific implementation process is similar to that of steps S201 to S204, and will not be described in detail here.

[0180] On the other hand, the full-slice image is split into subsettings, for example... Figure 8 The entire slice image is divided into green, yellow, blue, and pink regions. Then, each of the sub-image sets is further segmented, and a subset patch sequence is obtained, consisting of image patches from each sub-image region, following the initial image patch sequence. The positional information of each image patch in each subset patch sequence is recorded. Next, the corresponding backbone is used to generate sub-image patch features based on the corresponding subset patch sequence and its positional information. Subsequently, a merging module is used to concatenate all the sub-image patch features into a second-type activation map feature. The specific implementation process of the above steps is similar to that of steps S104 to S106, and will not be repeated here.

[0181] Finally, the first type of activation map features are input into a global average pooling (GAP) module, which outputs the first predicted log value. The first loss (Classification loss) is then applied based on this first predicted log value. The second type of activation map features are calculated; the second type of activation map features are input into another global average pooling module (GAP) to output the second predicted log value, and the second loss (Classification loss) is performed based on the first predicted log value. ) Calculate; calculate reconstruction loss based on features of the first and second type activation maps. The model parameters are updated through backpropagation using three loss calculations to obtain the target pathological image processing model. The specific implementation process of the above steps is similar to that of steps S501 to S506, and will not be repeated here.

[0182] Figure 8 Part (B) describes a method for image processing using a pathological image processing model. First, a target whole-slice image is acquired. Then, the target whole-slice image is input into a target pathological image processing model based on the WsiSSM method, which outputs target class activation map features and multiple image patch locations. Subsequently, target class activation map scores are obtained based on the target class activation map features, and these scores are then subjected to ReLU activation processing and normalization processing to obtain target normalized data. Next, a target class activation map (CAM) is generated using a mapping module based on the multiple image patch locations and the target normalized data. The specific implementation process of the above steps is similar to that of steps S601 to S603 and steps S701 to S706, and will not be elaborated upon here.

[0183] Please refer to Figure 9 , Figure 9 This is a design flowchart provided in an embodiment of this application. The design process consists of three steps: ① data collection; ② modeling and analysis; ③ result evaluation.

[0184] In the data collection phase ①, the unlabeled public dataset was first subjected to coarse segmentation (i.e., foreground region segmentation as mentioned above), image patching (i.e., segmenting the foreground region into blocks), and self-supervised pre-training of the original feature extractor and original slice-level encoder. Then, the labeled public dataset was subjected to coarse segmentation, image patching, and image patch feature extraction using the pre-trained original feature extractor and original slice-level encoder. Based on the extracted features, data suitable for model analysis from the CAMELYON and PANDA datasets were selected as the final data.

[0185] In the modeling and analysis section ②, firstly, 80% of the final data is randomly divided into a training set, 10% into a validation set, and 10% into a test set. Then, the training set is used to cross-validate the parameters obtained using the hyperparameter grid search technique.

[0186] In step ③, the optimal hyperparameters obtained through cross-validation are used to construct the model, which is then trained using the training and validation sets to obtain the final model. The final model is then evaluated using the test set.

[0187] Specifically, when evaluating and comparing model results, the evaluation criteria for classification results are sensitivity, specificity, and AUC (the area under the receiver operating characteristic). The horizontal axis of the ROC curve represents the False Positive Rate (FPR), and the vertical axis represents the True Positive Rate (TPR). The points on the curve are determined by the TPR and FPR of the sample's output probability at different classification thresholds (when the output probability is greater than or equal to the set threshold, the sample is predicted as positive; otherwise, it is predicted as negative). AUC, the area under the ROC curve, is used to evaluate classifier performance. A positive and a negative sample are randomly selected and input into the prediction model. The predicted probabilities of the two samples are output, arranged from largest to smallest. The probability that the positive sample is ranked before the negative sample is the AUC value (that is, the probability that the output probability of the positive sample is greater than that of the negative sample). As the classification threshold gradually decreases, more and more samples are predicted as positive, but these positives also contain genuine negative samples, meaning that both TPR and FPR will increase simultaneously. When the threshold is at its maximum, the corresponding ROC curve coordinate point is (0,0), and when the threshold is at its minimum, the corresponding coordinate point is (1,1). The ideal target of this application is TPR = 1 and FPR = 0, corresponding to the ROC curve coordinate point (0,1). Therefore, this application selects the probability value represented by the point on the ROC curve closest to the coordinate point (0,1) as the classification threshold during model training. Based on the optimal classification threshold, the classification sensitivity and specificity can be calculated using the following formulas:

[0188] Sensitivity = TP / (TP + FN)

[0189] Specificity = TN / (TN + FP)

[0190] Where TP stands for True Positive (actually positive, predicted positive), FP for False Positive (actually negative, predicted positive), TN for True Negative (actually negative, predicted negative), and FN for False Negative (actually positive, predicted negative).

[0191] The evaluation criteria for segmentation results are accuracy, Dice coefficient, and Intersection over Union (IoU). Accuracy is a fundamental metric for classifier performance, representing the proportion of correctly predicted samples out of all samples. The Dice coefficient emphasizes the proportion of overlapping regions in both sets, measuring the quality of overlap between the segmentation result and the ground truth. IoU represents the proportion of the overlapping region to the total coverage area of ​​the two regions. Both Dice and IoU range from [0,1], with values ​​closer to 1 indicating a higher degree of overlap. The calculation formulas are as follows:

[0192]

[0193] Where A represents the predicted segmentation region and B represents the actual segmentation region.

[0194] Based on the above evaluation criteria, the method of this application was compared with other existing methods (WOTC, WTC, CLAM, TransMIL, DSMIL, DTFD-MIL, WiKG, and Prov-GigaPath) in terms of both classification and segmentation.

[0195] In classification, the method in this application performs best. On the CAMELYON dataset, the method in this application achieves an AUC of 97.44%. On the PANDA dataset, the method in this application also leads the way with an AUC of 99.64%, which demonstrates the powerful representational capabilities of the model architecture in this application.

[0196] In terms of segmentation, after the same heatmap score mapping step as the WsiSSM method in this application, all heatmaps from the alignment methods are segmented into binary masks using a threshold of 0.5. Compared with the existing MIL method, the method in this application demonstrates breakthrough segmentation capabilities without pixel-level annotation and achieves the best segmentation performance. Specifically, on the CAMELYON dataset, the method in this application achieves a Dice coefficient of 81.98% and an IoU of 69.57%; on the PANDA dataset, the method in this application achieves a Dice coefficient of 79.26% and an IoU of 65.74%.

[0197] Please see Figure 10 , Figure 10 This is a visual comparison chart of the image processing results obtained by the method provided in this application embodiment and the processing results of other existing methods. Figure 10 Part (A) shows the heatmap visualization results of our method and other existing methods (CLAM, TransMIL, DSMIL, DTFD-MIL) on the CAMELYON and PANDA datasets. Annotated cancerous tissue is shown in green, cancerous tissue in the heatmap is shown in red, and normal tissue in the heatmap is shown in blue. Figure 10 Section (B) shows the mask visualization results of our method and other existing methods on the CAMELYON and PANDA datasets. Ground-Truth refers to the true mask visualization of the image, with cancerous tissue shown in white and normal tissue shown in black. Figure 10 As can be seen, WsiSSM can accurately locate suspicious areas in WSI. Even very subtle areas can be delineated by this application, while other existing methods are not as good. This visual comparison chart intuitively demonstrates the superiority of the method in this application.

[0198] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the training method or pathological image processing method of the aforementioned pathological image processing model. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0199] Please see Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0200] The processor 1101 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0201] The memory 1102 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1102 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the training method or pathological image processing method of the pathological image processing model of the embodiments of this application.

[0202] Input / output interface 1103 is used to implement information input and output;

[0203] The communication interface 1104 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0204] Bus 1105 transmits information between various components of the device (e.g., processor 1101, memory 1102, input / output interface 1103, and communication interface 1104);

[0205] The processor 1101, memory 1102, input / output interface 1103 and communication interface 1104 are connected to each other within the device via bus 1105.

[0206] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method or pathological image processing method of the above-described pathological image processing model.

[0207] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0208] The training method, electronic device, and storage medium for the pathological image processing model provided in this application acquire a whole-slice image of a pathological sample; the whole-slice image is segmented by a segmentation module to obtain an initial image block sequence, which includes at least two initial image blocks; each initial image block is input into a first main network for feature recognition to obtain a first class activation map feature of the whole-slice image of the pathological sample. By segmenting a large-sized whole-slice image of the pathological sample into small-sized image blocks, only one image block needs to be loaded and processed each time, significantly reducing memory and computational requirements. It also avoids feature dilution or neglect of details caused by the large overall image size, thereby improving feature extraction accuracy. In addition, by introducing class activation maps that can generate category-specific heatmaps into pathological image processing, features used for classification diagnosis can also be directly used for automatic segmentation of lesion areas. Furthermore, the initial image block sequence is first split to obtain at least two first image block sub-sequences, where the first image blocks of any two first image block sub-sequences do not overlap. Each first image block in each first image block sub-sequence is input into a second main network for feature recognition, obtaining class activation map sub-features for each first image block sub-sequence. The class activation map sub-features of each first image block sub-sequence are then concatenated to obtain the second class activation map features of the sample pathological whole-slice image. By splitting the image into sub-sequences before feature recognition, the independence and resolution of local features are enhanced. Furthermore, the parameters of the original pathological image processing model are updated based on the first and second class activation map features to obtain the target pathological image processing model. The target pathological image processing model trained using the global features obtained by the first main network and the local features obtained by the second main network considers both global and local aspects, resulting in the generation of more accurate class activation map features. This enhances the accuracy of classification diagnosis and automatic lesion region segmentation of pathological whole-slice images when using class activation map features for classification and segmentation.

[0209] The pathological image processing method, electronic device, and storage medium provided in this application acquire a target pathological whole-slice image of a target object; perform image processing on the target pathological whole-slice image using a target pathological image processing model obtained through the training method proposed in the first aspect, to obtain target class activation map features and multiple target image block positions; wherein, the target image block is the position of the target image block obtained by segmenting the target pathological whole-slice image through a segmentation module; perform image reconstruction processing based on the target class activation map features and the target image block positions to obtain a target class activation map; and perform masking processing on the target class activation map according to a preset segmentation threshold to obtain a target pathological mask map. This method obtains more accurate class activation map features through the target pathological image processing model, enabling both classification diagnosis and automatic segmentation of lesion areas based on class activation map features, and enhancing the accuracy of both classification diagnosis and automatic segmentation of lesion areas in the pathological whole-slice image, thereby improving the accuracy in both classification diagnosis and automatic segmentation of lesion areas.

[0210] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0211] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0212] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0213] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0214] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0215] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0216] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, and the indirect coupling or communication connection between the system or units may be electrical, mechanical, or other forms.

[0217] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0218] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0219] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0220] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A training method for a pathological image processing model, characterized in that, The original pathological image processing model includes a segmentation module, a first main network, and a second main network. The first main network includes a raw feature extractor, a raw slice-level encoder, and a raw classifier. The second main network is composed of at least two of the first main networks in parallel. The method includes: Obtain whole-section images of the sample for pathological examination; The segmentation module segments the whole pathological slide image of the sample to obtain an initial image block sequence, which includes at least two initial image blocks. Each of the initial image blocks is input into the first main network for feature recognition to obtain the first type of activation map features of the whole pathological slice image of the sample; The initial image block sequence is first split to obtain at least two first image block subsequences, and the first image blocks of any two first image block subsequences do not overlap. Each first image block in each first image block subsequence is input into the second main network for feature recognition to obtain class activation map sub-features for each first image block subsequence; The class activation map features of each of the first image block sub-sequences are concatenated to obtain the second class activation map features of the whole pathological slice image of the sample. The original pathological image processing model is updated with parameters based on the first type of activation map features and the second type of activation map features to obtain the target pathological image processing model.

2. The method according to claim 1, characterized in that, Before updating the parameters of the original pathological image processing model based on the first type of activation map features and the second type of activation map features to obtain the target pathological image processing model, the method further includes updating the second type of activation map features, specifically including: The initial image block sequence is split a second time to obtain at least two second image block subsequences; wherein at least one first image block subsequence differs from one of the second image block sequences; The class activation map sub-features of each second image block sub-sequence are concatenated to obtain auxiliary class activation map features; The second type of activation graph features are updated based on the auxiliary type activation graph features.

3. The method according to claim 1, characterized in that, The step of inputting each of the initial image blocks into the first main network for feature recognition to obtain the first type of activation map features of the whole pathological slice image of the sample includes: For each initial image patch, the original feature extractor is input sequentially to obtain a low-dimensional feature representation sequence; The positions of each initial image block in the whole pathological slice image of the sample are obtained to obtain the initial image block position sequence; The low-dimensional feature representation sequence and the initial image block position sequence are input into the original slice-level encoder for encoding to obtain hidden state features; The hidden state features are input into the original classifier for classification processing to obtain the first type of activation map features of the whole pathological slice image of the sample.

4. The method according to claim 1, characterized in that, The step of updating the parameters of the original pathological image processing model based on the first type of activation map features and the second type of activation map features to obtain the target pathological image processing model includes: Feature mapping is performed based on the features of the first type of activation map to obtain the first mapping data; Based on the features of the second type of activation map, feature mapping is performed to obtain the second mapping data; The first loss is calculated based on the first mapping data and the sample labels of the pre-acquired whole-slice pathological images to obtain the first loss data; A second loss is calculated based on the second mapping data and the sample labels to obtain the second loss data; The third loss is calculated based on the first type of activation map features and the second type of activation map features to obtain the third loss data. The original pathological image processing model is updated with parameters based on the first loss data, the second loss data, and the third loss data to obtain the target pathological image processing model.

5. The method according to claim 3, characterized in that, The original slice-level encoder includes a first encoding module and a second encoding module. The step of inputting the low-dimensional feature representation sequence and the initial image block position sequence into the original slice-level encoder for encoding processing to obtain hidden state features includes: The first encoding module projects the low-dimensional feature representation sequence onto an encoding space of a preset dimension to obtain a first matrix; Perform two-dimensional sine and cosine position embedding based on the initial image block position sequence to obtain a second matrix with the same dimension as the first matrix; Add the first matrix and the second matrix element by element to obtain the third matrix; The third matrix is ​​encoded and transformed by the second encoding module to obtain the hidden state features.

6. The method according to claim 3, characterized in that, The original slice-level encoder is obtained in the following way: The training image block sequence is obtained by segmenting unlabeled slice-level sample pathological images through the segmentation module. Obtain the training image patch feature sequence generated by the original feature extractor based on the training image patch sequence; The training image block sequence is subjected to random masking to obtain an unmasked image block sequence; The sequence of unmasked image blocks is input into the encoder to obtain the feature sequence of unmasked image blocks; The unmasked image patch feature sequence is filled using a learnable mask vector to obtain a filled image patch feature sequence that is consistent with the order of the training image patch sequence. The feature sequence of the filled image patch is input into the decoder to obtain the mask reconstruction feature sequence; The similarity between the training image patch feature sequence and the mask reconstruction feature sequence is calculated, and the encoder parameters are updated based on the similarity to obtain the original slice-level encoder.

7. A pathological image processing method, the method comprising: Obtain the target pathological whole-section image of the target object; The target pathological whole-slice image is processed by the target pathological image processing model obtained by the training method as described in any one of claims 1-6 to obtain target class activation map features and multiple target image block positions; wherein, the target image block is the position of the target image block obtained by segmenting the target pathological whole-slice image by the segmentation module; Image reconstruction is performed based on the target class activation map features and the target image block positions to obtain the target class activation map; The target class activation map is masked according to a preset segmentation threshold to obtain a target pathological mask map.

8. The method according to claim 7, characterized in that, The target class activation map features include candidate class activation map scores for at least two candidate categories. The image reconstruction process based on the target class activation map features and the target image patch location to obtain the target class activation map includes: The target category of the target pathological whole slice image is determined based on the candidate class activation map score of each candidate category; The target channel of the channel classifier is determined based on the target category; The target class activation map features are classified by channel weights of the target channels to obtain the target class activation map score of the target category; The target class activation graph score is activated to obtain target class activation data; The target class activation data is standardized to obtain target class standardized data; The target class normalized data and the target image block positions are reconstructed to obtain the target class activation map.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method of any one of claims 1-6 or the method of any one of claims 7-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-6 or the method of any one of claims 7-8.

Images