Method and apparatus for multiscale analysis of whole slide images

Abstract

The application discloses a multiscale analysis method and device for whole slide images, and relates to the technical field of whole slide image analysis. The method comprises the following steps: performing preprocessing and feature extraction operations on a whole slide image dataset to generate instance feature sets at different resolution levels, and establishing a spatial mapping relationship between the different resolution levels; based on the spatial mapping relationship, iteratively performing cross-scale feature interaction on the instance feature sets at different resolution levels to update low-resolution features in each iteration process, acquire and splice a generated high-resolution feature sequence, and use a class attention to aggregate features to generate a whole slide level pathological feature representation; and based on the whole slide level pathological feature representation, performing a preset feature prediction operation to generate a medical classification or prognosis prediction result corresponding to each whole slide image data, thereby greatly improving the completeness and discriminability of the whole slide level pathological feature representation.

Description

Technical Field

[0001] This application relates to the field of whole-slice image analysis technology, and in particular to a multi-scale analysis method and apparatus for whole-slice images. Background Technology

[0002] With the development of digital pathology, artificial intelligence has become a transformative tool for analyzing whole-slide images (WSI). However, due to the ultra-high resolution of gigapixels in WSI and the extreme scarcity of pixel-level annotations, weakly supervised multiple instance learning (MIL) has become the dominant paradigm in this field.

[0003] In actual clinical pathological diagnosis, pathologists often need to combine multiple fields of view. They need to observe the overall tissue structure and spatial distribution of lesions under low magnification, and switch to high magnification to observe microscopic details such as cell nuclei. Traditional single-scale multi-instance learning methods can only extract features at a fixed magnification. If only low resolution is relied upon, crucial cellular-level diagnostic clues will be lost; if only high resolution is relied upon, although its accuracy is high, the global field of view will be lost, and the computational efficiency will be low.

[0004] To capture rich diagnostic information embedded at different resolutions, various multi-scale MIL methods have been developed. To reduce computational costs, existing coarse-to-fine sampling methods typically employ a cascaded paradigm, first using a low-resolution discriminator to scan the global context and locate key suspicious regions; then guiding the model to selectively extract and analyze fine-grained features of the corresponding high-resolution regions.

[0005] However, existing coarse-to-fine multi-scale, multi-instance learning methods still face serious limitations in practical applications. Most existing techniques employ a unidirectional information flow architecture, meaning that region selection relies entirely on the initial coarse-grained stage, lacking a feedback mechanism from subsequent high-resolution, fine-grained analysis. This design leads to a situation where, once a low-resolution representation overlooks a critical diagnostic region due to limited field of view or insufficient feature visibility, that region is permanently excluded from subsequent analysis. The model cannot utilize the microscopic evidence provided by high resolution to correct the initial region selection bias, resulting in the irreversible loss of crucial diagnostic information.

[0006] Furthermore, the aforementioned irreversible information loss problem is amplified by existing static sampling strategies and hard attention mechanisms. Some existing techniques rely on static Top-K sampling strategies when selecting high-resolution image patches, making the model prone to getting stuck in "local optima" during training and inference, repeatedly focusing attention on obvious major lesions while ignoring secondary lesions with significant clinical importance. Meanwhile, traditional hard attention mechanisms have an overly narrow field of view; features of regions not initially selected by the sampling are completely discarded by the model. This complete discarding of information from unsampled regions not only severs the connection between lesions and the global context but also leads to the loss of potential diagnostic clues hidden in non-salient regions, severely limiting the completeness of the final slice-level feature representation and the accuracy of clinical diagnostic predictions.

[0007] In summary, existing multi-scale whole-slice image analysis techniques are prone to irreversible loss of key diagnostic information and tend to overlook secondary lesions, which urgently need to be addressed. Summary of the Invention

[0008] This application provides a multi-scale analysis method and apparatus for whole-slice images, which overcomes the limitations of traditional unidirectional processing in multi-scale analysis of whole-slice images. It enables iterative bidirectional cross-scale interaction of features at different resolutions, fully explores global background and local fine-grained diagnostic information, effectively avoids the loss of key diagnostic information and the neglect of minor clinical lesions, significantly improves the completeness and discriminativeness of whole-slice-level pathological feature representation, and significantly improves the accuracy and clinical reference value of medical classification, prognosis prediction results.

[0009] The first aspect of this application provides a multi-scale analysis method for whole-slice images, comprising the following steps: preprocessing and feature extraction operations on a preset whole-slice image dataset to generate instance feature sets at different resolution levels, and establishing a spatial mapping relationship between the instance feature sets at different resolution levels; based on the spatial mapping relationship, iteratively performing cross-scale feature interaction on the instance feature sets at different resolution levels to obtain and splice the target feature sequences that meet the preset resolution requirements generated in each iteration process to obtain the corresponding spliced ​​feature sequences, and performing a preset feature aggregation operation on the spliced ​​feature sequences to generate a pathological feature representation at the whole-slice level; Based on the pathological feature representation at the whole slice level, a preset feature prediction operation is performed to generate medical classification or prognostic prediction results corresponding to each whole slice image data.

[0010] Optionally, in one embodiment of this application, the preprocessing and feature extraction operations on the preset whole-slice image dataset to generate instance feature sets at different resolution levels, and the establishment of spatial mapping relationships between the instance feature sets at different resolution levels, include: performing cropping operations on the whole-slice image dataset at different resolution levels to obtain corresponding image patches, and encoding the image patches using a pre-trained pathology basic model to obtain the instance feature sets at different resolution levels; and determining the resolution regions corresponding to the instance feature sets at different resolution levels based on a preset mapping function to construct the spatial mapping relationship.

[0011] Optionally, in one embodiment of this application, the step of iteratively performing cross-scale feature interaction on the instance feature sets at different resolution levels based on the spatial mapping relationship to obtain and splice the target feature sequences that meet the preset resolution requirements generated in each iteration process, thereby obtaining the corresponding spliced ​​feature sequences, and performing a preset feature aggregation operation on the spliced ​​feature sequences to generate a full-slice-level pathological feature representation, includes: screening at least one key region of the low-resolution instance feature set in the instance feature sets at different resolution levels that meets the preset criticality requirements, and mapping the regional features of each key region in the at least one key region to a corresponding latent space vector, so as to calculate the sampling probability distribution and importance score of each key region based on the latent space vector; sampling the high-resolution instance feature set in the instance feature sets at different resolution levels according to the sampling probability distribution to obtain the corresponding The system collects sampled and unsampled features, and aggregates unsampled region features through a preset retrieval mechanism to generate a corresponding global retrieval token. The unsampled features and the global retrieval token are input into a preset attention layer to determine the global context relationship between the sampled and unsampled features based on the spatial mapping relationship. This global context relationship is then modeled to output the corresponding focused features. Based on a preset cross-attention mechanism, the focused features are iteratively interacted with the low-resolution instance feature set to update the low-resolution instance feature set and a preset historical mask matrix, obtaining a target feature sequence that meets the preset resolution requirements generated in each iteration. The target feature sequence is then concatenated to obtain the concatenated feature sequence, and feature aggregation is performed on the concatenated feature sequence based on a preset class-attention mechanism to generate the full-slice-level pathological feature representation.

[0012] Optionally, in one embodiment of this application, the step of performing a preset feature prediction operation based on the pathological feature representation at the whole slice level to generate a medical classification or prognostic prediction result corresponding to each whole slice image data includes: inputting the pathological feature representation at the whole slice level into a preset classifier to perform feature prediction using the total loss function in the classifier to generate a medical classification or prognostic prediction result model corresponding to each whole slice image data.

[0013] Optionally, in one embodiment of this application, the mathematical expression of the total loss function is:

[0014] in, Represents the total loss function; For hyperparameters; D Indicates the number of iteration rounds; This represents the entropy regularization term.

[0015] A second aspect of this application provides a multi-scale analysis apparatus for whole-slice images, comprising: a feature extraction module for preprocessing and extracting features from a preset whole-slice image dataset to generate instance feature sets at different resolution levels and establishing a spatial mapping relationship between the instance feature sets at different resolution levels; a cross-scale feature interaction module for iteratively performing cross-scale feature interaction on the instance feature sets at different resolution levels based on the spatial mapping relationship to obtain and splice target feature sequences that meet preset resolution requirements generated in each iteration to obtain corresponding spliced ​​feature sequences, and performing a preset feature aggregation operation on the spliced ​​feature sequences to generate a pathological feature representation at the whole-slice level; and a feature prediction module for performing a preset feature prediction operation based on the pathological feature representation at the whole-slice level to generate a medical classification or prognostic prediction result corresponding to each whole-slice image data.

[0016] Optionally, in one embodiment of this application, the feature extraction module includes: a cropping unit, used to perform cropping operations at different resolution levels on the whole slice image dataset to obtain corresponding image blocks, and to encode the image blocks using a pre-trained pathology basic model to obtain the instance feature sets at different resolution levels; and a construction unit, used to determine the resolution regions corresponding to the instance feature sets at different resolution levels based on a preset mapping function to construct the spatial mapping relationship.

[0017] Optionally, in one embodiment of this application, the cross-scale feature interaction module includes: a filtering unit, configured to filter at least one key region of the low-resolution instance feature set in the instance feature set at different resolution levels that meets a preset key requirement, and map the region features of each key region in the at least one key region to a corresponding latent space vector, so as to calculate the sampling probability distribution and importance score of each key region based on the latent space vector; a sampling unit, configured to sample the high-resolution instance feature set in the instance feature set at different resolution levels according to the sampling probability distribution, so as to obtain the corresponding sampled features and unsampled features, and aggregate the unsampled region features through a preset recycling mechanism to generate a corresponding global recycling token; and a modeling unit, configured to convert the unsampled region features into a modeling model. The sampled features and the global reclamation token are input into a preset attention layer to determine the global context relationship corresponding to the sampled features and the unsampled features based on the spatial mapping relationship, and to model the global context relationship to output the corresponding focused features; the interaction unit is used to iteratively interact with the focused features and the low-resolution level instance feature set based on a preset cross-attention mechanism to update the low-resolution level instance feature set and the preset history mask matrix to obtain the target feature sequence that meets the preset resolution requirements generated in each iteration process; the splicing unit is used to splice the target feature sequence to obtain the spliced ​​feature sequence, and to perform feature aggregation operation on the spliced ​​feature sequence based on a preset class attention mechanism to generate the full-slice level pathological feature representation.

[0018] Optionally, in one embodiment of this application, the feature prediction module includes: a generation unit, used to input the pathological feature representation at the whole slice level into a preset classifier, so as to use the total loss function in the classifier to perform feature prediction, so as to generate a medical classification or prognostic prediction result model corresponding to each whole slice image data.

[0019] Optionally, in one embodiment of this application, the mathematical expression of the total loss function is:

[0020] in, Represents the total loss function; For hyperparameters; D Indicates the number of iteration rounds; This represents the entropy regularization term.

[0021] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-scale analysis method for whole-slice images as described in the above embodiments.

[0022] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-scale analysis method for full-slice images as described above.

[0023] A fifth aspect of this application provides a computer program product, including a computer program that is executed to implement the above-described multi-scale analysis method for whole-slice images.

[0024] Therefore, the embodiments of this application have the following beneficial effects: The embodiments of this application preprocess and extract features from a pre-defined whole-slice image dataset to generate instance feature sets at different resolution levels and establish spatial mapping relationships between these sets. Based on this spatial mapping, cross-scale feature interactions are iteratively performed on the instance feature sets at different resolution levels to obtain and stitch together target feature sequences that meet the pre-defined resolution requirements generated in each iteration, resulting in corresponding stitched feature sequences. Class attention is then used to perform feature aggregation on the stitched feature sequences to generate a whole-slice-level pathological feature representation. Based on this whole-slice-level pathological feature representation, a pre-defined feature prediction operation is performed to generate medical classification or prognostic prediction results for each whole-slice image data. Therefore, this application overcomes the unidirectional processing limitations of traditional multi-scale analysis of whole-slice images, achieving iterative bidirectional cross-scale interaction of features at different resolutions. This fully mines global background and local fine-grained diagnostic information, effectively avoiding the loss of key diagnostic information and the neglect of minor clinical lesions, significantly improving the completeness and discriminativeness of whole-slice-level pathological feature representation, and significantly enhancing the accuracy and clinical reference value of medical classification and prognostic prediction results.

[0025] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0026] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart of a multi-scale analysis method for whole-slice images provided as an embodiment of this application; Figure 2This is a schematic diagram of the logical architecture of a multi-scale analysis method for whole-slice images provided according to an embodiment of this application; Figure 3 A schematic diagram of a region of interest for model selection is provided as an embodiment of this application; Figure 4 A visualization heatmap corresponding to a region of interest is provided as an embodiment of this application; Figure 5 This is an example diagram of a multi-scale analysis apparatus for a whole-slice image according to an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0027] Among them, 10 is a multi-scale analysis device for whole-slice images; 100 is a feature extraction module; 200 is a scale feature interaction module; 300 is a feature prediction module; 601 is a memory; 602 is a processor; and 603 is a communication interface. Detailed Implementation

[0028] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0029] The following describes a multi-scale analysis method and apparatus for whole-slice images according to embodiments of this application, with reference to the accompanying drawings. Addressing the problems mentioned in the background section, this application provides a multi-scale analysis method for whole-slice images. In this method, a preprocessing and feature extraction operation is performed on a preset whole-slice image dataset to generate instance feature sets at different resolution levels, and a spatial mapping relationship is established between these sets. Based on the spatial mapping relationship, cross-scale feature interaction is iteratively performed on the instance feature sets at different resolution levels to obtain and stitch together the target feature sequences generated in each iteration that meet the preset resolution requirements, resulting in corresponding stitched feature sequences. A preset feature aggregation operation is then performed on the stitched feature sequences to generate a whole-slice-level pathological feature representation. Based on the whole-slice-level pathological feature representation, a preset feature prediction operation is performed to generate a medical classification or prognostic prediction result corresponding to each whole-slice image data. This application solves the problems of irreversible information loss and the easy neglect of minor lesions in the prior art by establishing a cross-scale bidirectional interaction mechanism, a retrieval mechanism, and a memory mask strategy, thereby improving the accuracy of whole-slice image feature extraction and clinical task prediction. This solves the problems of irreversible loss of key diagnostic information and easy neglect of secondary lesions in existing whole-slice image analysis techniques.

[0030] Specifically, Figure 1 This is a flowchart illustrating a multi-scale analysis method for a whole-slice image provided in an embodiment of this application.

[0031] like Figure 1 As shown, the multi-scale analysis method for this whole-slice image includes the following steps: In step S101, preprocessing and feature extraction operations are performed on the preset full-slice image dataset to generate instance feature sets at different resolution levels, and spatial mapping relationships are established between instance feature sets at different resolution levels.

[0032] This application embodiment first collects a public dataset of full-slice images (i.e., a full-slice image dataset), preprocesses it, and extracts features from the full-slice images at two resolution levels (i.e., low resolution level and high resolution level) to obtain a low-resolution instance feature set and a high-resolution instance feature set, and establishes a spatial mapping relationship between high-resolution instance features and low-resolution regions.

[0033] Optionally, in one embodiment of this application, preprocessing and feature extraction operations are performed on a preset whole-slice image dataset to generate instance feature sets at different resolution levels, and a spatial mapping relationship is established between instance feature sets at different resolution levels. This includes: performing cropping operations at different resolution levels on the whole-slice image dataset to obtain corresponding image patches, and encoding the image patches using a pre-trained pathology basic model to obtain instance feature sets at different resolution levels; and determining the resolution regions corresponding to instance feature sets at different resolution levels based on a preset mapping function to construct a spatial mapping relationship.

[0034] It should be noted that the embodiments of this application first collect existing publicly available datasets of whole-slice images, and then crop corresponding image patches from the obtained WSI at two resolution levels. These patches are then aligned and encoded using a pre-trained pathology-based model to obtain a set of instance features at low magnification (i.e., low resolution). and the set of instance features at high magnification (i.e., high resolution). .

[0035] Secondly, in order to achieve cross-scale information interaction, embodiments of this application define a mapping function. To indicate high-resolution image patches Spatially, it belongs to a low-resolution region. .

[0036] Subsequently, in this embodiment of the application, the processed dataset (i.e., the set of instance features at different resolution levels) can be divided into training set and test set according to different cancer types, thereby ensuring that cases in the training set will not appear in the test set.

[0037] In step S102, based on the spatial mapping relationship, cross-scale feature interaction is iteratively performed on the instance feature sets at different resolution levels to obtain and splice the target feature sequences that meet the preset resolution requirements generated in each iteration process, thereby obtaining the corresponding spliced ​​feature sequences. The spliced ​​feature sequences are then subjected to preset feature aggregation operations to generate a full-slice level pathological feature representation.

[0038] Furthermore, embodiments of this application can input dual-scale features (i.e., sets of instance features at different resolution levels) into multiple cascaded and stacked overview-focus-reflection modules for iterative cross-scale feature interaction. For example... Figure 2 As shown, in each iteration, the following steps are performed sequentially: Step 1: Generate the corresponding sampling probability distribution and importance score based on the low-resolution instance features, and sample the high-resolution instance features according to the sampling probability distribution; at the same time, aggregate the unselected high-resolution region features through the recycling mechanism to generate a global recycling token (i.e., global recycling feature).

[0039] Step 2: Input the selected high-resolution instance features and global reclamation token into an attention layer with rotational position encoding to model the global context relationship between the selected and unselected regions, and output focused features.

[0040] Step 3: Feed the focused features output by the focusing module back to the low-resolution feature space through the cross-attention mechanism to recalibrate the low-resolution global representation; at the same time, update the historical mask matrix, explicitly suppress the regions that have been selected in the current round, and guide the model to shift its attention to the secondary lesion regions in the next round.

[0041] Furthermore, after multiple iterations, the embodiments of this application can splice together the high-resolution (i.e., preset resolution requirement) feature sequences (i.e. target feature sequences) generated in all rounds to obtain the corresponding spliced ​​feature sequences. The spliced ​​feature sequences are then aggregated through a class attention layer to obtain the final slice-level representation (i.e., full slice-level pathological feature representation).

[0042] Optionally, in one embodiment of this application, based on spatial mapping relationships, cross-scale feature interactions are iteratively performed on instance feature sets at different resolution levels to obtain and splice target feature sequences that meet preset resolution requirements generated in each iteration process, resulting in corresponding spliced ​​feature sequences. Preset feature aggregation operations are then performed on the spliced ​​feature sequences to generate a full-slice-level pathological feature representation. This includes: selecting at least one key region from the low-resolution instance feature set in the instance feature sets at different resolution levels that meets preset criticality requirements, and mapping the regional features of each key region in the at least one key region to a corresponding latent space vector, so as to calculate the sampling probability distribution and importance score of each key region based on the latent space vector; sampling the high-resolution instance feature set in the instance feature sets at different resolution levels according to the sampling probability distribution to obtain the target feature sequence that meets preset resolution requirements. The system collects corresponding sampled and unsampled features, and aggregates unsampled region features through a preset retrieval mechanism to generate a corresponding global retrieval token. The unsampled features and the global retrieval token are input into a preset attention layer to determine the global context relationship between the sampled and unsampled features based on spatial mapping, and the global context relationship is modeled to output the corresponding focused features. Based on a preset cross-attention mechanism, the focused features and a low-resolution instance feature set are iteratively interacted to update the low-resolution instance feature set and a preset historical mask matrix, obtaining a target feature sequence that meets the preset resolution requirements generated in each iteration. The target feature sequence is concatenated to obtain a concatenated feature sequence, and based on a preset class-attention mechanism, feature aggregation is performed on the concatenated feature sequence to generate the full-slice-level pathological feature representation.

[0043] As one possible approach, after obtaining the training set and test set mentioned above, the embodiments of this application can select appropriate loss functions and learning rates according to different tasks to train a model consisting of multiple cascaded and stacked overview-focus-reflection modules (GFR Blocks) and a classification module (wherein, except for the last GFR Block which does not contain a reflection module, each GFR Block consists of an overview module, a focus module and a reflection module), as described in the following process; 1. Overview Module: This module aims to filter key regions in a full-slice image at low resolution and calculate the importance score for each region. To achieve this, the module uses a gating network to encode the region features at low resolution. Mapped to latent space vectors According to equation (1), the sampling probability distribution of each region is obtained. And the importance score of each region : (1) in, Represents the Sigmoid function; This represents the attention score, which is the probability that the j-th low-resolution region is sampled. This represents the latent space vector of the k-th low-resolution region; Indicates the number of low-resolution regions; Indicates the number of high-resolution regions; Let represent the d-dimensional real space.

[0044] To enhance the model's exploratory capabilities, during training, embodiments of this application may base their training on a sampling probability distribution. Perform non-replacement polynomial sampling and determine the corresponding index set based on the selected low-resolution image patch. During the testing process, such as Figure 2 As shown, to ensure stability, the embodiments of this application can directly take Top-k%. To prevent attention collapse into a uniform distribution, the embodiments of this application can introduce an entropy regularization term according to equation (2): (2) in, Indicates the first i The probability of a low-resolution region being sampled.

[0045] For the selected high-resolution features (i.e., the sampled features) ( ), explicitly multiply by the importance coefficient of the corresponding low-resolution region, as shown in the following formula: (3) To mitigate information loss in unsampled areas, embodiments of this application can extract the index set of unselected areas. The high-resolution features (i.e., unsampled features) are used to calculate the global recovery features according to equation (4). As shown in the following formula: (4) in, Indicates the first A set of image patches within a region; Indicates the first j A set of image patches within a region; Indicates the first j The importance score of each region; Indicates the extraction of the first [item] at high resolution. One feature; This represents the importance score of the low-resolution region corresponding to the high-resolution region i. This represents the mapping from high-resolution region i to its corresponding low-resolution region, i.e., indicating a high-resolution image patch. It belongs to the low-resolution region in space.

[0046] 2. Focusing Module: Obtains high-resolution features corresponding to important regions. and global recycling features Subsequently, these high-resolution features are used to model the whole pathological slides. This module uses a single-layer transformer module with rotational position encoding for modeling. The self-attention mechanism and positional information can effectively help model the correlation between regional features and output the corresponding focused features. .

[0047] 3. Reflection Module: In order to simulate the behavior of pathologists repeatedly checking, this application's embodiments introduce a reflection mechanism.

[0048] Specifically, the output of the current round focusing module Through cross-attention mechanism and low-resolution features Interacting with the system allows high-resolution, fine-grained features to be injected into the low-resolution feature space to update the feature space. (i.e., the set of instance features at a low resolution level), as shown in equation (5), where, Representation layer normalization.

[0049] (5) in, Represents a class attention function; This represents the low-resolution features after the first iteration.

[0050] In this process, embodiments of this application enable the model to dynamically adjust its attention to global regions based on confirmed pathological details. Simultaneously, embodiments of this application can apply a memory constraint strategy, i.e., maintaining a historical mask matrix and explicitly masking regions already selected in the current round in the next iteration, forcing the model to shift its attention to find secondary but clinically significant regions in subsequent iterations, thereby constructing a more comprehensive slice representation and updating the... and The next GFR Block is passed in to start a new iteration.

[0051] 4. Feature Aggregation Module: After multiple iterations, the model accumulates multiple sets of high-resolution feature sequences (i.e., target feature sequences). Therefore, the embodiments of this application can concatenate the target feature sequences from all iterations to obtain the corresponding concatenated feature sequences, and use an attention-based aggregation layer to fuse the concatenated feature sequences, thereby obtaining the final full-slice-level pathological feature representation. : (6) (7) in, This represents the concatenation of features from round D; Indicates first i The focusing features output by the focusing module after each iteration.

[0052] Therefore, the embodiments of this application improve the reliability of whole-slice image analysis by inputting dual-scale features into multiple cascaded and stacked overview-focus-reflection modules for iterative cross-scale feature interaction, and splicing and aggregating the high-resolution feature sequences generated in all rounds.

[0053] In step S103, based on the pathological feature representation at the whole slice level, a preset feature prediction operation is performed to generate the medical classification or prognostic prediction result corresponding to each whole slice image data.

[0054] Subsequently, embodiments of this application can perform feature prediction based on slice-level representation, thereby outputting medical classification or prognostic prediction results for the target object.

[0055] Optionally, in one embodiment of this application, a preset feature prediction operation is performed based on the pathological feature representation at the whole slice level to generate a medical classification or prognostic prediction result corresponding to each whole slice image data. This includes: inputting the pathological feature representation at the whole slice level into a preset classifier to perform feature prediction using the total loss function in the classifier to generate a medical classification or prognostic prediction result model corresponding to each whole slice image data.

[0056] As one possible approach, embodiments of this application can represent pathological features at the whole-slice level. The data is fed into a classifier (i.e., the classification module) to make predictions using the model's total loss function, thereby generating medical classification or prognostic prediction results for each full-slice image data.

[0057] Optionally, in one embodiment of this application, the mathematical expression for the total loss function is:

[0058] in, Represents the total loss function; For hyperparameters; D Indicates the number of iteration rounds; This represents the entropy regularization term.

[0059] It should be noted that the total loss function of the model in this embodiment is composed of a specific task loss. For example, it consists of cross-entropy loss for classification or negative log-likelihood loss for survival analysis, plus a multi-round cumulative entropy regularization term. The mathematical expression for this total loss function is: (8) in, Represents the total loss function; This represents a hyperparameter, with a range of [0.1, 0.2]. D This represents the number of iterations for low-to-high rate sampling, which is also the number of GFR-Blocks and is a pre-set hyperparameter. This represents the entropy regularization term.

[0060] Therefore, the embodiments of this application effectively ensure the accuracy of medical classification or prognosis prediction results by constructing the above-mentioned total loss function for subsequent feature prediction operations.

[0061] Figure 3 A schematic diagram of the region of interest selected for the model; Figure 4 A visual heatmap of the region of interest. For example... Figure 3 As shown, through the closed-loop interaction mechanism provided in the embodiments of this application, the model can effectively locate key regions of interest (such as...). Figure 3 (as shown in the black box in the image), corresponding to the visualized heatmap (e.g.) Figure 4 As shown, the primary lesion is accurately covered. This embodiment achieves optimal performance by focusing on only 30% of the slice area, significantly balancing computational efficiency and prediction accuracy.

[0062] According to the multi-scale analysis method for whole-slice images proposed in this application, a preprocessing and feature extraction operation is performed on a preset whole-slice image dataset to generate instance feature sets at different resolution levels, and a spatial mapping relationship is established between instance feature sets at different resolution levels. Based on the spatial mapping relationship, cross-scale feature interaction is iteratively performed on instance feature sets at different resolution levels to obtain and splice the target feature sequences that meet the preset resolution requirements generated in each iteration process, resulting in the corresponding spliced ​​feature sequences. A preset feature aggregation operation is then performed on the spliced ​​feature sequences to generate a pathological feature representation at the whole-slice level. Based on the pathological feature representation at the whole-slice level, a preset feature prediction operation is performed to generate the medical classification or prognostic prediction result corresponding to each whole-slice image data. This application solves the problems of irreversible information loss and the easy neglect of minor lesions in the prior art by establishing a cross-scale bidirectional interaction mechanism, a retrieval mechanism, and a memory mask strategy, thereby improving the accuracy of whole-slice image feature extraction and clinical task prediction.

[0063] Secondly, a multi-scale analysis apparatus for whole-slice images according to an embodiment of this application is described with reference to the accompanying drawings.

[0064] Figure 5 This is a block diagram of a multi-scale analysis device for full-slice images according to an embodiment of this application.

[0065] like Figure 5 As shown, the multi-scale analysis device 10 for the whole slice image includes: a feature extraction module 100, a cross-scale feature interaction module 200, and a feature prediction module 300.

[0066] The feature extraction module 100 is used to preprocess and extract features from a preset full-slice image dataset to generate instance feature sets at different resolution levels and establish spatial mapping relationships between instance feature sets at different resolution levels.

[0067] The cross-scale feature interaction module 200 is used to iteratively perform cross-scale feature interaction on instance feature sets at different resolution levels based on spatial mapping relationships, so as to obtain and splice the target feature sequence that meets the preset resolution requirements generated in each iteration process, obtain the corresponding spliced ​​feature sequence, and perform preset feature aggregation operation on the spliced ​​feature sequence to generate a full-slice level pathological feature representation.

[0068] The feature prediction module 300 is used to perform preset feature prediction operations based on the pathological feature representation at the whole slice level, so as to generate medical classification or prognostic prediction results corresponding to each whole slice image data.

[0069] Optionally, in one embodiment of this application, the feature extraction module 100 includes a cropping unit and a construction unit.

[0070] The cropping unit is used to perform cropping operations at different resolution levels on the whole slice image dataset to obtain corresponding image patches, and to encode the image patches through a pre-trained pathology-based model to obtain instance feature sets at different resolution levels.

[0071] The construction unit is used to determine the resolution region corresponding to the instance feature set at different resolution levels based on a preset mapping function, so as to construct a spatial mapping relationship.

[0072] Optionally, in one embodiment of this application, the cross-scale feature interaction module 200 includes: a filtering unit, a sampling unit, a modeling unit, an interaction unit, and a splicing unit.

[0073] The filtering unit is used to filter at least one key region from the low-resolution instance feature set in the instance feature set of different resolution levels that meets the preset key requirements, and to map the regional features of each key region in the at least one key region to the corresponding latent space vector, so as to calculate the sampling probability distribution and importance score of each key region based on the latent space vector.

[0074] The sampling unit is used to sample the high-resolution instance feature set in the instance feature set at different resolution levels according to the sampling probability distribution, so as to obtain the corresponding sampled features and unsampled features, and to aggregate the unsampled regional features through a preset recycling mechanism to generate the corresponding global recycling token.

[0075] The modeling unit is used to input unsampled features and global reclamation tokens into a preset attention layer to determine the global context relationship between sampled features and unsampled features based on spatial mapping, and to model the global context relationship to output the corresponding focused features.

[0076] The interaction unit is used to iteratively interact with the focused feature and the low-resolution instance feature set based on a preset cross-attention mechanism, so as to update the low-resolution instance feature set and the preset historical mask matrix, so as to obtain the target feature sequence that meets the preset resolution requirements generated in each iteration process.

[0077] The splicing unit is used to splice the target feature sequence to obtain the spliced ​​feature sequence, and performs feature aggregation operation on the spliced ​​feature sequence based on a preset attention mechanism to generate a full-slice level pathological feature representation.

[0078] Optionally, in one embodiment of this application, the feature prediction module 300 includes: a generation unit, used to input the pathological feature representation at the whole slice level into a preset classifier, so as to use the total loss function in the classifier to perform feature prediction, so as to generate a medical classification or prognostic prediction result model corresponding to each whole slice image data.

[0079] Optionally, in one embodiment of this application, the mathematical expression for the total loss function is:

[0080] in, Represents the total loss function; For hyperparameters; D Indicates the number of iteration rounds; This represents the entropy regularization term.

[0081] It should be noted that the foregoing explanation of the multi-scale analysis method embodiment for whole slice images also applies to the multi-scale analysis device for whole slice images in this embodiment, and will not be repeated here.

[0082] The multi-scale analysis device for whole-slice images proposed in this application includes a feature extraction module 100, used to preprocess and extract features from a preset whole-slice image dataset to generate instance feature sets at different resolution levels and establish spatial mapping relationships between these sets; a cross-scale feature interaction module 200, used to iteratively perform cross-scale feature interaction on instance feature sets at different resolution levels based on the spatial mapping relationship, to obtain and splice target feature sequences that meet preset resolution requirements generated in each iteration, obtaining corresponding spliced ​​feature sequences, and performing preset feature aggregation operations on the spliced ​​feature sequences to generate a whole-slice-level pathological feature representation; and a feature prediction module 300, used to perform preset feature prediction operations based on the whole-slice-level pathological feature representation to generate medical classification or prognostic prediction results corresponding to each whole-slice image data. This application addresses the problems of irreversible information loss and the easy neglect of minor lesions in existing technologies by establishing a cross-scale bidirectional interaction mechanism, a retrieval mechanism, and a memory mask strategy, thereby improving the accuracy of whole-slice image feature extraction and clinical task prediction.

[0083] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0084] When the processor 602 executes the program, it implements the multi-scale analysis method for full-slice images provided in the above embodiments.

[0085] Furthermore, electronic devices also include: Communication interface 603 is used for communication between memory 601 and processor 602.

[0086] The memory 601 is used to store computer programs that can run on the processor 602.

[0087] The memory 601 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage.

[0088] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a PCI bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0089] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0090] Processor 602 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application.

[0091] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-scale analysis method for full-slice images as described above.

[0092] This application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-described multi-scale analysis method for whole-slice images.

[0093] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0094] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0095] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0096] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0097] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0098] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0099] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0100] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A multi-scale analysis method for whole-slice images, characterized in that, Includes the following steps: Preprocessing and feature extraction operations are performed on a preset full-slice image dataset to generate instance feature sets at different resolution levels, and a spatial mapping relationship is established between the instance feature sets at different resolution levels. Based on the spatial mapping relationship, cross-scale feature interaction is iteratively performed on the instance feature sets at different resolution levels to obtain and splice the target feature sequences that meet the preset resolution requirements generated in each iteration process, to obtain the corresponding spliced ​​feature sequences, and a preset feature aggregation operation is performed on the spliced ​​feature sequences to generate a full-slice level pathological feature representation. Based on the pathological feature representation at the whole slice level, a preset feature prediction operation is performed to generate medical classification or prognostic prediction results corresponding to each whole slice image data.

2. The multi-scale analysis method for whole-slice images according to claim 1, characterized in that, The preprocessing and feature extraction operations on the preset full-slice image dataset to generate instance feature sets at different resolution levels, and the establishment of spatial mapping relationships between the instance feature sets at different resolution levels, include: The whole slice image dataset is cropped at different resolution levels to obtain corresponding image patches, and the image patches are encoded using a pre-trained pathology basic model to obtain the instance feature set at different resolution levels. Based on a preset mapping function, the resolution regions corresponding to the instance feature sets at different resolution levels are determined to construct the spatial mapping relationship.

3. The multi-scale analysis method for whole-slice images according to claim 2, characterized in that, Based on the spatial mapping relationship, the method iteratively performs cross-scale feature interaction on the instance feature sets at different resolution levels to obtain and splice the target feature sequences that meet the preset resolution requirements generated in each iteration, thereby obtaining the corresponding spliced ​​feature sequences. A preset feature aggregation operation is then performed on the spliced ​​feature sequences to generate a full-slice-level pathological feature representation, including: Filter at least one key region from the low-resolution instance feature set in the instance feature set of different resolution levels that meets the preset keyness requirements, and map the region features of each key region in the at least one key region to the corresponding latent space vector, so as to calculate the sampling probability distribution and importance score of each key region based on the latent space vector; Based on the sampling probability distribution, the high-resolution instance feature set in the instance feature set of different resolution levels is sampled to obtain the corresponding sampled features and unsampled features. The unsampled region features are aggregated through a preset recycling mechanism to generate the corresponding global recycling token. The unsampled features and the global reclamation token are input into a preset attention layer to determine the global context relationship corresponding to the sampled features and the unsampled features based on the spatial mapping relationship, and the global context relationship is modeled to output the corresponding focused features; Based on a preset cross-attention mechanism, the focused feature and the low-resolution instance feature set are iteratively interacted to update the low-resolution instance feature set and the preset historical mask matrix, so as to obtain the target feature sequence that meets the preset resolution requirements generated in each iteration process. The target feature sequence is spliced ​​together to obtain the spliced ​​feature sequence, and based on a preset attention mechanism, feature aggregation operation is performed on the spliced ​​feature sequence to generate the whole-slice level pathological feature representation.

4. The multi-scale analysis method for whole-slice images according to claim 1, characterized in that, The process of performing a preset feature prediction operation based on the pathological feature representation at the whole-slice level to generate a medical classification or prognostic prediction result corresponding to each whole-slice image data includes: The pathological features represented at the whole slice level are input into a preset classifier, and the total loss function in the classifier is used to perform feature prediction to generate a medical classification or prognostic prediction model corresponding to each whole slice image data.

5. The multi-scale analysis method for whole-slice images according to claim 4, characterized in that, The mathematical expression for the total loss function is: in, Represents the total loss function; For hyperparameters; D Indicates the number of iteration rounds; This represents the entropy regularization term.

6. A multi-scale analysis device for whole-slice images, characterized in that, include: The feature extraction module is used to preprocess and extract features from a preset full-slice image dataset to generate instance feature sets at different resolution levels and to establish a spatial mapping relationship between the instance feature sets at different resolution levels. The cross-scale feature interaction module is used to iteratively perform cross-scale feature interaction on the instance feature sets at different resolution levels based on the spatial mapping relationship, so as to obtain and splice the target feature sequence that meets the preset resolution requirements generated in each iteration process, obtain the corresponding spliced ​​feature sequence, and perform preset feature aggregation operation on the spliced ​​feature sequence to generate a full-slice level pathological feature representation. The feature prediction module is used to perform preset feature prediction operations based on the pathological feature representation at the whole slice level, so as to generate medical classification or prognostic prediction results corresponding to each whole slice image data.

7. The multi-scale analysis device for whole-slice images according to claim 6, characterized in that, The feature extraction module includes: The cropping unit is used to perform cropping operations at different resolution levels on the whole slice image dataset to obtain corresponding image patches, and to encode the image patches through a pre-trained pathology basic model to obtain the instance feature set at the different resolution levels. The construction unit is used to determine the resolution region corresponding to the instance feature set at different resolution levels based on a preset mapping function, so as to construct the spatial mapping relationship.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the multi-scale analysis method for whole-slice images as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the multi-scale analysis method for full-slice images as described in any one of claims 1-5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the multi-scale analysis method for full-slice images as described in any one of claims 1-5.

Images