Full-slice image encoder training method and full-slice image encoding method, device, equipment, medium and product

Abstract

The application discloses a whole slice image encoder training method and an image encoding method, device, equipment, medium and product, and is applied to the field of medical image processing. The whole slice image encoder training method comprises the following steps: pre-processing images in a pathological whole slice sample image set to obtain an image block pixel sequence and a two-dimensional space coordinate set; inputting the image block pixel sequence and the two-dimensional space coordinate set into a first whole slice image encoder to obtain an image encoding result; performing loss calculation based on the image encoding result and first attribute information to obtain a learning loss; and updating parameters of the first whole slice image encoder based on the learning loss to obtain a second whole slice image encoder. The technical scheme of the embodiment of the application changes the existing whole slice image encoder training paradigm, realizes end-to-end training of the whole slice encoder, and enables the encoder to comprehensively perceive the spatial distribution law of tissues and learn general features and task-specific features.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and artificial intelligence, and in particular to a whole-slice image encoder training method and a whole-slice image encoding method, apparatus, device, medium and product. Background Technology

[0002] Whole Slide Image (WSI) is an ultra-high resolution image obtained by completely digitizing glass pathology slides using a high-powered scanner. A single image can reach a size of 100,000 × 100,000 pixels. The resolution of WSI far exceeds the direct processing capability of existing neural networks. Current WSI encoders produce coarse feature extraction results and lose a lot of image information, resulting in poor application performance of the image encoding results. Summary of the Invention

[0003] This invention provides a training method for a full-slice image encoder and a method, apparatus, device, medium, and product for encoding full-slice images. By changing the training mechanism, the trained full-slice image encoder can perceive the spatial distribution patterns of tissues and learn task-specific features, significantly improving the accuracy and reliability of the encoding results of the full-slice image encoder.

[0004] In a first aspect, embodiments of the present invention provide a method for training a full-slice image encoder, the method comprising: A set of whole pathological slide sample images is obtained, and each sample image in the set is preprocessed to obtain the image block pixel sequence and two-dimensional spatial coordinate set of each sample image; The image block pixel sequence and the set of two-dimensional spatial coordinates are input into the first full-slice image encoder to obtain the image encoding result output by the first full-slice image encoder; The learning loss is calculated based on the image encoding results and the first attribute information of each sample image; wherein, the first attribute information is at least one of the classification label information and pathological diagnosis text information corresponding to each sample image. The parameters of the first full-slice image encoder are updated based on the backward gradient of the learning loss to obtain the second full-slice image encoder.

[0005] Secondly, embodiments of the present invention provide a method for encoding a whole-slice image, the method comprising: Obtain the first pathological whole slide image; The first pathological whole slice image is preprocessed to obtain the first image block pixel sequence and the first coordinate set, wherein the first coordinate set is the two-dimensional spatial coordinate set corresponding to the first image block pixel sequence; The first image block pixel sequence and the first coordinate set are input into the first full-slice image encoder to obtain the first image encoding result output by the first full-slice image encoder. The first full-slice image encoder is an encoder obtained by the full-slice image encoder training method of any embodiment.

[0006] Thirdly, embodiments of the present invention also provide a full-slice image encoder training apparatus, the apparatus comprising: The sample preprocessing module is used to obtain a set of pathological whole slide sample images and preprocess each sample image in the set of pathological whole slide sample images to obtain the image block pixel sequence and two-dimensional spatial coordinate set of each sample image; The sample input module is used to input the image block pixel sequence and the two-dimensional spatial coordinate set into the first full-slice image encoder to obtain the image encoding result output by the first full-slice image encoder; The loss calculation module is used to calculate the learning loss based on the image encoding result and the first attribute information of each sample image; wherein, the first attribute information is at least one of the classification label information and pathological diagnosis text information corresponding to each sample image; The training update module is used to update the parameters of the first full-slice image encoder based on the backward gradient of the learning loss, so as to obtain the second full-slice image encoder.

[0007] Fourthly, embodiments of the present invention also provide a full-slice image encoding apparatus, the apparatus comprising: The image acquisition module is used to obtain the first pathological whole slide image; The image preprocessing module is used to preprocess the first pathological whole slice image to obtain a first image block pixel sequence and a first coordinate set, wherein the first coordinate set is a two-dimensional spatial coordinate set corresponding to the first image block pixel sequence; The image encoding module is used to input the pixel sequence of the first image block and the first coordinate set into the first full-slice image encoder to obtain the first image encoding result output by the first full-slice image encoder, wherein the first full-slice image encoder is an encoder obtained by the full-slice image encoder training method described in any embodiment.

[0008] Fifthly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the full-slice image encoder training method or the full-slice image encoding method as described in any of the embodiments of the present invention.

[0009] In a sixth aspect, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the full-slice image encoder training method or the full-slice image encoding method as described in any of the embodiments of the present invention.

[0010] In a seventh aspect, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the full-slice image encoder training method or the full-slice image encoding method as described in any of the embodiments of the present invention.

[0011] In this embodiment of the invention, a set of pathological whole-section sample images is obtained, and each sample image in the set is preprocessed to obtain the image block pixel sequence and two-dimensional spatial coordinate set of each sample image. The image block pixel sequence and two-dimensional spatial coordinate set are input into a first whole-section image encoder to obtain the image encoding result output by the first whole-section image encoder. Based on the image encoding result and the first attribute information of each sample image, a loss is calculated to obtain the learning loss. The first attribute information is at least one of the classification label information and pathological diagnosis text information corresponding to each sample image. Based on the learning loss, the parameters of the first whole-section image encoder are updated using the backward gradient to obtain a second whole-section image encoder. This technical solution solves the problems of cross-regional spatial information loss and the inherent upper limit of image recognition classification accuracy in existing whole-section image encoder separation training methods. It enables the trained whole-section image encoder to perceive the spatial distribution pattern of tissues and learn task-specific features, significantly improving the accuracy and reliability of high-difficulty pathological image recognition tasks such as tumor subtype identification and gene mutation prediction. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a full-slice image encoder training method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating another full-slice image encoder training method provided in this embodiment of the invention; Figure 3 A flowchart of a full-slice image encoding method provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a full-slice image encoder training device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a full-slice image encoder training device provided in an embodiment of the present invention; Figure 6 A flowchart illustrating an application example of a full-slice image encoder training method provided in this embodiment of the invention; Figure 7 This is a schematic diagram of the structure of a panoramic pathological image coding network provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of a panoramic spatial attention module in a panoramic pathological image coding network provided in an embodiment of the present invention; Figure 9 A schematic diagram of window division provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0014] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0015] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of national laws and regulations.

[0016] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the relevant content of the solution.

[0017] Digital whole-slide images (WSI) are ultra-high-resolution images obtained by completely digitizing glass pathology slides using a high-powered scanner, with single images reaching sizes on the order of 100,000 × 100,000 pixels. Since the resolution of WSI far exceeds the direct processing capabilities of existing neural networks, effectively training and inferring WSI is a core technical challenge in this field. Currently, most WSI image encoders rely on a two-stage separate training method to complete the encoding modeling of WSI: The first stage involves independently training the feature extraction network. Specifically, a large number of small pathological image patches are used as training data, and self-supervised contrastive learning and other methods are employed to pre-train the feature extraction network, enabling it to represent general pathological image features. After training, the network parameters are frozen and no longer updated. In practical use, the WSI (Web Image Sensor) is divided into thousands to tens of thousands of small image patches, which are fed into the frozen network one by one. Each patch outputs an independent feature vector; no information is passed between the image patches, and they are completely independent of each other.

[0018] The second stage involves the independent training of the aggregation network. Specifically, the aggregation network is trained independently on WSI data with diagnostic labels, using all the image patch feature vectors output from the first stage as fixed input. The aggregation network receives the set of feature vectors, learns to summarize them into slice-level representations, and finally outputs the diagnostic results.

[0019] However, in practical applications, the above two-stage separate training method has the following two core problems: Question 1: The accuracy of image encoding results in subsequent image recognition results has an inherent upper limit.

[0020] Because the parameters of the feature extraction network in the first stage are frozen, the gradient of the diagnostic error in the second stage cannot be backpropagated to the patch feature layer. The extracted features are general features for understanding general pathological images, rather than specialized features optimized for specific image recognition and classification tasks. When facing challenging image recognition and classification tasks such as fine identification of tumor subtypes and prediction of gene mutation states, a significant gap exists between the general features and the specific classification target, causing the aggregation network to face an information bottleneck. No matter how the second-stage aggregation network is optimized, the information capacity of its input features is locked by the first stage, forming an insurmountable accuracy ceiling. This accuracy ceiling directly leads to an increased risk of misdiagnosis in clinical scenarios, affecting doctors' patient treatment decisions.

[0021] Question 2: Loss of spatial information across regions restricts the overall image recognition and classification capabilities.

[0022] Because image patches are independent and do not transmit information to each other during the feature extraction stage, neural networks cannot perceive cross-regional spatial distribution patterns. However, in pathological image analysis, cross-regional features such as the spatial continuity of tumor invasion patterns, the regional distribution patterns of immune cells in the tumor microenvironment, and the spatial structure of the tumor-stromal boundary are crucial for pathologists to determine malignancy and predict prognosis. The loss of cross-regional spatial information directly affects the accuracy of WSI tumor grading and survival prediction.

[0023] Figure 1This is a flowchart illustrating a method for training a full-slice image encoder according to an embodiment of the present invention. This embodiment is applicable to scenarios where a full-slice image encoder is trained in an end-to-end manner. The method can be executed by a full-slice image encoder training device, which can be implemented in software and / or hardware and integrated into a computer device.

[0024] like Figure 1 As shown, the training method for a full-slice image encoder includes the following steps: S110. Obtain a set of whole pathological slide sample images, and preprocess each sample image in the set of whole pathological slide sample images to obtain the image block pixel sequence and two-dimensional spatial coordinate set of each sample image.

[0025] Among them, the pathological whole slide sample image set can be a collection of multiple digital pathological whole slide images. Each whole slide image is an ultra-high resolution image obtained by digitizing a glass pathological slide using a high-power microscope scanner, and the size of a single image can reach the level of 100,000 by 100,000 pixels.

[0026] Preprocessing of each sample image in the pathological whole slide sample image set can be performed by identifying and segmenting the tissue region of each whole slide image, decomposing it into a large number of small image blocks that can be directly processed by the network, and simultaneously recording the position information of each image block in the original whole slide.

[0027] The resulting image block pixel sequence after preprocessing of each sample image in the pathological whole slide sample image set is a sequence of pixel data composed of image blocks arranged in a certain spatial order (such as from left to right or from top to bottom). Each image block contains local tissue structure pixel information. The two-dimensional spatial coordinate set can record the position coordinates of each image block in the above sequence on its corresponding whole slide image, such as the row and column coordinates of the upper left corner of the image block in the original image, to preserve the spatial relationship between blocks.

[0028] In one optional implementation, tissue region detection can be performed on the full-slice image first to segment out regions containing effective tissue. Then, using a fixed-size, non-overlapping grid, image blocks are sequentially cropped starting from the upper left corner of the tissue region. The cropped image blocks are arranged in the scanning order to form a pixel sequence, and the horizontal and vertical coordinates corresponding to the upper left corner of each image block are stored in a two-dimensional spatial coordinate set. Alternatively, a sliding window with a fixed step size can be used to extract image blocks with some overlap in the tissue region. All image blocks are then arranged in the sliding window traversal order to form a pixel sequence, and the two-dimensional coordinates of the center point or upper left corner of each window are recorded as a spatial coordinate set.

[0029] In another optional implementation, preprocessing the images of each sample in the pathological whole-section sample image set can include three steps. First, the WSI low-resolution thumbnail is segmented into tissue regions using an image segmentation model, automatically identifying and removing background blank areas, retaining only the effective areas containing actual tissue. The WSI low-resolution thumbnail is an overview image obtained by downsampling the ultra-high resolution whole-section image to a lower resolution, significantly reducing the total number of pixels for faster processing while preserving the overall spatial distribution relationship between the tissue outline and the blank background. Next, an arbitrary color normalization method is used to normalize the staining of the effective tissue region, eliminating color differences between different slide batches and staining conditions. Further, the effective tissue region is divided into N non-overlapping image blocks by sliding at a fixed size p×p, and the row and column coordinates of the top-left corner of each image block in the original image are used to preserve the spatial relationship between the blocks.

[0030] This step transforms the giant full-slice image into an ordered sequence of image block pixels and displays the two-dimensional spatial coordinates of each image block, enabling the subsequent encoder to not only acquire local tissue details but also accurately understand the spatial relationships between the image blocks.

[0031] S120. Input the image block pixel sequence and the two-dimensional spatial coordinate set into the first full-slice image encoder to obtain the image encoding result output by the first full-slice image encoder.

[0032] The first full-slice image encoder can be a neural network model that can simultaneously receive visual information and corresponding spatial location information of image blocks. It can jointly model the image block sequence with a set of two-dimensional coordinates to generate an image encoding result that integrates spatial context.

[0033] The image encoding result is the full-slice-level feature representation output by the first full-slice image encoder after processing the entire image block sequence and spatial coordinates. It can be a fixed-dimensional vector that integrates local morphological features and cross-regional spatial distribution information.

[0034] In one alternative implementation, the first full-slice image encoder can adopt a Transformer structure based on a self-attention mechanism. First, each image block is processed through a convolutional sub-network to extract initial visual features. Then, a positional code is generated using two-dimensional spatial coordinates and added to the visual features, so that each image block feature carries its spatial positional information. Subsequently, all block features with positional information are combined into a sequence and fed into a multi-layer self-attention network. Cross-regional spatial information is fused through global interaction, and finally, the full-slice level image encoding result is output.

[0035] In another alternative implementation, the first full-slice image encoder uses a graph neural network, taking the visual features of each image block as node features, and constructing edge connections between adjacent or near-neighboring image blocks based on two-dimensional spatial coordinates to form a spatial graph structure; through multiple message passing and node feature updates, the features of each node are fused with its spatial neighborhood information, and finally all node features are aggregated to obtain the image encoding result.

[0036] In another optional implementation, the first full-slice image encoder includes an image patch embedding layer, a multi-layer panoramic spatial attention coding network layer, and a feature aggregation layer (Gated Attention-based Multiple Instance Learning, Gated ABMIL). The image patch pixel sequence and the set of two-dimensional spatial coordinates are input into the first full-slice image encoder to obtain the image encoding result output by the first full-slice image encoder, including: S1201. Input the image block pixel sequence and the two-dimensional spatial coordinate set into the image block embedding layer in the first full-slice image encoder to obtain the image block unit feature vector sequence output by the shared linear projection of the image block embedding layer.

[0037] The image patch embedding layer is the front-end network layer of the first full-slice image encoder, which is used to map the original pixel data of each image patch into a fixed-dimensional numerical vector, completing the initial transformation from image space to feature space.

[0038] Shared linear projection refers to projecting all image patches using the same linear transformation matrix, sharing parameters that do not change due to differences in patch position or content, thus ensuring fairness and efficiency in the embedding. The image patch unit feature vector sequence is a list of feature vectors that correspond one-to-one with the input image patch sequence. Each vector is a low-dimensional numerical representation of an image patch, carrying basic tissue texture, cell morphology, and other visual information about that patch.

[0039] In one implementation, each image patch can be flattened into a one-dimensional pixel vector. All image patches are then passed in parallel through the same fully connected linear layer for shared projection, outputting feature vectors of uniform dimension. These feature vectors are then arranged in the original image patch order to obtain a sequence of image patch unit feature vectors. For example, a full slice can be divided into 10,000 image patches of size 256 by 256. Each patch can be flattened into a 65,536-dimensional pixel vector, and then the dimension can be reduced to 1,024 dimensions through shared linear projection, resulting in a sequence of 10,024-dimensional feature vectors.

[0040] Alternatively, instead of using fully connected layers directly, a small convolutional neural network can be used as a shared embedding module. Each image patch extracts local texture features individually through this small network, and all patches share the network parameters. The output feature map is then globally pooled to form a sequence of image patch unit feature vectors, thus possessing a certain degree of invariance and local structure extraction capability during the embedding stage. In another implementation, position-adaptive embedding can be introduced. While sharing linear projection, a lightweight offset coefficient or scaling factor is generated based on the two-dimensional spatial coordinates to modulate the result of the linear projection. This allows the embedding vector to be slightly aware of its own spatial position in the initial stage, while the basic projection weights remain shared.

[0041] Shared linear projection transforms all image patches into feature vectors based on a unified rule, ensuring the consistency of the feature space and avoiding feature shifts caused by different mappings. Simultaneously, the preservation of two-dimensional spatial coordinates provides a regular and comparable input basis for the subsequent attention network to simultaneously process visual content and spatial relationships, enabling the network to treat every local region equally and not miss any lesion information.

[0042] S1202. Input the image block unit feature vector sequence and the two-dimensional spatial coordinate set into the multi-layer panoramic spatial attention coding network layer to obtain multiple image block unit feature vector sub-sequences and feature vector sequences extracted by the local attention sub-module and the global attention sub-module connected in the multi-layer panoramic spatial attention coding network layer; wherein, the image block unit feature vector sub-sequence is obtained by grouping the image block unit feature vector sequence, and each image block unit feature vector sub-sequence contains the same number of image block unit feature vectors.

[0043] Among them, the multi-layer panoramic spatial attention coding network layer is a deep network module composed of multiple coding layers stacked together. It is specifically designed to extract spatial context features between image patches through an attention mechanism under the premise of fusing spatial coordinates.

[0044] The local attention submodule of the multi-layer panoramic spatial attention encoding network performs attention operations within each image patch subsequence, calculating only the mutual influence between image patches within the same group, aiming to extract local organizational morphology and neighboring structural relationships. The global attention submodule of the multi-layer panoramic spatial attention encoding network can perform attention operations across different subsequences, acting on the features of all image patches or the features represented by the subsequence, to capture long-distance cross-regional spatial distribution patterns and macroscopic inter-organizational relationships.

[0045] Image block unit feature vector subsequences are several shorter, equal-length subsequences obtained by uniformly dividing the original image block unit feature vector sequence according to spatial proximity or a predetermined grouping strategy. The feature vector subsequences are updated features output after each subsequence has been alternately encoded with local and global attention. Their number is consistent with the number of feature vectors within the subsequence, but they have already incorporated intra-group and cross-group spatial information.

[0046] For example, image patch feature vector sequences can be grouped based on two-dimensional spatial coordinates and spatial location. For instance, a full slice can be divided into multiple equally sized spatial grids, with image patch features falling within the same grid grouped into a subsequence. Each subsequence is input into a local attention submodule, where each feature vector is weighted by attention to aggregate information from other features within the group, resulting in locally enhanced subsequence features. Subsequently, the outputs of all subsequences, along with their spatial region representative coordinates, are fed into a global attention submodule for cross-regional attention interaction at a global scale. This allows macroscopic features from different organizational regions to flow and correct each other, ultimately outputting a subsequence feature vector sequence with global spatial awareness. Alternatively, in another implementation, local attention employs window attention based on spatial distance constraints, allowing each image patch to interact directly only with its K spatially neighboring image patches, without forcing strict grouping, thus forming a large number of overlapping local receptive fields. Then, a long-range global attention across windows is used to globally correlate the aggregated features of each local window, resulting in a feature sequence that fuses multi-scale spatial information.

[0047] By linking local attention and global attention, the encoding network can first understand the local neighborhood features of an organization in detail, and then model the global spatial relationship between different organizational regions, thus completely opening up the information exchange channel between local details and overall spatial distribution.

[0048] S1203. Input the feature vector sub-sequences of multiple image block units into the feature aggregation layer to obtain the image encoding result after feature aggregation by the feature aggregation layer.

[0049] The feature aggregation layer can be a pooling aggregation module that combines attention and gating mechanisms. It is used to compress a set of feature vectors of varying numbers of subsequences into a fixed-dimensional feature representation at the full slice level, while automatically identifying and enhancing key instances for image recognition and classification, and suppressing background or noisy instances.

[0050] The image encoding result is the full-slice-level feature vector output by the aggregation layer, which encapsulates the local morphological features, spatial distribution features, and decisive instance information of the image recognition and classification results of the entire slice.

[0051] In one alternative implementation, all subsequence feature vectors can be expanded into a large feature set. Each feature vector is weighted by an attention branch and its information validity is determined by a gating branch. The two are multiplied and then weighted and summed to obtain a aggregated feature vector as the image encoding result. For example, for a slice containing a large number of normal tissue blocks and a small number of tumor regions, the attention branch will assign high weights to the tumor region features, while the gating mechanism will suppress the uncertainty contribution of blank areas or blurred image blocks, ensuring that the aggregated result focuses on the key tissues in the image recognition and classification results.

[0052] Gated attention-based aggregation layers can selectively focus on the most discriminative image patch features for image recognition and classification tasks, and automatically suppress irrelevant regions, thereby extracting a high-information-density full-slice representation from the entire set of image patch features. This makes the image encoding result no longer a simple average or random mixture of information from all regions, but a compact representation with strong classification result directionality, providing a high-quality information foundation for subsequent task decisions.

[0053] S130. Based on the image encoding results and the first attribute information of each sample image, the loss is calculated to obtain the learning loss.

[0054] The first attribute information is at least one of the classification label information and pathological diagnosis text information corresponding to each sample image.

[0055] The image encoding result is the full-slice-level feature representation obtained by S120, which incorporates spatial information. The first attribute information can be task supervision information associated with the full-slice image, classification labels such as tumor grade and subtype, diagnostic description text written by a pathologist, or a combination of both. The first attribute information can be selected and confirmed based on the training target of the full-slice image encoder.

[0056] The learning loss is a scalar loss value calculated by measuring the consistency between the image encoding result and the first attribute information. It is used to guide the network parameter update so that the encoding result can accurately reflect the image recognition and classification feature information related to the task.

[0057] For example, when the first attribute information is a classification label, the image encoding result can be input into a pre-trained classification prediction head to obtain the classification prediction result for each sample image. Based on the classification prediction result and classification label information, a cross-entropy loss is obtained and used as a learning loss to optimize the image encoding result towards correctly distinguishing different category labels. The pre-trained classification prediction head can be a shallow neural network module that has undergone preliminary training or random initialization and will continue to learn in this task. It is typically a fully connected layer with Softmax activation, used to map the image encoding result to the predicted probabilities of each category. The classification prediction result is the network's probability estimate of the current whole-slice image belonging to each possible category, such as an 80% probability of tumor subtype A and a 20% probability of subtype B. The classification label information is the true category label of the whole-slice sample image of the case, such as the pathologically confirmed tumor subtype category.

[0058] Specifically, the image encoding results are passed through a classification prediction head consisting of a fully connected layer and a Softmax function, outputting probability vectors for each preset category. These probability vectors are then used to calculate the standard cross-entropy loss with the one-hot encoded true classification labels, serving as the learning loss. For example, in the bladder cancer grading information recognition task, the image encoding results are processed by the classification head to output low-level and high-level probabilities, which are then used to calculate cross-entropy with the "high-level" label annotated by the pathologist, driving the network to improve the prediction probability of high-level categories. In an optional implementation, a weighted cross-entropy loss can be used on top of the above classification prediction head, assigning different weights to the losses of different categories to address clinical situations with severe imbalances in sample categories; the weights can be set based on the inverse of the sample count for each category or the sensitivity of the physician's focus. In another implementation, a focus loss can be used instead of the cross-entropy loss. By reducing the weight of the loss for easily classified samples, the training process becomes more focused on difficult samples and cases with ambiguous boundaries, suitable for more challenging classification tasks such as fine identification of tumor subtypes. By directly calculating the loss between the image encoding result and the image classification label, the entire full-slice image encoder can be optimized end-to-end from the final image recognition target, breaking through the bottleneck of frozen feature network parameters and inability to transmit task signals in traditional separation training. As a result, the feature extraction process can autonomously discover subtle tissue differences that are highly correlated with the classification label, making the image encoding result closely match the task requirements and significantly improving the accuracy of fine classification.

[0059] Alternatively, when the first attribute information is pathological diagnostic text information, the pathological diagnostic text information can be text-encoded to obtain a text embedding vector. Based on the text embedding vector and the image encoding result, an information noise contrast estimation loss can be obtained, which is then used as a learning loss to align the image encoding result with the diagnostic semantics. The pathological diagnostic text information can be a diagnostic description written by a pathologist for a whole-slice image, microscopic findings, or structured report text, containing rich semantic knowledge, such as "invasive ductal carcinoma is visible, and immune cells are clustered in the stroma." The text embedding vector can be generated using a pre-trained natural language processing model, transforming the variable-length diagnostic text into a fixed-dimensional numerical vector while preserving the medical semantic information of the text. The information noise contrast estimation loss can be a contrastive learning loss designed to bring matching image-text pairs closer in the common embedding space and push away representations of mismatched pairs, thereby aligning the image encoding result with the diagnostic semantics.

[0060] For example, a text encoder pre-trained on a medical corpus can be used to extract the embedding vector of the pathological diagnosis text corresponding to each full slice; simultaneously, the image encoding result is mapped to the same dimensional space as the text embedding vector through a projection head. A training batch is constructed, containing multiple matching image-text pairs. For each image, its projected image encoding result is used as the anchor point, the matching text embedding vector is the positive sample, and other texts in the same batch are negative samples. The information-noise contrast estimation loss is calculated to maximize the cosine similarity of positive sample pairs and minimize the similarity of negative sample pairs.

[0061] By using pathological diagnostic text as a supervisory signal and employing cross-modal contrastive learning, the image encoding results can incorporate the diagnostic semantic knowledge of pathologists, encoding features rich in medical meaning. This overcomes the information sparsity limitation caused by using only discrete category labels, enabling the image encoding results to capture richer and more nuanced features associated with image recognition and classification decisions. For tasks lacking clear classification boundaries, such as gene mutation prediction, it can provide more effective feature representation, further improving the performance of image recognition and prediction.

[0062] In addition, for the knowledge distillation task of the full-slice image encoder, feature alignment loss or KL divergence loss can be used.

[0063] When the training objective includes multiple tasks, the total loss is a weighted sum of the losses from each task, and the weights can be flexibly adjusted according to the importance of the tasks and the training stage. All of the above loss functions satisfy the differentiability condition, and their gradients can be backpropagated throughout the entire training process.

[0064] S140. Update the parameters of the first full-slice image encoder based on the backward gradient of the learning loss to obtain the second full-slice image encoder.

[0065] Backward gradient update can be achieved by using the learning loss to calculate the gradient of the loss with respect to all trainable parameters of the first full-slice image encoder through backpropagation algorithm, and then using an optimizer to adjust the parameter values ​​in the direction of reducing the loss, thus realizing end-to-end learning. The second full-slice image encoder is the full-slice image encoder that has been trained and updated with parameters based on the above-mentioned pre-defined task (such as image recognition and classification) loss. It has transformed from a general feature representation capability into a dedicated full-slice image encoder for specific pathological slide image recognition tasks.

[0066] For example, after calculating the learning loss, the automatic differentiation mechanism can propagate the gradient from the loss value backward along the forward computation graph of the network to the parameters of each layer of the first full-slice image encoder. The adaptive moment estimation optimizer is used to update the weights of all convolutional layers, self-attention layers and related mapping layers according to the set learning rate. After the update is completed, the current parameter state is saved as the second full-slice image encoder.

[0067] In another implementation, different learning rates can be set for different modules within the full-slice image encoder during backpropagation. For example, a smaller learning rate can be used for the low-level visual feature extraction part, while a larger learning rate can be used for the high-level spatial fusion and task adaptation parts. This balances the preservation of general visual knowledge with task-specific adaptation, resulting in a second full-slice image encoder after parameter updates. Alternatively, momentum update or gradient clipping mechanisms can be introduced into the backpropagation gradient update to stabilize the training process. This process is iterated multiple times until the learning loss converges, and finally, the parameters of the converged full-slice image encoder are fixed as those of the second full-slice image encoder. The technical solution of this invention involves obtaining a set of pathological whole-slice sample images and preprocessing each sample image in the set to obtain the image block pixel sequence and two-dimensional spatial coordinate set of each sample image. The image block pixel sequence and two-dimensional spatial coordinate set are then input into a first whole-slice image encoder to obtain the image encoding result output by the first whole-slice image encoder. The first whole-slice image encoder includes an image block embedding layer, a multi-layer panoramic spatial attention coding network layer, and a feature aggregation layer. A loss calculation is performed based on the image encoding result and the first attribute information of each sample image to obtain a learning loss. The first attribute information includes at least one of the classification label information and pathological diagnosis text information corresponding to each sample image. The parameters of the first whole-slice image encoder are updated based on the learning loss and backward gradient to obtain a second whole-slice image encoder. This technical solution solves the problems of cross-regional spatial information loss and the inherent upper limit of image recognition classification accuracy in existing whole-slice image encoder separation training methods. By enabling the trained whole-slice image encoder to perceive the spatial distribution pattern of tissues and learn task-specific features, it significantly improves the accuracy and reliability of challenging pathological image recognition tasks such as tumor subtype identification and gene mutation prediction.

[0068] Figure 2 This is a flowchart illustrating a full-slice image encoder training method provided in an embodiment of the present invention. This embodiment is applicable to end-to-end full-slice image encoder training scenarios, i.e., it shares the same inventive concept as the above embodiments, and further explains the feature extraction process of the encoder's multi-layer panoramic spatial attention coding network layer. This method can be executed by a full-slice image encoder training device, implemented in software and / or hardware, and integrated into a computer device.

[0069] like Figure 2 As shown, the training method for a full-slice image encoder includes the following steps: S210. Obtain a set of whole pathological slide sample images, and preprocess each sample image in the set of whole pathological slide sample images to obtain the image block pixel sequence and two-dimensional spatial coordinate set of each sample image.

[0070] S220. Input the image block pixel sequence and the two-dimensional spatial coordinate set into the image block embedding layer in the first full-slice image encoder to obtain the image block unit feature vector sequence output by the shared linear projection of the image block embedding layer.

[0071] The image patch embedding layer can be the frontmost network layer of the first full-slice image encoder, responsible for mapping the pixel data of each original image patch to a fixed-dimensional numerical vector, completing the initial transformation from the image domain to the feature domain. Shared linear projection allows all image patches to be projected using the same learnable linear transformation matrix, with parameters that do not change with the position or content of the image patch, thus ensuring that all image patches obtain a unified initial representation in the same feature space.

[0072] The image patch unit feature vector sequence can be a list of feature vectors that correspond one-to-one with the input image patch sequence. Each vector is a low-dimensional feature representation of the corresponding image patch, carrying basic visual information such as cell morphology and tissue texture within the image patch.

[0073] For example, the pixel values ​​of each image patch can be flattened into a one-dimensional vector. All image patches are then linearly projected through a shared fully connected layer in parallel, outputting feature vectors of uniform dimension. These feature vectors are then arranged in the original image patch order to form a sequence of image patch unit feature vectors. For instance, a full slice can be divided into 10,000 image patches of size 256 by 256. Each patch can be flattened into a 65,536-dimensional pixel vector and then reduced to 1,024 dimensions through a shared linear projection, resulting in a sequence of 1,024-dimensional feature vectors.

[0074] In another implementation, a small convolutional neural network can be used as a shared embedding module. Each image block independently passes through the convolutional network to extract local texture and morphological features. All image blocks share all the parameters of the convolutional network. Finally, the feature map of each image block is globally pooled into a fixed-dimensional feature vector, forming a sequence of image block unit feature vectors.

[0075] Shared linear projection transforms all image patches into feature vectors according to a unified rule, ensuring consistency and comparability within the feature space and enabling the network to treat each local region equally. Based on this, the output sequence of image patch unit feature vectors provides a regular input foundation with complete preservation of positional information for subsequent spatial attention encoding, helping the network to detect lesions scattered throughout the network without bias.

[0076] S230. Based on the local attention submodule in the multi-layer panoramic spatial attention coding network layer, the image block unit feature vector sequence output by the image block embedding layer is grouped to obtain multiple image block unit feature vector subsequences.

[0077] The multi-layer panoramic spatial attention coding network layer can be the core deep network module in the first full-slice image encoder. It is composed of multiple coding sub-layers stacked together and aims to fuse spatial coordinate information and extract cross-regional spatial context features through an attention mechanism.

[0078] The local attention submodule in each panoramic spatial attention coding network layer can be a submodule within that coding network layer, specifically responsible for interacting with and enhancing image patch features within a local spatial range. Its operation is based on grouping the image patch feature sequence. The image patch unit feature vector subsequence is that the original image patch unit feature vector sequence is divided into multiple shorter equal-length subsequences according to spatial location. The feature vector in each subsequence corresponds to a set of image patches within a local rectangular region on the full slice image.

[0079] The image patch unit feature vector sequence output from the image patch embedding layer is grouped to obtain multiple image patch unit feature vector sub-sequences. This grouping can be based on spatial coordinates within a two-dimensional coordinate set. Each image patch unit feature vector in a sub-sequence forms a sample image sub-region with the same length and width. Essentially, the image patch unit feature vector sequence corresponding to the full-slice image is divided into several rectangular sub-regions of equal size. All image patch feature vectors within each sub-region constitute an image patch unit feature vector sub-sequence, ensuring that the tissue region corresponding to each sub-sequence has a geometric shape with equal length and width. For example, the entire image can be uniformly divided into a 16x16 grid, with each grid containing image patches forming a sub-sequence.

[0080] In another alternative implementation, a dynamic grouping strategy based on spatial distance can be adopted, in which a fixed number of image blocks that are spatially nearest to each image block are selected to form a subsequence. Although there may be overlap between subsequences, each subsequence still maintains the same number of feature vectors, which allows local attention to be performed in a more flexible spatial neighborhood.

[0081] The massive set of image patch features is grouped into multiple equal-length subsequences based on spatial relationships. This preserves the local spatial proximity of image patches within the original tissue slices while decomposing global computation into controllable intra-group computations. This allows the local attention submodule to efficiently extract fine-grained local spatial context features within each tissue subregion. This grouping operation is a crucial bridge connecting the original image patches with spatially aware coding, laying the foundation for subsequent hierarchical spatial modeling that proceeds from local to global perspectives.

[0082] S240. Insert a learnable unit feature vector into each image block unit feature vector subsequence, and perform feature calculation on each image block unit feature vector subsequence and learnable unit feature vector based on the multi-head self-attention mechanism to obtain the first subsequence feature vector, the first subsequence mean feature vector, and the subsequence coordinate label information.

[0083] The learnable unit feature vector can be a special vector that is randomly initialized and continuously optimized through training. It is inserted into each image patch unit feature vector subsequence and acts as an information aggregation node, used to extract and summarize global information at the subsequence level from the image patch features of that subsequence.

[0084] Multi-head self-attention is a computational method that allows each element in a sequence to interact with all other elements based on its own content, capturing different types of association patterns through multiple parallel attention heads. The first subsequence feature vector is the learnable unit feature vector updated after multi-head self-attention computation. It aggregates information from all image patch features within its subsequence and can serve as a summary representation of that subsequence.

[0085] The mean feature vector of the first subsequence is the feature vector obtained by averaging the feature vectors of all image blocks within the subsequence after multi-head self-attention operation, reflecting the average organizational characteristics of the sub-region. The subsequence coordinate labeling information can be the geometric center coordinates of the spatial rectangular sub-region corresponding to the subsequence, which can be obtained by calculating the centroid from the two-dimensional spatial coordinates of all image blocks within the sub-region, and is used to represent the spatial position of the subsequence in the original whole slice image.

[0086] For example, the relative position information of each image block unit feature vector within its corresponding image block unit feature vector subsequence can be first superimposed onto each image block unit feature vector to obtain an image block unit feature vector carrying subsequence position information. This involves generating a relative position encoding within its subsequence for each image block unit feature vector and superimposing it onto the feature vector, thus carrying layout information within the subsequence. Next, the learnable unit feature vector is concatenated with all image block unit feature vectors carrying subsequence position information to form a joint feature sequence. This yields the first subsequence feature vector and the first subsequence mean feature vector. The centroid of the spatial coordinates of the image block unit feature vector subsequence corresponding to the joint feature sequence is used as the subsequence coordinate label information. Here, the first subsequence feature vector is the learnable unit feature vector processed by multi-head self-attention, and the first subsequence mean feature vector is the mean of the image block unit feature vectors carrying subsequence position information processed by multi-head self-attention. For example, a subsequence contains sixteen image patch features and a learnable vector. After multi-head self-attention interaction, the learnable vector contains the comprehensive information of these sixteen patches and becomes the representative of the local region.

[0087] Alternatively, in another optional implementation, instead of using a concatenation method, the learnable unit feature vector and the image patch feature vector are treated as the same sequence. During the attention operation, an attention mask is used to force all image patch features to interact only with the image patch features and the learnable vector. Finally, the learnable vector is extracted as the first subsequence feature vector, and the mean pooling result of the image patch features is used as the mean feature vector of the first subsequence.

[0088] By inserting learnable unit feature vectors and combining them with multi-head self-attention, each subsequence can automatically learn how to extract the most generalized and context-relevant local region representations from its internal image patches, avoiding the information loss caused by simple pooling. Simultaneously, the generated subsequence coordinate labels and mean feature vectors provide comprehensive sub-region features with precise localization and statistical properties for subsequent global spatial modeling, enabling the network to perform higher-level cross-regional spatial reasoning using sub-regions as basic units.

[0089] In some alternative implementations, the network of the local attention submodule can be replaced with engineering-optimized implementations such as shift window attention (Swin Transformer style), linear attention, Flash attention, or Grouped Query Attention.

[0090] S250. Based on the global attention submodule in the multi-layer panoramic spatial attention coding network layer, spatial information encoding is performed on the feature vector of the first subsequence and the coordinate label information of the subsequence to obtain the feature vector of the second subsequence.

[0091] The global attention submodule is the submodule in the multi-layer panoramic spatial attention encoding network that follows the local attention layer. It is responsible for cross-regional spatial information interaction and feature fusion among all subsequences. Spatial information encoding here refers to using the coordinate label information of the subsequences to determine the spatial relationship between the feature vectors of each first subsequence, and performing weighted feature aggregation accordingly, so that each subsequence feature can absorb information from other subsequences in its spatial neighborhood or under specific spatial relationships.

[0092] The second subsequence feature vector can be a subsequence feature vector enhanced by spatial information encoding. It integrates feature information from spatially adjacent regions on the basis of the original first subsequence feature vector, and has the ability to perceive spatial context across local regions.

[0093] The process of obtaining the feature vector of the second subsequence can be as follows: Based on the coordinate labeling information of the subsequence, obtain multiple spatially related feature vectors for each feature vector of the first subsequence. The spatially related feature vectors are the feature vectors of the first subsequence that satisfy a preset spatial positional relationship with the feature vector of the first subsequence. Based on the spatial positional relationship between the feature vector of the first subsequence and the multiple spatially related feature vectors, obtain the attention weights between the feature vector of the first subsequence and the multiple spatially related feature vectors. Based on the attention weights, encode the feature information of the feature vector of the first subsequence and the multiple spatially related feature vectors to obtain the feature vector of the second subsequence.

[0094] The subsequence coordinate labeling information refers to the spatial centroid coordinates of each image patch unit feature vector subsequence generated in S240, representing the two-dimensional spatial position of the subsequence in the original whole-slice image. The first subsequence feature vector is the summary feature representation of each subsequence obtained after processing by the local attention submodule, which is updated from the learnable unit feature vector through multi-head self-attention. The spatial association feature vector refers to the first subsequence feature vector of other subsequences in the surrounding neighborhood of the current subsequence that satisfy preset conditions (such as distance within a threshold, being in adjacent directions, etc.). The preset spatial positional relationship can be a predefined spatial neighborhood rule, such as all subsequences within a preset radius centered on the coordinates of the current subsequence, or four spatially adjacent subsequences: above, below, left, and right.

[0095] For example, a two-dimensional spatial index can be constructed based on the coordinate labeling information of all subsequences. For each first subsequence feature vector, using its coordinates as the center and a preset distance threshold as the radius, all other subsequences falling within the circular neighborhood in the two-dimensional coordinate space are retrieved, and the first subsequence feature vectors corresponding to these subsequences are used as the spatial association feature vectors of the current subsequence. For instance, a subsequence located in the central region of a tumor would select the features of subsequences in its surrounding stroma region and border region as spatial association objects.

[0096] Furthermore, spatial relationships can be geometric relationships between the current subsequence and the subsequences corresponding to each spatially associated feature vector, typically quantified as Euclidean distance, orientation angle, or relative coordinate difference. Attention weights are scalar coefficients determined by spatial relationships or learned, used to measure the importance of each spatially associated feature vector to the features of the current subsequence; spatially associated feature vectors that are closer are usually assigned higher weights. In one implementation, the Euclidean distance between the coordinate label information of the current subsequence and the coordinates of the subsequence corresponding to each spatially associated feature vector is calculated. The reciprocal or negative distance is then normalized using Softmax and used as the attention weight, resulting in closer neighboring subsequences receiving greater attention weights and farther subsequences receiving smaller weights. For example, mesenchymal sequences adjacent to the tumor margin are assigned higher weights, while distant normal tissue subsequences have weights close to zero. Alternatively, the relative coordinate difference between the coordinates of the current subsequence and the coordinates of spatially associated subsequences can be input into a small learnable network. This network outputs the corresponding attention weights, allowing the model to adaptively learn which spatial relationships are more important based on task requirements, rather than relying entirely on a pre-defined distance function.

[0097] The first subsequence feature vector of the current subsequence is weighted and summed with all spatially associated feature vectors according to their respective attention weights. The current subsequence itself is also assigned a retention weight. The resulting weighted sum is the fused vector. This fused vector is either directly used as the second subsequence feature vector, or added to the original first subsequence feature vector as a residual. For example, a subsequence located at the tumor invasion front, after fusing features from adjacent stromal regions and tumor interior regions, can have its second subsequence feature vector simultaneously reflect the spatial distribution characteristics of tumor cell morphology and stromal response. Alternatively, a multi-head attention mechanism can be used for feature information encoding. The first subsequence feature vector is used as the query vector, and all spatially associated feature vectors are used as key and value vectors. The attention weights are obtained by calculating the similarity between the query and the key. The value vectors are then weighted and aggregated, and the updated features are output as the second subsequence feature vector, giving the fusion process richer expressive power.

[0098] Alternatively, all the feature vectors of the first subsequence can be input into a graph neural network based on coordinate distance, with the subsequence coordinate label information as the node position, and a nearest neighbor graph or a complete graph constructed according to the spatial distance. Message passing is performed through a graph attention network, and the updated node features are used as the feature vectors of the second subsequence.

[0099] Spatial information encoding makes each subsequence feature no longer isolated, but selectively integrates features from other spatially related sub-regions, thereby explicitly injecting cross-regional spatial distribution patterns into the features.

[0100] The spatial information encoding in the sub-modules of global attention can be any of the following encoding methods: RankPE (Rank Positional Encoding), Rotary Position Embedding (RoPE), two-dimensional sine / cosine fixed position encoding, or learnable absolute position encoding.

[0101] S260. Based on the global attention submodule, the farthest point sampling compression is performed based on the feature vector of the first subsequence, the mean feature vector of the first subsequence, and the coordinate label information of the subsequence to obtain a first preset number of sampled feature vectors; and multi-head cross-attention operation is performed on the feature vector of the second subsequence and the first preset number of sampled feature vectors to obtain the multi-head cross-attention operation result.

[0102] Farthest Point Sampling (FPS) compression is an iterative sampling algorithm that uniformly selects representative points in space, so that the selected small number of representative points can cover the original data space to the greatest extent, thereby achieving the purpose of compressing the amount of data and preserving the spatial distribution structure.

[0103] The first preset number of sampled feature vectors can be a small number of comprehensive feature vectors obtained after sampling from the farthest point and feature aggregation. Each vector corresponds to a spatial representative center, which aggregates the first subsequence features and mean feature information of the region surrounding the center. The number is much smaller than the number of feature vector subsequences of the original image block unit.

[0104] The first preset number of sampled feature vectors are obtained, which can be achieved by iterative sampling in a two-dimensional coordinate space using a farthest point sampling algorithm, and then filtering to obtain a second preset number of spatially uniformly distributed representative centers; wherein the second preset number is less than the number of image block unit feature vector subsequences; each image block unit feature vector subsequence is assigned to the representative center with the closest spatial distance to it; for each representative center, the first subsequence feature vectors of the image block unit feature vector subsequences associated with each representative center are aggregated by maximizing the maximum value to obtain a second preset number of first sampled feature vectors; and the mean feature vectors of the first subsequences of the image block unit feature vector subsequences associated with each representative center are aggregated by maximizing the maximum value to obtain a second preset number of second sampled feature vectors; the first sampled feature vectors and the second sampled feature vectors are concatenated and fused to obtain the first preset number of sampled feature vectors.

[0105] Specifically, the algorithm can take a two-dimensional point set composed of coordinate markers of all sub-sequences as input, execute the farthest point sampling algorithm, and iteratively select a second preset number of spatially uniform representative centers. Then, the feature vector sub-sequence of each image patch unit is assigned to the nearest representative center. For each representative center, the first sub-sequence feature vectors of all its associated sub-sequences are collected and aggregated element-wise to obtain a first sampled feature vector. Simultaneously, the mean feature vectors of the first sub-sequences of these sub-sequences are also aggregated to obtain a second sampled feature vector. The first and second sampled feature vectors are concatenated and linearly mapped to obtain a first preset number of sampled feature vectors. Subsequently, using these sampled feature vectors as queries and the second sub-sequence feature vectors as keys and values, a multi-head cross-attention operation is performed to obtain the cross-attention result. For example, fifty representative centers can be selected using FPS, each center aggregates features from surrounding sub-regions to form a compressed global anchor feature, and these anchor features interact with all second sub-sequence features to capture the global context.

[0106] Farthest-point sampling compression significantly reduces the number of features while ensuring full spatial coverage, resolving the computational bottleneck caused by the massive number of features in a full-slice image patch, while preserving the representativeness of the entire slice space. Subsequent cross-attention operations allow these compressed global anchors to fully interact with the sub-sequence features carrying spatial information, feeding back the macro-spatial context to each local region feature. This enables each sub-sequence feature to be recalibrated and enhanced within a global perspective, providing a globally consistent information foundation for further aggregation decisions.

[0107] In another alternative implementation, instead of using farthest-point sampling, adaptive sampling based on cluster centers can be employed. A first preset number of learnable query vectors are obtained through learning. These query vectors are then directly cross-attentioned with the feature vectors of the second subsequence, and automatically focused on key spatial regions through training. Alternatively, sampling methods such as K-means cluster center sampling, local grid uniform sampling, random sampling, learnable spatial quantization, or hierarchical nearest neighbor sampling can also be used.

[0108] S270. The result of the multi-head cross-attention operation is superimposed with the feature vector subsequences of multiple image block units in a residual manner to obtain the feature superposition result. The feature superposition result is then processed by a feedforward neural network and a normalization layer to obtain the feature extraction result of the multi-layer panoramic spatial attention coding network layer.

[0109] Among them, the residual method is a connection method that directly adds the output of the network layer to the input, which helps gradient propagation and training deep networks while preserving the original information. Feedforward neural networks are usually multilayer perceptrons composed of fully connected layers and nonlinear activation functions, used to perform nonlinear transformations and further enhance features. Normalization layers: Network layers that normalize features, such as layer normalization, are used to stabilize the training process and accelerate convergence.

[0110] The feature extraction result is the final feature sequence output by the multi-layer panoramic spatial attention coding network layer, where each feature vector corresponds to an original image patch unit, but has been integrated with local and global spatial information and global context.

[0111] For example, the multi-head cross-attention operation result obtained in S260 can be added to each feature vector in the original multiple image block unit feature vector sub-sequence by means of dimension mapping or direct broadcasting, so that global context information is injected into each local image block feature; then the superimposed features are fed into a feedforward neural network consisting of two fully connected layers and activation functions, and after layer normalization processing, the updated feature sequence is finally output as the feature extraction result of the encoding network of this layer.

[0112] Alternatively, the cross-attention result can be residually fused with the feature vector of the first subsequence, and then the updated subsequence features can be mapped back to the feature vectors of each image block unit contained therein, realizing the backpropagation of global information to local features. Finally, the feature sequence is obtained by passing through a feedforward network and a normalization layer.

[0113] By using residual connections, the cross-attention results, which contain global spatial and cross-regional information, are directly superimposed onto the feature vector of each image patch unit. This allows each local feature to obtain a macroscopic view from the entire slice, completely breaking the limitation of isolated local features. The feedforward neural network and normalization layer further refine and stabilize these fused features. The final feature extraction result retains both local details and global spatial awareness, providing extremely rich feature representation for the final feature aggregation.

[0114] S280. Input the feature extraction results of the multi-layer panoramic spatial attention coding network layer into the feature aggregation layer to obtain the image coding result output by the first full-slice image encoder.

[0115] The feature aggregation layer can be a pooling module that combines attention weights and gating mechanisms. It can automatically select key instances that are closely related to the image recognition and classification task from a set of image patch features of varying quantity and importance, and aggregate them into a fixed-dimensional full-slice-level representation.

[0116] The image encoding result is the feature representation vector of the whole slice image finally output by the first whole slice image encoder, which integrates the morphological information, spatial distribution information and key discriminative information of the whole slice.

[0117] For example, all image patch feature vectors in the feature extraction result are treated as a multi-instance set. Each feature vector is processed through an attention branch to calculate its importance score for the image recognition and classification task, and through a gating branch to calculate its information reliability threshold. The importance score and the threshold value are multiplied element-wise to obtain the final weight. This weight is then used to sum all feature vectors to obtain a full-slice feature vector as the image encoding result. For instance, in a slice containing small lesions of tumor, the attention branch will assign high weights to a few tumor region image patches, while the gating branch will suppress the influence of blank background patches, ensuring that the encoding result accurately focuses on the lesion region.

[0118] In another alternative implementation, a grouping aggregation strategy can be added to the gated attention aggregation. First, the features are divided into several groups according to spatial or semantic information and then the attention pooling within each group is performed. Then, the representative features of each group are aggregated globally to obtain the final image encoding result in a hierarchical manner, so as to capture richer multi-scale features that contribute to the image recognition and classification results.

[0119] Gated attention-based aggregation layers selectively focus on the most discriminative tissue regions for image recognition and classification through learnable weights, effectively suppressing irrelevant or noisy regions and extracting high-information-density full-slice representations from large and redundant image patch features. This makes the final image encoding result no longer a mechanical average of all regions, but a compact feature with a clear classification result orientation, providing a high-quality information source for downstream accurate diagnostic decisions.

[0120] In some implementations, the gated attention-based aggregation layer can be replaced with standard ABMIL (Attention-based Multiple Instance Learning), TransMIL (Transformer Multiple Instance Learning), CLAM (Clustering-constrained Attention Multiple Instance Learning), global averaging / max pooling, sequence aggregators based on the Mamba state space model, or multi-granularity aggregation and other feature aggregation methods.

[0121] S290. Based on the image encoding result and the first attribute information of each sample image, the loss is calculated to obtain the learning loss; and based on the learning loss, the parameters of the first full-slice image encoder are updated by the reverse gradient to obtain the second full-slice image encoder.

[0122] The technical solution of this invention involves obtaining a set of pathological whole-section sample images and preprocessing each sample image in the set to obtain the image block pixel sequence and two-dimensional spatial coordinate set of each sample image; inputting the image block pixel sequence and two-dimensional spatial coordinate set into the image block embedding layer of the first whole-section image encoder to obtain the image block unit feature vector sequence output by the image block embedding layer through shared linear projection; grouping the image block unit feature vector sequence output by the image block embedding layer based on the local attention submodule in the multi-layer panoramic spatial attention coding network layer to obtain multiple image block unit feature vector subsequences; inserting a learnable unit feature vector into each image block unit feature vector subsequence, and performing feature calculation on each image block unit feature vector subsequence and learnable unit feature vector based on a multi-head self-attention mechanism to obtain the first subsequence feature vector, the first subsequence mean feature vector, and the subsequence coordinate label information; and using the global attention submodule in the multi-layer panoramic spatial attention coding network layer to process the first subsequence feature vector and the subsequence feature vector. The coordinate marking information is spatially encoded to obtain the second subsequence feature vector. Based on the global attention submodule, the farthest point sampling compression is performed on the first subsequence feature vector, the first subsequence mean feature vector, and the subsequence coordinate marking information to obtain a first preset number of sampled feature vectors. Multi-head cross-attention operation is then performed on the second subsequence feature vector and the first preset number of sampled feature vectors to obtain the multi-head cross-attention operation result. The multi-head cross-attention operation result is superimposed with multiple image block unit feature vector subsequences in a residual manner to obtain the feature superposition result. The feature superposition result is then processed through a feedforward neural network and a normalization layer to obtain the feature extraction result of the multi-layer panoramic spatial attention encoding network layer. The feature extraction result of the multi-layer panoramic spatial attention encoding network layer is input into the feature aggregation layer to obtain the image encoding result output by the first full-slice image encoder. Loss is calculated based on the image encoding result and the first attribute information of each sample image to obtain the learning loss. The parameters of the first full-slice image encoder are then updated based on the learning loss and the backward gradient to obtain the second full-slice image encoder. This solution addresses the spatial information loss problem caused by the independent nature of image patches in traditional whole-slice image encoder training methods for pathological slides. It also fully integrates cross-regional spatial information into the image encoding process, fundamentally eliminating this spatial information loss issue. Furthermore, image classification labels or diagnostic text can be used as supervisory signals for end-to-end optimization of the entire encoder, overcoming the accuracy limitations caused by feature extraction network freezing. This allows the resulting whole-slice image encoder to simultaneously perceive tissue spatial distribution patterns and learn task-specific features, significantly improving the accuracy and clinical reliability of the output results for image information recognition tasks.

[0123] Figure 3This is a flowchart illustrating a whole-slice image encoding method provided in an embodiment of the present invention. This embodiment is applicable to scenarios involving whole-pathological slide image processing. The method can be executed by a whole-slice image encoding device, implemented in software and / or hardware, and integrated into a computer device.

[0124] like Figure 3 As shown, the whole-slice image encoding method includes the following steps: S310. Obtain the first pathological whole slide image.

[0125] The first pathological whole slide image refers to a newly acquired digital pathological whole slide image that requires pathological analysis in clinical diagnosis or practical application. This image is also an ultra-high resolution image generated by completely digitizing glass pathological slides using a high-powered scanner, with a single image size reaching the level of 100,000 by 100,000 pixels, and a resolution far exceeding the direct processing capabilities of conventional neural networks.

[0126] Obtaining the first whole-slide image can be achieved by scanning the patient's glass slides for diagnosis using a slide scanner equipped in the hospital's pathology department, directly generating a digitized first whole-slide image, and then transmitting it to an analysis workstation equipped with a trained whole-slide image encoder. Alternatively, it can be obtained from the hospital's pathology image archiving and communication system by querying and retrieving previously scanned and stored first whole-slide image files based on patient identification and examination number, and loading them into computer memory for subsequent processing. Another option is to read the first whole-slide image file from local storage devices or cloud storage; this image may be case data transmitted via network from other collaborating hospitals or external scanning centers for analysis.

[0127] By directly obtaining ultra-high resolution first pathological whole slice images, the entire fine structure of the tissue slices can be completely preserved, avoiding information loss caused by any lossy compression or downsampling. This provides original information assurance for the subsequent whole slice image encoder to extract high-quality features that are valuable for image recognition and classification results.

[0128] S320. Preprocess the first pathological whole slice image to obtain the first image block pixel sequence and the first coordinate set.

[0129] The first coordinate set is the set of two-dimensional spatial coordinates corresponding to the pixel sequence of the first image block.

[0130] Preprocessing the first pathological whole slice image can be performed on the first pathological whole slice image by performing the same tissue region recognition and image segmentation operations as in the training phase, decomposing the huge image into a large number of smaller image blocks, and simultaneously recording the spatial position of each image block in the original image.

[0131] The first image block pixel sequence is a sequence of pixel data formed by arranging all the small image blocks obtained after segmenting the first pathological whole slide image in a certain spatial order. The first coordinate set records the two-dimensional spatial coordinates of each image block in the first image block pixel sequence on the original first pathological whole slide image, for example, represented by the row and column number of the top left corner of the image block.

[0132] In one specific embodiment, preprocessing the first pathological whole-slice image can involve first detecting tissue regions and excluding blank backgrounds, then dividing the effective tissue region into fixed-size, non-overlapping grids, and sequentially cropping image blocks from the top left to the bottom right. The cropped image blocks are then arranged in the scanning order to obtain the first image block pixel sequence, while the top-left row and column coordinates of each image block are recorded as the first coordinate set. Another implementation method uses the same sliding window parameters as in the training phase, extracting overlapping image blocks in the tissue region with adjacent blocks maintaining a fixed overlap ratio. All image blocks are arranged in the sliding window traversal order to form the first image block pixel sequence, and the center point or top-left corner coordinates of each image block are recorded as the first coordinate set to maintain spatial continuity consistent with the training phase.

[0133] This step transforms the ultra-high resolution first pathological whole-slice image into an image patch sequence and spatial coordinate set that are completely consistent with the format of the training phase. This ensures that the input data structure in the application phase is strictly aligned with that in the training phase, guaranteeing that the trained whole-slice image encoder can seamlessly receive and correctly process the data. Simultaneously, by explicitly preserving the two-dimensional coordinates of each image patch, the necessary spatial location information is provided for the spatial attention mechanism built into the whole-slice image encoder to exert its cross-regional contextual awareness capability. This allows the encoder to effectively capture spatial features such as tumor invasion patterns and immune cell distribution during inference.

[0134] S330. Input the pixel sequence of the first image block and the first coordinate set into the first full-slice image encoder to obtain the first image encoding result output by the first full-slice image encoder.

[0135] The first full-slice image encoder is an encoder obtained by the full-slice image encoder training method of any of the above embodiments.

[0136] The first full-slice image encoder can refer to a full-slice image encoder model that has completed all training steps and fixed parameters. It contains an image patch embedding layer, a multi-layer panoramic spatial attention coding network layer, and a feature aggregation layer. It has been trained end-to-end and has cross-regional spatial perception capability and task-specific feature extraction capability.

[0137] The first image encoding result is a full-slice-level feature vector output by the first full-slice image encoder after performing forward computation on the pixel sequence of the first image block and the first coordinate set. This vector encapsulates the local morphological details of the entire slice, cross-regional spatial distribution information, and discriminative features highly correlated with subsequent image recognition tasks. The subsequent image recognition task can also be a training objective consistent with that of the first full-slice image encoder during training.

[0138] The process of feature encoding of the first full-slice image encoder for the first image block pixel sequence and the first coordinate set can be as follows: each image block in the first image block pixel sequence is paired with its corresponding spatial coordinate in the first coordinate set and input into the first full-slice image encoder. Internally, the encoder first uses an image block embedding layer to map each image block into a feature vector. Then, a local attention module in a multi-layer panoramic spatial attention coding network group the features based on the coordinates and extracts local sub-sequence features. Next, a global attention module combines spatial information encoding to perform cross-regional feature interaction and compression. Finally, a gated attention aggregation layer aggregates all image block features into the first image encoding result, completing one forward inference. Alternatively, the first full-slice image encoder can be deployed as an online inference service. It receives the first image block pixel sequence and the first coordinate set sent from the front end via an interface. After completing the forward computation on the server side, it returns the first image encoding result to the caller for direct use by downstream classifiers or retrieval systems.

[0139] The technical solution of this invention involves obtaining a first pathological whole-slice image; preprocessing the first pathological whole-slice image to obtain a first image block pixel sequence and a first coordinate set, wherein the first coordinate set is a two-dimensional spatial coordinate set corresponding to the first image block pixel sequence; inputting the first image block pixel sequence and the first coordinate set into a first whole-slice image encoder to obtain a first image encoding result output by the first whole-slice image encoder, wherein the first whole-slice image encoder is an encoder obtained through the whole-slice image encoder training method of any embodiment. That is, this application solution utilizes a whole-slice image encoder that has completed end-to-end training and possesses spatial awareness capabilities to directly encode the newly obtained pathological whole-slice image, obtaining a whole-slice-level representation that integrates local morphological features and cross-regional spatial distribution patterns in one go, eliminating the spatial information loss caused by the independent processing of image blocks in traditional methods. The image encoding result output by the whole-slice image encoder can be applied to image recognition combined with clinical needs, significantly improving the accuracy and application efficiency of pathological image recognition.

[0140] Figure 4This is a schematic diagram of the structure of a whole-slice image encoder training device provided in an embodiment of the present invention. This embodiment is applicable to end-to-end pathological whole-slice image encoder training scenarios. The device can be implemented by software and / or hardware and integrated into a computer device.

[0141] like Figure 4 As shown, the whole-slice image encoder training device includes: a sample preprocessing module 410, a sample input module 420, a loss calculation module 430, and a model training and update module 440.

[0142] The sample preprocessing module 410 is used to obtain a set of pathological whole-slice sample images and preprocess each sample image in the set to obtain the image block pixel sequence and two-dimensional spatial coordinate set of each sample image; the sample input module 420 is used to input the image block pixel sequence and two-dimensional spatial coordinate set into the first whole-slice image encoder to obtain the image encoding result output by the first whole-slice image encoder; the loss calculation module 430 is used to calculate the learning loss based on the image encoding result and the first attribute information of each sample image; wherein, the first attribute information is at least one of the classification label information and pathological diagnosis text information corresponding to each sample image; the training update module 440 is used to update the parameters of the first whole-slice image encoder based on the backward gradient of the learning loss to obtain the second whole-slice image encoder.

[0143] The technical solution of this invention involves obtaining a set of whole-slice pathological sample images and preprocessing each sample image in the set to obtain the image block pixel sequence and two-dimensional spatial coordinate set of each sample image. The image block pixel sequence and two-dimensional spatial coordinate set are then input into a first whole-slice image encoder to obtain the image encoding result output by the first whole-slice image encoder. A loss calculation is performed based on the image encoding result and the first attribute information of each sample image to obtain a learning loss. The first attribute information includes at least one of the classification label information and pathological diagnosis text information corresponding to each sample image. The parameters of the first whole-slice image encoder are updated based on the backward gradient of the learning loss to obtain a second whole-slice image encoder. This technical solution solves the problems of cross-regional spatial information loss and the inherent upper limit of accuracy in classification based on encoded features in existing whole-slice image encoder separation training methods. By enabling the trained whole-slice image encoder to perceive the spatial distribution patterns of tissues and learn task-specific features, it significantly improves the accuracy and reliability of challenging pathological image recognition tasks such as tumor subtype-related feature information recognition and gene mutation prediction.

[0144] In one alternative implementation, the first full-slice image encoder includes an image patch embedding layer, a multi-layer panoramic spatial attention coding network layer, and a feature aggregation layer.

[0145] In one alternative implementation, the sample input module 420 is specifically used for: The image block pixel sequence and the set of two-dimensional spatial coordinates are input into the image block embedding layer in the first full-slice image encoder to obtain the image block unit feature vector sequence output by the shared linear projection of the image block embedding layer; The image patch unit feature vector sequence and the two-dimensional spatial coordinate set are input into a multi-layer panoramic spatial attention coding network layer to obtain multiple image patch unit feature vector sub-sequences and feature vector sequences extracted by the local attention sub-module and global attention sub-module concatenated in the multi-layer panoramic spatial attention coding network layer; wherein, the image patch unit feature vector sub-sequence is obtained by grouping the image patch unit feature vector sequence, and each image patch unit feature vector sub-sequence contains the same number of image patch unit feature vectors. The feature vector subsequences of multiple image block units are input into the feature aggregation layer to obtain the image encoding result after feature aggregation.

[0146] In an alternative implementation, the sample input module 420 can also be used for: Based on the local attention submodule in the multi-layer panoramic spatial attention coding network, the image block unit feature vector sequence output by the image block embedding layer is grouped to obtain multiple image block unit feature vector subsequences. For each image block unit feature vector subsequence, a learnable unit feature vector is inserted. Based on the multi-head self-attention mechanism, feature calculation is performed on each image block unit feature vector subsequence and learnable unit feature vector to obtain the first subsequence feature vector, the first subsequence mean feature vector, and the subsequence coordinate label information. Based on the global attention submodule in the multi-layer panoramic spatial attention coding network, spatial information encoding is performed on the feature vector of the first subsequence and the coordinate labeling information of the subsequence to obtain the feature vector of the second subsequence. Based on the global attention submodule, the farthest point sampling compression is performed on the feature vector of the first subsequence, the mean feature vector of the first subsequence, and the coordinate label information of the subsequence to obtain a first preset number of sampled feature vectors; and multi-head cross-attention operation is performed on the feature vector of the second subsequence and the first preset number of sampled feature vectors to obtain the multi-head cross-attention operation result; The result of the multi-head cross-attention operation is superimposed on the feature vector subsequences of multiple image block units in a residual manner to obtain the feature superposition result. The feature superposition result is then processed by a feedforward neural network and a normalization layer to obtain the feature extraction result of the multi-layer panoramic spatial attention coding network layer.

[0147] In an alternative implementation, the sample input module 420 can also be used for: Based on the spatial coordinates in the two-dimensional spatial coordinate set, the image block unit feature vector sequence is grouped to obtain multiple image block unit feature vector sub-sequences; In this subsequence, each image block corresponding to the feature vector of an image block unit constitutes a sample image sub-region with the same length and width.

[0148] In an alternative implementation, the sample input module 420 can also be used for: The relative position information of each image block unit feature vector in the corresponding image block unit feature vector subsequence is superimposed onto each image block unit feature vector to obtain an image block unit feature vector carrying the subsequence position information; For each image block unit feature vector subsequence, the unit feature vector can be learned and concatenated with the corresponding image block unit feature vector carrying the subsequence position information to form a joint feature sequence; Perform multi-head self-attention operation on the joint feature sequence to obtain the first subsequence feature vector and the first subsequence mean feature vector; and use the centroid of the spatial coordinates of the image block unit feature vector subsequence corresponding to the joint feature sequence as the subsequence coordinate labeling information. Wherein, the first subsequence feature vector is the learnable unit feature vector after multi-head self-attention operation, and the first subsequence mean feature vector is the mean of the image block unit feature vector carrying subsequence position information after multi-head self-attention operation.

[0149] In an alternative implementation, the sample input module 420 can also be used for: Based on the subsequence coordinate labeling information, multiple spatially related feature vectors are obtained for each first subsequence feature vector. The spatially related feature vector is the first subsequence feature vector whose spatial positional relationship with the first subsequence feature vector satisfies a preset spatial positional relationship. Based on the spatial positional relationship between the first subsequence feature vector and multiple spatially related feature vectors, the attention weights between the first subsequence feature vector and multiple spatially related feature vectors are obtained; Based on attention weights, feature information is encoded between the feature vector of the first subsequence and multiple spatially related feature vectors to obtain the feature vector of the second subsequence.

[0150] In an alternative implementation, the sample input module 420 can also be used for: The farthest point sampling algorithm is used to iteratively sample in the two-dimensional coordinate space to select a second preset number of spatially uniform representative centers; where the second preset number is less than the number of feature vector sub-sequences of image block units; Each image block unit feature vector subsequence is assigned to the representative center that is spatially closest to it; For each representative center, the first subsequence feature vector of the image block unit feature vector subsequence associated with each representative center is aggregated by the maximum value to obtain a second preset number of first sampled feature vectors; and the mean feature vector of the first subsequence of the image block unit feature vector subsequence associated with each representative center is aggregated by the maximum value to obtain a second preset number of second sampled feature vectors. The first sampled feature vector and the second sampled feature vector are concatenated and fused to obtain a first preset number of sampled feature vectors.

[0151] In one alternative implementation, the loss calculation module 430 is specifically used for: The image encoding result is input into the pre-trained classification prediction head to obtain the classification prediction result of each sample image. Based on the classification prediction result and classification label information, the cross-entropy loss is obtained and used as the learning loss. Alternatively, the pathological diagnosis text information can be encoded to obtain a text embedding vector, and the information noise contrast estimation loss can be obtained based on the text embedding vector and the image encoding result, and the information noise contrast estimation loss can be used as the learning loss.

[0152] The full-slice image encoder training device provided in this embodiment of the invention can execute the full-slice image encoder training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0153] Figure 5 This is a schematic diagram of the structure of a whole-slice image encoding device provided in an embodiment of the present invention. This embodiment has the same inventive concept as the above embodiments and can be applied to scenarios involving whole-slice image processing of medical cases. The device can be implemented in software and / or hardware and integrated into a computer device.

[0154] like Figure 5 As shown, the full-slice image encoding device includes: an image acquisition module 510 to be encoded, an image preprocessing module 520, and an image encoding module 530.

[0155] The image acquisition module 510 is used to acquire a first pathological whole slide image; the image preprocessing module 520 is used to preprocess the first pathological whole slide image to obtain a first image block pixel sequence and a first coordinate set, wherein the first coordinate set is a two-dimensional spatial coordinate set corresponding to the first image block pixel sequence; the image encoding module 530 is used to input the first image block pixel sequence and the first coordinate set into a first whole slide image encoder to obtain a first image encoding result output by the first whole slide image encoder, wherein the first whole slide image encoder is an encoder obtained by the whole slide image encoder training method of any embodiment.

[0156] The technical solution of this invention involves obtaining a first pathological whole-slice image; preprocessing the first pathological whole-slice image to obtain a first image block pixel sequence and a first coordinate set, wherein the first coordinate set is a two-dimensional spatial coordinate set corresponding to the first image block pixel sequence; inputting the first image block pixel sequence and the first coordinate set into a first whole-slice image encoder to obtain a first image encoding result output by the first whole-slice image encoder, wherein the first whole-slice image encoder is an encoder obtained through the whole-slice image encoder training method of any embodiment. That is, this application solution utilizes a whole-slice image encoder that has completed end-to-end training and possesses spatial awareness capabilities to directly encode the newly obtained pathological whole-slice image, obtaining a whole-slice-level representation that integrates local morphological features and cross-regional spatial distribution patterns in one go, eliminating the spatial information loss caused by the independent processing of image blocks in traditional methods. The image encoding result output by the whole-slice image encoder can be applied to image recognition combined with clinical needs, significantly improving the accuracy and application efficiency of pathological image recognition.

[0157] The full-slice image encoding device provided in this embodiment of the invention can execute the full-slice image encoding method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0158] Furthermore, this specification also provides a specific application example of the above-described whole-slice image encoder training method.

[0159] Specifically, the training process of the whole-slice image encoder is divided into three steps, such as... Figure 6 As shown.

[0160] Step 1: Preprocess the input WSI, sequentially performing effective region selection, color normalization, and patch partitioning, outputting N image patch pixel sequences and their two-dimensional spatial coordinate sets for use in Step 2. Step 1 does not contain any trainable parameters.

[0161] Step 2: Using the image patch pixel sequence and coordinate set output from Step 1 as input, features are extracted layer by layer through the image patch embedding layer and the pathological image coding network (PanoPath panoramic pathological neural network) composed of an L-layer panoramic spatial attention module (Panoramic Spatial Attention, PanoAttn). The features are then aggregated into slice-level features by the aggregation module, and the image recognition classification prediction results are output for Step 3 to calculate the loss.

[0162] Step 3: Based on the preset training objective, calculate the differentiable loss function using the slice-level features and prediction results output in Step 2 as input. Backpropagate the loss gradient to the pixel layer of the image patch throughout the process, and synchronously update all trainable parameters. Repeat Steps 2 and 3 iteratively until the model converges.

[0163] In step one, the input pathological WSI is used to sequentially complete the selection of effective regions, staining normalization, patch division, and coordinate recording.

[0164] Specifically, effective region selection involves segmenting the tissue region of the low-resolution WSI thumbnail using an image segmentation model, automatically identifying and removing background blank areas, and retaining only the effective region containing actual tissue. Color normalization can be performed using color normalization methods such as Macenko to normalize the effective tissue region, eliminating color differences between different production batches and coloring conditions. Patch partitioning and coordinate recording can be performed by sliding the effective tissue region at a fixed size p×p, generating N non-overlapping image patches. The flattened RGB pixels of the i-th image patch are denoted as xi. Simultaneously, the two-dimensional spatial coordinates ci=(ui, vi) of each image patch in the original WSI coordinate system are recorded and summarized into a coordinate set C=[c1,…,cN]. Step one does not contain any trainable parameters; the output {xi} and C are used as inputs for step two.

[0165] Step two involves feature extraction and aggregation based on the panoramic attention mechanism. The overall architecture of the Panoramic Pathology Image Encoder (PanoPath) can be referenced as follows: Figure 7 The structure shown.

[0166] Figure 7 The network structure takes the N image patch pixel sequences {xi} and coordinate set C output from step one as input, and consists of three parts in sequence: image patch embedding layer, L-layer panoramic spatial attention coding network (PanoAttn), and Gated ABMIL aggregation module.

[0167] The image patch embedding layer first maps each image patch xi to a d-dimensional token vector through a trainable shared linear projection. The specific mapping process can be represented by the following mapping formula: .

[0168] This allows the entire WSI to be transformed into an initial token sequence Z of length N and dimension d. （0） We and be are trainable parameters, and their gradients can be backpropagated to the pixel layer of the image block throughout step three. （0） The coordinate set C, along with the input coordinate set C, is then fed into an encoding network consisting of L layers of PanoAttn modules for layer-by-layer processing. Each PanoAttn layer consists of two sub-modules, local attention and global attention, connected in series, supplemented by a feed-forward network (FFN), residual connections, and layer normalization. The coordinate set C is read in each layer for spatial location calculation. After processing by the L layers, the final patch-level token features are fed into the Gated ABMIL aggregation module to be summarized into slice-level features. The final output is the image recognition classification prediction result, which includes both slice-level and patch-level features.

[0169] Specifically, the PanoAttn spatial attention module is the core module of the PanoPath WSI image encoding network in step two. Each layer consists of two sub-modules, local attention and global attention, connected in series. They are responsible for capturing fine-grained cell morphology features within the window and spatial context information across the entire slice, respectively. It is also supplemented by a feedforward neural network (FFN), residual connections, and layer normalization, as detailed below. Figure 8 As shown.

[0170] Furthermore, the local attention submodule uses the current layer input token sequence Z （l） (N tokens) and the coordinate set C are the inputs. First, group the N tokens into G = N / w groups according to their spatial coordinates. 2 Each window (g, each of size w×w) introduces one learnable Nexus Token feature s. g (corresponding to the learnable unit feature vector in the aforementioned embodiments), s g With the window inside w 2 The image patch tokens are concatenated into a joint sequence, and the centroid of the coordinates of the tokens contained in the window is used as the window's representative coordinate p. g The diagrams illustrating window division and image blocks within a single window can be found in [reference needed]. Figure 9 .

[0171] Subsequently, the learnable absolute positional encoding within the window is superimposed onto each token, enabling it to carry positional information within the window (for the token at position (i,j) within the window, the learnable vector corresponding to its position is uniformly superimposed; this positional encoding is defined on a w×w grid, and all windows share the same positional encoding table). Multi-head self-attention with shared weights is performed among the tokens within the window, allowing for full interaction and fusion of fine-grained pathological information. Within the window, Nexus tokens automatically focus on the region within the window that contributes the most to image recognition through attention weighting. Simultaneously, each window computes in parallel without interference, and the computational load is linearly related to the total number of tokens N.

[0172] Each window outputs two complementary descriptive features, one of which is the Nexus Token. g This feature includes attention-weighted aggregation of the most salient cell morphology and pathological structure information within the window. Another is the mean-descriptive feature. This feature can be obtained by averaging all non-Nexus tokens after local self-attention update, which can preserve overall morphological distribution statistics (such as low-significance features like matrix proportion and inflammatory infiltration density). The two features are complementary: Nexus tokens are responsible for high-significance lesion information, and the mean feature is responsible for low-significance background distribution. Together, they constitute the local pathological information description of this window area and are passed into the global attention submodule.

[0173] The global attention submodule outputs two descriptive features {(s) from the G windows output by the local attention submodule. g , )} and window representing coordinates {p g Taking} as input, cross-regional full-slice information fusion is achieved through three progressive steps: RankPE spatial information encoding, farthest point sampling compression (FPS), and global cross-attention fusion. The final output is the updated features S2={s} corresponding to G windows. g 2}, that is Figure 8 The updated hub image block unit feature vector sequence S.

[0174] Wherein, the RankPE spatial information encoding is the Nexus Token sequence feature S={s} output by the local attention step. g} and window represent coordinates {p g} as input, the execution space adjacent context is fused to obtain the updated Nexus Token sequence S1={s g 1}, so that the S1 feature carries the context information of the spatial neighborhood before participating in the global calculation.

[0175] Specifically, it can be based on the coordinates represented by the window {p g For each window, find its nearest neighboring windows in spatial location, assign weights according to distance (closer windows have higher weights, decreasing exponentially), and then sum the weighted NexusToken features of the neighboring windows and add them to the NexusToken feature s of the current window. g The updated Nexus Token features for each window are obtained. g 1.

[0176] Furthermore, it could be based on the Nexus Token sequence feature S={s g}, based on the coordinates represented by the window {p g Convert the spatial distance between windows into rankings: .

[0177] in, r is the spatial offset between query window g and another window h. gh This represents the number of windows closer to window g than window h, with the nearest neighbor ranking being 0, and monotonically increasing outwards. After obtaining the ordinal ranking of each window pair, each attention head t calculates a ranking decay weight for the window pair (g, h): .

[0178] Where t is each attention head, λ t u is the learnable rank decay rate per head. t Let G be the learnable self-weight for each head, and G be the total number of windows. The weights increase with ranking r. gh As the distance increases, the weight decreases linearly; for windows with larger rankings, the weight decreases at an independent rate λt for each head, while the weight u... t The degree to which each Nexus Token retains its own information is controlled, and then all weights are Softmax normalized to obtain the final attention weights. Each attention head uses these weights to weighted aggregate adjacent Nexus Tokens, generating position-aware residuals that are then superimposed on each attention head. g Generate s g 1. This ensures that the spatial neighborhood context information is carried before entering the global cross attention.

[0179] RankPE relies solely on distance ranking rather than the original coordinate values, remaining invariant to arbitrary translations, rotations, and scaling of the WSI, adapting to any placement orientation and scan resolution of clinical slides. The generated {s g 1} will be used as the query for global cross-attention.

[0180] Farthest Point Sampling Compression (FPS) uses the representative coordinates {p} of all G windows output by the local attention step. g} and two-way descriptive features {(s g , Taking )} as input, the farthest point sampling algorithm iteratively selects m spatially uniformly distributed representative centers from the coordinate space (m is a preset constant, much smaller than G), ensuring coverage of each region. After assigning each window to the nearest representative center, in the Nexus Token branch feature {s g} and mean descriptor sub-branch features { The maximum value aggregation is performed on each of the}, resulting in m token features. After concatenation, a total of 2m global representative context token features are obtained. The overall global branch computation is linearly related to the total number of tokens N.

[0181] Next, a global cross-attention fusion calculation is performed. Using the G Nexus Token sequence features S1 processed by RankPE as the query and the 2m global representative context Token features generated by FPS as the key and value, a multi-head cross-attention operation is executed. The output is superimposed onto the Nexus Token sequence features S generated by local attention in a residual manner. Then, FFN and layer normalization are used to complete the final feature update of the current layer, generating the Nexus Token sequence features S2. Each window perceives the organizational distribution information of each region of the full slice through this step. After being stacked through L layers, it simultaneously carries the fine-grained morphological features within the block and the spatial context of the full slice across regions, and is then input into the aggregation module.

[0182] Therefore, {s} can be represented by G final Nexus Token features after L-layer PanoAttn encoding. g L} is used as input, and the Gated ABMIL module adaptively calculates the aggregation weight for each Nexus Token feature. After weighted summation, a slice-level feature vector is obtained, which is then mapped to the recognition and classification prediction result by the image recognition classification head, and used for loss calculation in step three.

[0183] Furthermore, the loss calculation and end-to-end parameter update in step three use the slice-level features and prediction results output in step two as input, and calculate the corresponding differentiable loss function based on the preset training objective. The specific form is not restricted. For supervised classification tasks (tumor subtype classification, tumor pathological grading, gene mutation prediction, etc.), cross-entropy loss can be used; for image-text comparison pre-training tasks, contrastive learning loss such as InfoNCE (Information Noise-Contrastive Estimation) can be used, with the similarity between slice-level features and corresponding pathology report text features as the optimization objective; for knowledge distillation tasks, feature alignment loss or KL divergence loss can be used; when the training objective includes multiple tasks, the total loss is the weighted sum of the losses of each task, and the weights can be flexibly adjusted according to the importance of the task and the training stage. All of the above loss functions satisfy the differentiability condition, and the gradients can be backpropagated along the entire path of this step. Through the backpropagation algorithm, the loss gradient is propagated along the following path without any breakpoints or frozen layers: Image recognition classification head → Gated ABMIL aggregation module → PanoAttn multilayer coding network (L layer) → Image patch embedding layer → Image patch pixel layer.

[0184] During parameter updates, the AdamW optimizer can be used to update all trainable parameters. Then, steps two and three are repeated iteratively until the model's performance on the target task converges to a stable level, completing the entire end-to-end training process. The end-to-end gradient continuity fundamentally solves the accuracy ceiling problem caused by parameter freezing and gradient breakage in the traditional two-stage paradigm. This allows the feature extraction network to continuously update based on the loss signal of the training target in each iteration, learning task-specific feature representations.

[0185] The training process of the aforementioned whole-slice image encoder merges feature extraction and image recognition classification into a single end-to-end training stage. The loss gradient is propagated along the entire path without any breakpoints or frozen layers. The feature extraction network is continuously updated in each iteration based on the image classification loss signal, learning specialized feature representations for specific image recognition classification tasks, fundamentally breaking through the accuracy limit of the two-stage paradigm. Furthermore, the PanoAttn module in the whole-slice image encoder gathers fine-grained cell morphology information within the window through local attention, and fuses the cross-regional whole-slice spatial context through RankPE and FPS global cross-attention. This ensures that each image block feature fully perceives the spatial neighborhood context and the overall spatial distribution of the slice, fundamentally compensating for the deficiency of cross-regional spatial information loss in existing methods. The Nexus Token and the mean descriptor are complementary, taking into account both high and low significance pathological information. The learnable Nexus Token within each local window automatically focuses on the region most significant for image classification and recognition, gathering high-significance lesion information; the mean descriptor retains low-significance background distribution statistics such as matrix proportion and inflammatory infiltration density. Two complementary features are fed into the global attention, enhancing the model's ability to cover information from different pathological patterns. RankPE possesses translation, rotation, and scaling invariance. RankPE transforms the spatial distance between windows into ordinal rankings, relying solely on distance rankings rather than the original absolute coordinate values. It remains invariant to arbitrary translations, rotations, and scaling of the WSI, adapting to arbitrary placement orientations of clinical slides and various scanner resolutions, exhibiting stronger cross-device generalization capabilities. The overall computational complexity is O(N), significantly reducing GPU memory requirements and making end-to-end training of ultra-large WSIs feasible. PanoAttn reduces the overall computational load to O(N) and significantly lowers the GPU memory requirements by using local window parallel computation (O(N) within each window) and global FPS compression. At the typical scale of WSI, it saves more than 1,000 times the computational and GPU memory overhead compared to global self-attention, eliminating the hardware bottleneck of ultra-high resolution WSI end-to-end encoder training and making it feasible on standard multi-GPU servers.

[0186] Moreover, the loss function is not restricted in form and supports multiple training paradigms. Within the same framework, no modification to the network architecture is required, and it can directly support supervised classification (cross-entropy loss), image-text comparison pre-training (InfoNCE loss), knowledge distillation (feature alignment or KL divergence loss), and multi-task joint training. The weights can be flexibly adjusted, making it widely applicable to various tasks.

[0187] In clinical applications, the image encoding results output by the full-slice image encoder trained in this embodiment significantly improve the accuracy of image recognition in the scenario of fine classification of tumor subtypes. For example, in tumor subtype classification tasks such as image recognition of lung adenocarcinoma and lung squamous cell carcinoma, and classification of breast cancer subtypes, end-to-end gradient backpropagation enables the feature extraction network to continuously optimize towards the classification target, learning specialized discriminative features for subtype identification. Compared with the two-stage frozen feature baseline method, it shows significant improvements in accuracy and other indicators. Its advantages are particularly prominent in subtype identification tasks with subtle morphological differences and reliance on high-level semantic features, effectively reducing the risk of clinical misdiagnosis. Furthermore, the image encoding results output by the full-slice image encoder trained in this embodiment also improve the accuracy of full-slice image recognition in cases such as tumor microenvironment analysis and tumor grading. Tumor grading and tumor microenvironment analysis are highly dependent on cross-regional spatial features such as the spatial continuity of tumor invasion patterns and the distribution patterns of immune cell regions. By explicitly modeling the aforementioned cross-regional spatial structure through a panoramic spatial attention mechanism, the feature encoding process can fully perceive the spatial tissue distribution pattern of the entire slice. Compared with existing methods that process each image patch independently, this method outperforms existing methods in tumor grading accuracy and tumor microenvironment-related analysis tasks.

[0188] Furthermore, the model trained in this embodiment can be directly used as an auxiliary tool for whole-slice pathology image recognition, outputting image recognition prediction results for tasks such as tumor subtype classification and pathological grading, thereby assisting pathologists in making diagnostic decisions. Simultaneously, the aggregation module can automatically generate an image recognition attention heatmap on the original WSI, highlighting high-risk lesion areas on which the model's judgment is based, achieving spatial interpretability of the image recognition results, helping doctors understand and verify the basis of AI (Artificial Intelligence) predictions, and enhancing clinical credibility and acceptability. AI-assisted medical image recognition tools can be applied to AI platforms, remote pathology systems, and other related analysis systems.

[0189] The aforementioned model framework can cover a variety of clinical medical image processing and recognition scenarios, possessing broad clinical applicability. Within the same framework, without modifying the network architecture, it can be directly applied to various clinical medical image recognition scenarios such as tumor subtype classification, tumor pathological grading, gene mutation prediction, survival prediction, and image-text comparison pre-training. It covers multiple training paradigms including supervised learning, self-supervised pre-training, and knowledge distillation, serving as a general-purpose WSI intelligent analysis algorithm engine with broad clinical task applicability and promotional value.

[0190] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 10 A block diagram of an exemplary computer device 12 suitable for implementing embodiments of the present invention is shown. Figure 10The computer device 12 shown is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the present invention. The computer device 12 can be any terminal device with computing capabilities, such as intelligent controllers and servers, mobile phones, and other terminal devices.

[0191] like Figure 10 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0192] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0193] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0194] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 10 Not shown; usually referred to as a "hard drive"). Although Figure 10Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a Compact Disc Read-Only Memory CD-ROM, a Digital Video Disc Read-Only Memory DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0195] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0196] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the computer device 12, and / or with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although... Figure 10 As not shown, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID (Redundant Array of Independent Disks) systems, tape drives, and data backup storage systems.

[0197] Processing unit 16 executes various functional applications and data processing by running programs stored in system memory 28, such as implementing the full-slice image encoder training method provided in any embodiment of the present invention, which includes: A set of whole pathological slide sample images is obtained, and each sample image in the set is preprocessed to obtain the image block pixel sequence and two-dimensional spatial coordinate set of each sample image; The image block pixel sequence and the set of two-dimensional spatial coordinates are input into the first full-slice image encoder to obtain the image encoding result output by the first full-slice image encoder; The learning loss is calculated based on the image encoding results and the first attribute information of each sample image; wherein, the first attribute information is at least one of the classification label information and pathological diagnosis text information corresponding to each sample image. The parameters of the first full-slice image encoder are updated based on the backward gradient of the learning loss to obtain the second full-slice image encoder.

[0198] Alternatively, the processing unit 16 executes various functional applications and data processing by running a program stored in the system memory 28, such as implementing the full-slice image encoding method provided in any embodiment of the present invention, which includes: Obtain the first pathological whole slide image; The first pathological whole slice image is preprocessed to obtain the first image block pixel sequence and the first coordinate set, wherein the first coordinate set is the two-dimensional spatial coordinate set corresponding to the first image block pixel sequence; The first image block pixel sequence and the first coordinate set are input into the first full-slice image encoder to obtain the first image encoding result output by the first full-slice image encoder. The first full-slice image encoder is an encoder obtained by the full-slice image encoder training method of any embodiment.

[0199] This embodiment provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the program implements the full-slice image encoder training method as provided in any embodiment of the present invention. The method includes: A set of whole pathological slide sample images is obtained, and each sample image in the set is preprocessed to obtain the image block pixel sequence and two-dimensional spatial coordinate set of each sample image; The image block pixel sequence and the set of two-dimensional spatial coordinates are input into the first full-slice image encoder to obtain the image encoding result output by the first full-slice image encoder; The learning loss is calculated based on the image encoding results and the first attribute information of each sample image; wherein, the first attribute information is at least one of the classification label information and pathological diagnosis text information corresponding to each sample image. The parameters of the first full-slice image encoder are updated based on the backward gradient of the learning loss to obtain the second full-slice image encoder.

[0200] Alternatively, when the program is executed by a processor, it implements the full-slice image encoding method as provided in any embodiment of the present invention, the method comprising: Obtain the first pathological whole slide image; The first pathological whole slice image is preprocessed to obtain the first image block pixel sequence and the first coordinate set, wherein the first coordinate set is the two-dimensional spatial coordinate set corresponding to the first image block pixel sequence; The first image block pixel sequence and the first coordinate set are input into the first full-slice image encoder to obtain the first image encoding result output by the first full-slice image encoder. The first full-slice image encoder is an encoder obtained by the full-slice image encoder training method of any embodiment.

[0201] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0202] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0203] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF (Radio Frequency), etc., or any suitable combination thereof.

[0204] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0205] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0206] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the full-slice image encoder training method or the full-slice image encoding method provided in any embodiment of this application.

[0207] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0208] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for training a full-slice image encoder, characterized in that, include: A set of whole pathological slide sample images is obtained, and each sample image in the set of whole pathological slide sample images is preprocessed to obtain the image block pixel sequence and two-dimensional spatial coordinate set of each sample image; The image block pixel sequence and the two-dimensional spatial coordinate set are input into the first full-slice image encoder to obtain the image encoding result output by the first full-slice image encoder; The learning loss is calculated based on the image encoding result and the first attribute information of each sample image; wherein, the first attribute information is at least one of the classification label information and pathological diagnosis text information corresponding to each sample image; The parameters of the first full-slice image encoder are updated based on the backward gradient of the learning loss to obtain the second full-slice image encoder.

2. The method according to claim 1, characterized in that, in, The first full-slice image encoder includes an image patch embedding layer, a multi-layer panoramic spatial attention coding network layer, and a feature aggregation layer.

3. The method according to claim 2, characterized in that, The step of inputting the image block pixel sequence and the two-dimensional spatial coordinate set into the first full-slice image encoder to obtain the image encoding result output by the first full-slice image encoder includes: The image block pixel sequence and the two-dimensional spatial coordinate set are input into the image block embedding layer in the first full-slice image encoder to obtain the image block unit feature vector sequence output by the shared linear projection of the image block embedding layer; The image patch unit feature vector sequence and the two-dimensional spatial coordinate set are input into the multi-layer panoramic spatial attention coding network layer to obtain multiple image patch unit feature vector sub-sequences and feature vector sequences extracted by the local attention sub-module and the global attention sub-module connected in series in the multi-layer panoramic spatial attention coding network layer; wherein, the image patch unit feature vector sub-sequence is obtained by grouping the image patch unit feature vector sequence, and each image patch unit feature vector sub-sequence contains the same number of image patch unit feature vectors; The feature vector subsequences of the multiple image block units are input into the feature aggregation layer to obtain the image encoding result after feature aggregation by the feature aggregation layer.

4. The method according to claim 2 or 3, characterized in that, The process of feature vector extraction in the multi-layer panoramic spatial attention coding network includes: Based on the local attention submodule in the multi-layer panoramic spatial attention coding network layer, the image block unit feature vector sequence output by the image block embedding layer is grouped to obtain multiple image block unit feature vector subsequences; A learnable unit feature vector is inserted into each of the image block unit feature vector subsequences, and feature calculation is performed on each of the image block unit feature vector subsequences and the learnable unit feature vectors based on a multi-head self-attention mechanism to obtain the first subsequence feature vector, the first subsequence mean feature vector, and the subsequence coordinate label information; Based on the global attention submodule in the multi-layer panoramic spatial attention coding network layer, spatial information encoding is performed on the first subsequence feature vector and the subsequence coordinate labeling information to obtain the second subsequence feature vector; Based on the global attention submodule, the farthest point sampling compression is performed on the first subsequence feature vector, the first subsequence mean feature vector and the subsequence coordinate labeling information to obtain a first preset number of sampled feature vectors; and multi-head cross attention operation is performed on the second subsequence feature vector and the first preset number of sampled feature vectors to obtain the multi-head cross attention operation result. The result of the multi-head cross-attention operation is superimposed on the feature vector sub-sequences of the multiple image block units in a residual manner to obtain the feature superposition result. The feature superposition result is then processed by a feedforward neural network and a normalization layer to obtain the feature extraction result of the multi-layer panoramic spatial attention coding network layer.

5. The method according to claim 4, characterized in that, The image block unit feature vector sequence output from the image block embedding layer is grouped to obtain multiple image block unit feature vector sub-sequences, including: Based on the spatial coordinates in the set of two-dimensional spatial coordinates, the image block unit feature vector sequence is grouped to obtain multiple image block unit feature vector sub-sequences; In this context, each image block corresponding to the feature vector of an image block unit in the image block unit feature vector sub-sequence constitutes a sample image sub-region with the same length and width.

6. The method according to claim 5, characterized in that, The multi-head self-attention mechanism performs feature calculations on each of the image block unit feature vector subsequences and the learnable unit feature vectors to obtain a first subsequence feature vector, a first subsequence mean feature vector, and subsequence coordinate labeling information, including: The relative position information of each image block unit feature vector in the corresponding image block unit feature vector subsequence is superimposed onto each image block unit feature vector to obtain an image block unit feature vector carrying subsequence position information; For each of the image block unit feature vector subsequences, the learnable unit feature vector is concatenated with the corresponding image block unit feature vector carrying subsequence position information to form a joint feature sequence; Perform multi-head self-attention operation on the joint feature sequence to obtain the first sub-sequence feature vector and the first sub-sequence mean feature vector; and use the centroid of the spatial coordinates of the image block unit feature vector sub-sequence corresponding to the joint feature sequence as the sub-sequence coordinate labeling information; Wherein, the first subsequence feature vector is the learnable unit feature vector after multi-head self-attention operation, and the first subsequence mean feature vector is the mean of the image block unit feature vector carrying subsequence position information after multi-head self-attention operation.

7. The method according to claim 4, characterized in that, The step of spatially encoding the first subsequence feature vector and the subsequence coordinate labeling information to obtain the second subsequence feature vector includes: Based on the subsequence coordinate labeling information, multiple spatially related feature vectors are obtained for each first subsequence feature vector, wherein the spatially related feature vector is a first subsequence feature vector whose spatial positional relationship with the first subsequence feature vector satisfies a preset spatial positional relationship; Based on the spatial positional relationship between the first subsequence feature vector and the plurality of spatially associated feature vectors, the attention weight between the first subsequence feature vector and the plurality of spatially associated feature vectors is obtained; Based on the attention weights, feature information is encoded on the first subsequence feature vector and the plurality of spatially associated feature vectors to obtain the second subsequence feature vector.

8. The method according to claim 4, characterized in that, The global attention submodule performs farthest point sampling compression based on the first subsequence feature vector, the first subsequence mean feature vector, and the subsequence coordinate labeling information to obtain a first preset number of sampled feature vectors, including: The farthest point sampling algorithm is used to iteratively sample in the two-dimensional coordinate space to select a second preset number of spatially uniform representative centers; wherein the second preset number is less than the number of feature vector sub-sequences of the image block unit; Each image block unit feature vector sub-sequence is assigned to the representative center that is spatially closest to it; For each representative center, the first subsequence feature vector of the image block unit feature vector subsequence associated with each representative center is aggregated by maximum value to obtain a second preset number of first sampled feature vectors; and the mean feature vector of the first subsequence of the image block unit feature vector subsequence associated with each representative center is aggregated by maximum value to obtain a second preset number of second sampled feature vectors. The first sampled feature vector and the second sampled feature vector are concatenated and fused to obtain the first preset number of sampled feature vectors.

9. The method according to claim 1, characterized in that, The loss calculation based on the image encoding result and the first attribute information of each sample image to obtain the learning loss includes: The image encoding result is input into a pre-trained classification prediction head to obtain the classification prediction result of each sample image. Based on the classification prediction result and the classification label information, the cross-entropy loss is obtained and the cross-entropy loss is used as the learning loss. Alternatively, the pathological diagnosis text information can be text encoded to obtain a text embedding vector, and an information noise contrast estimation loss can be obtained based on the text embedding vector and the image encoding result, and the information noise contrast estimation loss can be used as the learning loss.

10. A method for encoding a whole-slice image, characterized in that, include: Obtain the first pathological whole slide image; The first pathological whole slice image is preprocessed to obtain a first image block pixel sequence and a first coordinate set, wherein the first coordinate set is a two-dimensional spatial coordinate set corresponding to the first image block pixel sequence; The first image block pixel sequence and the first coordinate set are input into the first full-slice image encoder to obtain the first image encoding result output by the first full-slice image encoder, wherein the first full-slice image encoder is an encoder obtained by the full-slice image encoder training method described in any one of claims 1-9.

11. A training device for a full-slice image encoder, characterized in that, include: The sample preprocessing module is used to obtain a set of pathological whole slide sample images and to preprocess each sample image in the set of pathological whole slide sample images to obtain the image block pixel sequence and two-dimensional spatial coordinate set of each sample image. The sample input module is used to input the image block pixel sequence and the two-dimensional spatial coordinate set into the first full-slice image encoder to obtain the image encoding result output by the first full-slice image encoder; The loss calculation module is used to perform loss calculation based on the image encoding result and the first attribute information of each sample image to obtain the learning loss; wherein, the first attribute information is at least one of the classification label information and pathological diagnosis text information corresponding to each sample image; The training update module is used to update the parameters of the first full-slice image encoder based on the backward gradient of the learning loss, so as to obtain the second full-slice image encoder.

12. A full-slice image encoding device, characterized in that, include: The image acquisition module is used to obtain the first pathological whole slide image; The image preprocessing module is used to preprocess the first pathological whole slice image to obtain a first image block pixel sequence and a first coordinate set, wherein the first coordinate set is a two-dimensional spatial coordinate set corresponding to the first image block pixel sequence; An image encoding module is used to input the pixel sequence of the first image block and the first coordinate set into a first full-slice image encoder to obtain a first image encoding result output by the first full-slice image encoder, wherein the first full-slice image encoder is an encoder obtained by the full-slice image encoder training method described in any one of claims 1-9.

13. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the full-slice image encoder training method or the full-slice image encoding method as described in any one of claims 1-10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the full-slice image encoder training method or the full-slice image encoding method as described in any one of claims 1-10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the full-slice image encoder training method or the full-slice image encoding method as described in any one of claims 1-10.

Images