Prostate gleason grading method and device based on improved deformable attention

Abstract

The application provides a prostate Gleason grading method and device based on improved deformable attention, which comprises the following steps: stacking a plurality of continuous image frames corresponding to the same case to obtain pseudo 3D volume data; constructing a prostate Gleason grading model comprising an encoding module, a bottleneck module, a decoding module and a classification head based on a U-net architecture; the encoding module extracts features of the pseudo 3D volume data through a plurality of encoding layers to obtain a pathological section feature sequence; the pathological section feature sequence is input into the decoding module through the bottleneck module to obtain a decoding result; and the classification head classifies the decoding result to obtain a Gleason grading result of each image frame. Through the deformable attention module of the encoding layer, the spatial continuity prior between the continuous sections is actively utilized to guide the attention sampling process, the context information of the sequence data is fully mined, and the missed diagnosis risk of high-grade lesions is reduced.

Description

Technical Field

[0001] This application relates to the field of medical image processing, and in particular to a method and apparatus for prostate Gleason grading based on improved deformable attention. Background Technology

[0002] Prostate cancer is a common malignant tumor of the male urinary system. The Gleason classification is a core pathological basis for clinically assessing the malignancy of prostate tumors, formulating treatment plans, and predicting prognosis. Essentially, it involves pixel-level sub-region segmentation of prostate pathological tissue images to accurately distinguish benign tissue from lesions of different malignant grades. With the development of digital pathology technology, automated pathological image segmentation methods based on deep learning have become an important technical approach to improve diagnostic efficiency and alleviate the workload of pathologists. The accuracy and spatial consistency of the segmentation results directly determine the clinical reliability of intelligent grading.

[0003] Currently, prostate pathological image segmentation is mostly based on two-dimensional slices. Convolutional neural networks, represented by U-Net, have become the mainstream infrastructure in this field due to their excellent local feature extraction capabilities, and various improved schemes such as ResUNet and nnUNet have been derived, achieving certain results in the segmentation of conventional lesion areas. However, prostate pathological tissues have natural three-dimensional spatial continuity, and the morphology and location of the same lesion on consecutive slices are closely spatially correlated. Existing two-dimensional segmentation methods generally treat each slice as an independent sample, completely severing the spatial context between consecutive slices, and failing to utilize three-dimensional structural information to assist segmentation decisions. On the one hand, for high-grade lesions such as Gleason grades 4 and 5, which have irregular shapes, blurred boundaries, and subtle texture differences, it is difficult to extract sufficient discriminative features from a single frame image, resulting in insufficient segmentation accuracy and sensitivity for high-grade lesions, and a high risk of missed diagnoses. On the other hand, the segmentation results of independently predicted consecutive slices are prone to inter-frame jitter and flicker, with poor spatial consistency, and cannot directly support stable three-dimensional lesion reconstruction and clinical evaluation.

[0004] To extract three-dimensional spatial information, some studies have used 3D convolutional networks, such as VNet, to directly process volumetric data, capturing local correlations between adjacent slices through 3D convolutional kernels. However, the computational cost and memory overhead of 3D convolution increase dramatically with data resolution, while pathological whole-slice images typically have extremely high resolution. Directly using 3D convolutional networks for processing would incur prohibitive computational costs, significantly limiting its practical clinical applications.

[0005] In recent years, the VisionTransformer architecture, with its global long-range dependency modeling capabilities, has been introduced into the field of medical image segmentation. Hybrid architectures such as TransUNet, Swin-UNet, and UNETR have been proposed, demonstrating superior performance compared to pure CNN models in global context capture. However, the computational complexity of standard self-attention mechanisms increases quadratically with the length of the input sequence, resulting in extremely high computational costs when directly applied to high-resolution 3D medical images, making practical applications difficult. To address this issue, deformable attention mechanisms dynamically predict sparse sampling points, reducing computational cost while retaining long-range modeling capabilities. However, existing deformable attention mechanisms predict sampling points based solely on the static content of a single frame, making the optimization process prone to getting trapped in local optima. They cannot actively utilize the prior spatial continuity between consecutive slices to guide the sampling process, making it difficult to fully extract contextual information from the sequence data. Furthermore, tissues and lesions in medical images exhibit significant anisotropic structural features, which general attention architectures have not adapted to, resulting in insufficient ability to characterize the irregularly shaped and oriented glandular and lesion structures in prostate pathological tissues.

[0006] Furthermore, in encoder-decoder segmentation architectures, skip connections are key components for restoring segmentation details and boundary accuracy. However, traditional skip connections only transmit shallow detail features within a single frame and fail to combine complementary information from adjacent frames to assist in boundary and detail restoration. The decoding stage also lacks a constraint mechanism for spatial continuity between slices, resulting in insufficient smoothness of the continuous segmentation results output by the model in the spatial dimension, and the inter-frame jitter problem cannot be effectively suppressed. Summary of the Invention

[0007] This application provides a method and apparatus for prostate Gleason grading based on improved deformable attention. The deformable attention module of the coding layer fuses the current feature map with the adjacent feature maps above and below to obtain spatial fusion input, and performs deformable attention calculation. It actively uses the spatial continuity prior between consecutive slices to guide the attention sampling process, fully explores the contextual information of the sequence data, improves the feature extraction and discrimination ability of high-grade lesions with irregular shape and blurred boundaries, and reduces the risk of missed diagnosis of high-grade lesions.

[0008] In a first aspect, embodiments of this application provide a prostate Gleason grading method based on improved deformable attention, the method comprising:

[0009] Pseudo-3D volume data is obtained by stacking multiple consecutive image frames corresponding to the same case, wherein the image frames are pathological slide images; A prostate Gleason classification model is constructed based on the U-net architecture. The prostate Gleason classification model includes an encoding module, a bottleneck module, a decoding module, and a classification head. The pseudo-3D volume data is input into the prostate Gleason grading model. The encoding module extracts features from the pseudo-3D volume data using multiple encoding layers. Each encoding layer includes an encoder and a deformable attention module. The encoder extracts features from the input of the current layer to obtain feature maps that correspond one-to-one with the image frames, forming a feature map sequence. The deformable attention module constructs a spatial fusion input for each feature map in the feature map sequence and performs deformable attention calculation based on the spatial fusion input to obtain pathological slice features that correspond one-to-one with the image frames, forming a pathological slice feature sequence. The spatial fusion input is the fusion result of the current feature map with the previous adjacent feature map and the next adjacent feature map. The bottleneck module processes the pathological slice feature sequence output by the last coding layer and inputs the corresponding pathological slice features to the decoding module frame by frame. The decoding module decodes the received current pathological slice features to obtain the decoding result corresponding to the current image frame. Each decoding layer in the decoding module is connected to the corresponding coding layer through a skip connection. The decoding result corresponding to each image frame is input into the classification head to obtain the Gleason classification result for each image frame of the corresponding case.

[0010] Secondly, embodiments of this application provide a prostate Gleason grading device based on improved deformable attention, comprising: The stacking module is used to stack multiple consecutive image frames corresponding to the same case to obtain pseudo-3D volume data, wherein the image frames are pathological slide images; The module is based on the U-net architecture to build a prostate Gleason classification model, which includes an encoding module, a bottleneck module, a decoding module, and a classification head. The grading module inputs pseudo-3D volumetric data into the prostate Gleason grading model. The encoding module extracts features from the pseudo-3D volumetric data using multiple encoding layers. Each encoding layer includes an encoder and a deformable attention module. The encoder extracts features from the input of the current layer to obtain a feature map sequence corresponding to each image frame. The deformable attention module constructs a spatial fusion input for each feature map in the feature map sequence and performs deformable attention calculation based on the spatial fusion input to obtain a pathological slice feature sequence corresponding to each image frame. The spatial fusion input is the fusion result of the current feature map with the previous and next adjacent feature maps. The bottleneck module processes the pathological slice feature sequence output from the last encoding layer and inputs the corresponding pathological slice features to the decoding module frame by frame. The decoding module decodes the received current pathological slice features to obtain the decoding result corresponding to the current image frame. Each decoding layer in the decoding module is connected to the corresponding encoding layer through skip connections. The decoding result corresponding to each image frame is input to the classification head to obtain the Gleason grading result for each image frame of the corresponding case.

[0011] Thirdly, embodiments of this application provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform a prostate Gleason grading method based on improved deformable attention.

[0012] Fourthly, embodiments of this application provide a readable storage medium storing a computer program that, when executed by a processor, implements a prostate Gleason grading method based on improved deformable attention.

[0013] The main contributions and innovations of this invention are as follows: This application embodiment uses a deformable attention module in the coding layer to fuse the current feature map with adjacent feature maps above and below to obtain spatial fusion input, and performs deformable attention calculation. It actively utilizes the prior spatial continuity between consecutive slices to guide the attention sampling process, fully mining the contextual information of the sequence data, improving the feature extraction and discrimination ability for high-grade lesions with irregular shapes and blurred boundaries, and reducing the risk of missing high-grade lesions. In the decoding stage, this application embodiment adopts a cross-frame optical flow alignment fusion mechanism for skip connections. The skip connection features of non-first frames are first fused with the features of the previous frame aligned with the regressed optical flow field, and then passed into the corresponding decoding layer, so that the decoding layer can reconstruct the details and boundaries of the current frame. At the same time, it can explicitly refer to historical frame information after motion compensation, effectively improving the accuracy of segmentation detail restoration and boundary segmentation, while enhancing the spatial coherence of continuous slice segmentation results; the decoding layer of this application introduces the Gleason segmentation result of the previous adjacent frame as prior information for non-first frame features, which is then fused with the current decoding features after global translation vector space alignment, and output through residual structure to impose inter-frame spatial consistency constraints on the decoding process, effectively suppressing inter-frame jitter and flicker of continuous slice segmentation results, generating smoother and more stable segmentation results, supporting subsequent three-dimensional lesion reconstruction and clinical evaluation; the residual connection simultaneously ensures the preservation of original feature information and the effective transmission of gradients.

[0014] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of a prostate Gleason grading method based on improved deformable attention according to an embodiment of this application; Figure 2 This is a structural schematic diagram of a deformable attention module according to an embodiment of this application; Figure 3 This is a schematic diagram of a structure for obtaining a new skip connection feature according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a decoding layer according to an embodiment of this application; Figure 5 This is a comparison diagram of the segmentation results of a prostate Gleason grading model and a partial contrast model according to an embodiment of this application; Figure 6 This is a structural block diagram of a prostate Gleason grading device based on improved deformable attention, according to an embodiment of this application. Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0016] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0017] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0018] Example 1 This application provides a Gleason grading method for the prostate based on improved deformable attention. It uses a deformable attention module in the coding layer to fuse the current feature map with adjacent feature maps to obtain a spatial fusion input, and then performs deformable attention calculation. It actively utilizes the spatial continuity prior between consecutive slices to guide the attention sampling process, fully mining the contextual information of the sequence data. This improves the feature extraction and discrimination ability for high-grade lesions with irregular shapes and blurred boundaries, reducing the risk of missed diagnosis of high-grade lesions. Specifically, refer to... Figure 1 The method includes: Pseudo-3D volume data is obtained by stacking multiple consecutive image frames corresponding to the same case, wherein the image frames are pathological slide images; A prostate Gleason classification model is constructed based on the U-net architecture. The prostate Gleason classification model includes an encoding module, a bottleneck module, a decoding module, and a classification head. The pseudo-3D volume data is input into the prostate Gleason grading model. The encoding module extracts features from the pseudo-3D volume data using multiple encoding layers. Each encoding layer includes an encoder and a deformable attention module. The encoder extracts features from the input of the current layer to obtain feature maps that correspond one-to-one with the image frames, forming a feature map sequence. The deformable attention module constructs a spatial fusion input for each feature map in the feature map sequence and performs deformable attention calculation based on the spatial fusion input to obtain pathological slice features that correspond one-to-one with the image frames, forming a pathological slice feature sequence. The spatial fusion input is the fusion result of the current feature map with the previous adjacent feature map and the next adjacent feature map. The bottleneck module processes the pathological slice feature sequence output by the last coding layer and inputs the corresponding pathological slice features to the decoding module frame by frame. The decoding module decodes the received current pathological slice features to obtain the decoding result corresponding to the current image frame. Each decoding layer in the decoding module is connected to the corresponding coding layer through a skip connection. The decoding result corresponding to each image frame is input into the classification head to obtain the Gleason classification result for each image frame of the corresponding case.

[0019] In the current embodiment, multiple consecutive image frames are stacked on the spatial Z-axis to obtain pseudo-3D volume data, and the image frames in this scheme are two-dimensional pathological image slices.

[0020] Specifically, pseudo-3D volumetric data is constructed by stacking data along the Z-axis in space. This retains the computational efficiency of single-frame image processing in 2D convolution while introducing cross-frame 3D spatial information. It eliminates the need to directly input the entire high-resolution pathological slice into the 3D convolutional network, significantly reducing the computational cost of 3D spatial information mining and adapting to the processing needs of whole-slice pathological images in clinical scenarios.

[0021] In the current embodiment, the constructed prostate Gleason grading model is trained using publicly available medical image datasets such as Gleason2019 and PANDA, which are already labeled prostate pathology datasets. During the training process, the total loss function is constrained by a combination of cross-entropy loss and inter-frame continuity loss.

[0022] Specifically, the encoder used for feature map extraction, as well as the bottleneck module and decoding module in the prostate Gleason hierarchical model trained in this scheme, are all existing technologies unless otherwise mentioned in this scheme. This scheme does not impose any additional restrictions on the structure of the encoder, bottleneck module and decoding module.

[0023] In the current embodiment, in the step of constructing spatial fusion input for each feature map, if the image frame corresponding to the current feature map is the first frame, then the current feature map is used as the previous adjacent feature map; if the image frame corresponding to the current feature map is the last frame, then the current feature map is used as the next adjacent feature map.

[0024] Specifically, in order to avoid numerical artifacts generated by the first and last frames when constructing spatial fusion input, and to enable attention calculation by constructing multi-frame collaborative spatial fusion input while maintaining the continuity of the model feature space, the feature map of the current frame is used as the previous or next adjacent feature map when constructing spatial fusion input for the feature map corresponding to the first frame and for the feature map corresponding to the last frame.

[0025] In other words, the formula for obtaining spatial fusion input is expressed as:

[0026] in, Input into spatial integration, For splicing and merging operations, This is the feature map corresponding to the image frame at time t. Let be the feature map preceding the previous one at time t-1. For the next adjacent feature map at time t+1, when t is the first frame, When t is the last frame, .

[0027] In the current embodiment, the structure of the deformable attention module is as follows: Figure 2 As shown, the spatial fusion input is projected into a query vector using a query projection matrix. The query vector is then input into the offset prediction subnetwork to obtain the regression offset and scale factor. The initial anchor coordinates of the spatial fusion input are obtained. The initial anchor coordinates are then deformed based on the regression offset and scale factor to obtain the deformed point coordinates. Key vectors and value vectors are constructed based on the deformed point coordinates. Multi-head attention is then performed based on the query vector, key vector, and value vector to obtain the corresponding pathological slice features.

[0028] Specifically, the offset prediction subnetwork in the deformable attention module is a multi-layer convolutional structure. The offset prediction subnetwork is used to predict the anisotropic deformation parameters of each sampling point in the spatial fusion input.

[0029] Specifically, the initial anchor point coordinates in this scheme are generated based on the geometric rules of spatial fusion input. For example, they can be generated using methods such as uniform grid generation.

[0030] Specifically, the formula for projecting the spatial fusion input into a query vector is expressed as:

[0031] in, For query vector, For spatial fusion input, To query the projection matrix.

[0032] Specifically, using the pre-trained offset prediction subnetwork as an example, the formulas for obtaining the regression offset and scale factor are expressed as follows:

[0033] in, This is the regression offset. As a scale factor, For query vector, As an offset prediction subnetwork, the regression offset is used to correct the degree of offset of the initial anchor point coordinates so that it can adapt to the irregular variation of lesion morphology. The scale factor is used to adjust the receptive field scale of deformation in the x and y directions to match the feature extraction requirements of glandular structures of different sizes in prostate pathological tissue, so that the model can autonomously adjust the sampling range for lesion structures of different sizes and improve the feature representation ability of anisotropic structures.

[0034] Specifically, the formulas for constructing the key vector and value vector are as follows:

[0035]

[0036] in, For key vectors, To use regression offset and scale factor The coordinates of the deformed points after deformation. The key projection matrix, For value vectors, The projection matrix is ​​the value. Given a differentiable bilinear interpolation sampling function, then for the coordinates of non-integer deformed points after deformation... Its sampled value is a weighted average of the eigenvalues ​​of the four nearest neighbor integer coordinate points around the point.

[0037] Specifically, in multi-head attention calculation, the calculation formula for each attention head is expressed as:

[0038] in, The calculation result for the m-th attention head is... For the Softmax function, Let be the dimension of the key vector. To conform to the relative position offset term designed for the Swing Transformer, Let m be the query vector for the m-th attention head. This is the transpose of the key vector of the m-th attention head. Let m be the value vector of the attention head.

[0039] The corresponding pathological slide features are obtained by concatenating the calculation results of all attention heads, as expressed by the formula:

[0040] in, The output pathological slide features, The projection matrix used for the output. This represents the total number of attention heads.

[0041] In the current embodiment, the bottleneck module is a general convolutional layer used to further aggregate global features of the deep pathological slice feature sequence output by the encoding module, fully integrate the contextual association information between different pathological slices, provide more discriminative high-level semantic features for the subsequent decoding stage, and ensure the accuracy of subsequent hierarchical prediction. The bottleneck module in this solution can be implemented by stacking multiple convolutional layers. There are no additional restrictions on the specific number of layers and channels of the bottleneck module in this solution. Those skilled in the art can adjust it according to the computing resource requirements and accuracy requirements.

[0042] In the current embodiment, the number of decoding layers in the decoding module is the same as the number of encoding layers in the encoding module and they correspond one-to-one. The pathological slide features input into the decoding layer are used as features to be decoded. During the skip connection process, each decoding layer obtains the pathological slide features corresponding to the features to be decoded in the corresponding encoding layer as skip connection features. The decoding layer decodes the joint decoding features to obtain the decoding result of the current decoding layer. The joint decoding features are the concatenation result of the features to be decoded and the skip connection features.

[0043] Furthermore, when the image frame corresponding to the skip connection feature is the same as the feature corresponding to the first image frame, the skip connection feature is directly concatenated with the feature to be decoded to obtain the joint decoding feature; when the skip connection feature is not the feature corresponding to the first image frame, the previous adjacent pathological slice feature in the pathological slice feature sequence is obtained as the preceding feature, the regression optical flow field of the preceding feature transformed into the skip connection feature is predicted, and then the preceding feature is mapped to the coordinate system of the skip connection feature based on the regression optical flow field to obtain the aligned preceding feature. The aligned preceding feature is fused with the skip connection feature to obtain a new skip connection feature, and the new skip connection feature is concatenated with the feature to be decoded to obtain the joint decoding feature. The structure diagram of the new skip connection feature is shown below. Figure 3 As shown.

[0044] Specifically, when the image frame corresponding to the skip connection feature is the same as the feature corresponding to the first image frame, in order to avoid feature redundancy and degradation, the skip connection feature is directly concatenated with the feature to be decoded.

[0045] Specifically, with As a feature of skip connections, As a preceding feature, the formula for obtaining the regressed optical flow field is expressed as:

[0046] in, The regressive optical flow field is transformed from pre-sequence feature transformation to skip connection features. For lightweight motion estimation networks, For skip connection features, with The preceding features are represented by a lightweight motion estimation network in this scheme, which performs feature processing through four convolutional layers. Each convolutional layer is followed by a ReLU activation function, and finally, the regressed optical flow field is obtained by regression through a convolutional layer with two output channels. The regressed optical flow field represents the amount of displacement required for each spatial position when the preceding features are transformed into the jump-connected feature coordinate system.

[0047] Specifically, the formula for aligning the preceding features to the coordinate system of the skip connection features is expressed as:

[0048] in, To align preceding features, This represents a differentiable bilinear sampling operation. As a preceding feature, To return to the optical flow field.

[0049] Finally, the formula for obtaining the new skip connection features is expressed as:

[0050] in, For the new skip connection feature, This means reducing the number of channels in the concatenated features back to the original dimensionality. To align preceding features, This is a skip connection feature. For the splicing operation, new skip connection features are used to replace the original features and are passed to the corresponding decoding layer. This allows the decoding layer to explicitly refer to motion-compensated historical information when reconstructing the details of the current frame, thereby effectively improving the spatial coherence and accuracy of the segmentation results.

[0051] In the current embodiment, if the image frame corresponding to the joint decoding feature of the current decoding layer is not the first frame, the Gleason classification result corresponding to the previous adjacent image frame is obtained as the prior image frame segmentation result, and the joint decoding feature is fused with the corresponding prior image frame segmentation result as a new joint decoding feature. The decoding layer decodes the new joint decoding feature.

[0052] Specifically, if the image frame corresponding to the joint decoding feature of the current decoding layer is the first frame, then in order to avoid invalid temporal pre-verification injection, the joint decoding feature can be decoded directly.

[0053] Furthermore, the structural diagram of the decoding layer is as follows: Figure 4 As shown, if the image frame corresponding to the joint decoding feature of the current decoding layer is not the first frame, the prior image frame segmentation result is aligned to the channel dimension of the joint decoding feature to obtain historical segmentation information. Based on the concatenation result of the historical segmentation information and the joint decoding feature, a global translation vector is predicted. Based on the global translation vector, the historical segmentation information is aligned to the coordinate system of the joint decoding feature to obtain aligned segmentation information. The aligned segmentation information and the joint decoding feature are fused and output through residual connection to obtain a new joint decoding feature.

[0054] Specifically, starting with the second image frame, this scheme uses the segmentation results of the previous adjacent image frame to provide temporal consistency constraints, thereby strictly ensuring the consistency of the inference conditions during training. This scheme aligns the prior image frame segmentation results to the channel dimension of the joint decoding features through bilinear interpolation downsampling, as expressed by the formula:

[0055] in, To segment information for historical purposes, This indicates a bilinear interpolation downsampling operation. The result is the prior image frame segmentation. These are the height and width, respectively.

[0056] Specifically, a multilayer perceptron is constructed to predict the global translation vector of historical segmentation information shifted to the joint decoded features. The formula for obtaining the global translation vector is as follows:

[0057] in, It is the transpose of the global translation vector. For joint decoding features, To segment information for historical purposes, For feature splicing, This is a global average pooling operation. For a multilayer perceptron, the global translation vector represents the translation amount of historical segmentation information in the x and y directions of the plane. The global translation vector encodes the overall motion prior between adjacent frames.

[0058] Specifically, the process of obtaining aligned segmentation information involves using a global translation vector to perform motion compensation on historical segmentation information. This is achieved by aligning the information to the coordinate system of the jointly decoded features through a differentiable network sampling operation. The formula for obtaining the aligned segmentation information is expressed as:

[0059] in, To align the segmented information, It is an affine transformation network generated based on the global translation vector. For the actual sampling operation performed using bilinear interpolation, It is a global translation vector. The output is historical segmentation information. This represents the feature representation after motion compensation using a global translation vector. With joint decoding features Coarse-grained alignment has been achieved in space.

[0060] Finally, the alignment segmentation information and the joint decoding features are fused to obtain fused features. Then, the fused features and the joint decoding features are residually concatenated to obtain new joint decoding features, as expressed by the formula:

[0061]

[0062] in, For a 1×1 convolution operation, For feature splicing, For joint decoding features, To align the segmented information, As a feature of fusion, This is a new joint decoding feature.

[0063] Specifically, the enhanced new joint decoding features include motion-compensated contextual information from historical frames. The entire network can explicitly utilize spatial consistency priors to effectively suppress jitter in inter-frame segmentation results, thereby generating smoother and more stable dynamic medical image segmentation results. Furthermore, residual connections can effectively preserve original information and ensure gradient flow.

[0064] In the current embodiment, the classification head outputs the different Gleason level probabilities of each image frame and the corresponding image segmentation results as Gleason classification results through a fully connected layer.

[0065] To verify the effectiveness of the prostate Gleason grading model in this scheme, we use DAV-Former to represent the prostate Gleason grading model in this scheme for segmentation performance verification: A comparative experiment on DAV-Former and other general medical segmentation models was conducted to compare the Dice coefficients. The results of the Dice coefficient comparison experiment are shown in Table 1. Table 1. Experimental Results of Dice Coefficient Comparison

[0066] First, as shown in Table 1, in terms of overall performance, DAV-Former achieved the best result in terms of average Dice coefficient (0.9108), which is 1.58% and 2.51% higher than the second-best performing nnUNet (0.8950) and TransUNet (0.8857), respectively. This improvement indicates that the design combining the anisotropic deformable attention mechanism and the spatial smoothing decoder proposed in this invention can more effectively capture the anisotropic structural features of prostate tissue and its spatial continuity between slices. Regarding boundary segmentation accuracy, DAV-Former achieved 6.53 mm on the HD95 metric, significantly outperforming baseline models such as VNet (19.99 mm) and ResUNet (10.92 mm), and comparable to the best-performing nnUNet (6.44 mm), verifying the model's robustness in complex boundary segmentation.

[0067] Secondly, in terms of segmentation performance across different Gleason grades, DAV-Former demonstrated a significant advantage in segmenting the most clinically challenging high-grade lesions (G4 and G5). Specifically, at the G4 grade, the method studied in this chapter achieved a Dice coefficient of 0.9479, significantly outperforming all other comparative methods; at the G5 grade, it also achieved a leading result of 0.8681. This indicates that, given the irregular morphology and indistinct boundaries of high-grade tumor regions, the cross-scale attention mechanism employed by DAV-Former can adaptively focus on more discriminative areas, thereby achieving precise delineation of lesions with higher malignancy. Furthermore, in terms of sensitivity, DAV-Former also achieved the best average value (0.9108), further demonstrating its effectiveness in avoiding missed diagnoses.

[0068] Pure Transformer models (such as TransUNet) have a Dice coefficient of 0.8857, which is better than some CNN models, but fails to surpass state-of-the-art CNN architectures such as nnUNet. This indicates that while Transformers are powerful in capturing global context, they are insufficient in modeling fine-grained local features of medical images. DAV-Former, by combining anisotropic deformable attention with the local perception capabilities of CNNs and introducing spatial smoothness constraints, ultimately achieves the best balance across various metrics, surpassing both pure Transformer and pure CNN methods.

[0069] A comparative experiment was conducted between DAV-Former and other general medical segmentation models using HD95. The results of the HD95 comparative experiment are shown in Table 2. Table 2. Comparative Experiment Results of HD95

[0070] As shown in Table 2, DAV-Former's performance is second only to nnUNet and significantly better than other comparative models. In-depth analysis of each lesion subregion reveals that DAV-Former performs exceptionally well in benign (6.13 mm), G3 (6.20 mm), and G4 (6.46 mm) grades. For G5 grade, DAV-Former's HD95 is 7.34 mm, lower than nnUNet's 6.43 mm. This result indicates that although DAV-Former possesses top-tier boundary segmentation capabilities when processing most tissues, there is still room for improvement in its boundary accuracy for the most morphologically complex and poorly defined G5 grade high-grade tumors. However, considering the scarcity and annotation difficulty of G5 grade data, DAV-Former's ability to maintain acceptable segmentation accuracy under such extreme challenges demonstrates good robustness.

[0071] Sensitivity comparison experiments were conducted between DAV-Former and other general medical segmentation models. The results of the sensitivity comparison experiments are shown in Table 3. Table 3 Results of the sensitivity comparison experiment

[0072] As shown in Table 3, in the comparative analysis of sensitivity indicators, DAV-Former achieved an average value of 0.9108, outperforming other models in the comparison. This result reflects the model's excellent ability to identify various tissue regions of the prostate, especially in avoiding missed lesions.

[0073] Detailed data across different lesion grades show that DAV-Former exhibits stable and outstanding sensitivity in high-grade lesions, a region of high clinical concern. Specifically, it achieved a sensitivity of 0.9479 for G4 lesions, outperforming other methods in comparison; and reached a sensitivity of 0.8681 for G5 lesions. This performance demonstrates that DAV-Former's anisotropic deformable attention mechanism effectively captures the discriminative features of high-grade tumors, which are characterized by irregular morphology and indistinct boundaries, thereby enhancing its ability to identify malignant lesions. This mechanism adaptively focuses on areas with high structural heterogeneity, avoiding the loss of crucial information due to complex textures and unclear edges, which is essential for improving the model's sensitivity and robustness in high-grade lesions.

[0074] nnUNet, as a current state-of-the-art medical image segmentation framework, demonstrates robust overall sensitivity (mean 0.8950), particularly achieving a high sensitivity of 0.9288 for G3-level lesions. While Transformer-based models (such as TransUNet) perform well on G4-level lesions (0.9282), their sensitivity fluctuates at other levels, especially G5. In contrast, DAV-Former combines deformable attention with the local feature extraction capabilities of CNNs and introduces spatial smoothness constraints across slices. This approach maintains stable and high sensitivity across all lesion levels, exhibiting a relative advantage, especially in identifying high-level lesions. Its hybrid architecture retains the advantages of convolutional operations in extracting local details and spatial hierarchical features while enhancing the model's ability to model global morphology and complex structures through deformable attention, thus achieving a better generalization balance between high and low-level lesions. This design allows the model to effectively capture subtle structural changes when dealing with high-level lesions such as G4 and G5, while maintaining semantic consistency in the 3D context, significantly reducing the risk of missed detections of high-level malignant lesions.

[0075] The output results of the prostate Gleason grading model (DAV-Former) constructed in this scheme are compared with those of some contrasting models, for example... Figure 5 As shown, by Figure 5 It can be seen that, compared with other models, DAV-Former generally performs better in all subregions.

[0076] In addition, to ensure the effectiveness of different modules in the prostate Gleason grading model constructed in this scheme, ablation experiments were conducted by replacing different modules: First, the coding layer in the coding module was replaced with a standard Transformer module, and ablation experiments were conducted using regular skip connections and regular decoding layers. The results of the Dice coefficient ablation experiments are shown in Table 4, the HD95 ablation experiments are shown in Table 5, and the sensitivity ablation experiments are shown in Table 6. Table 4. Results of Dice coefficient ablation experiments

[0077] Table 5. Results of HD95 ablation experiments

[0078] Table 6. Results of Sensitivity Ablation Experiment

[0079] As shown in Tables 4, 5, and 6, when the coding layer is replaced with a standard Transformer module, all performance indicators decrease, with the average Dice coefficient decreasing to 0.8914, the average HD95 increasing to 8.64 mm, and the average sensitivity decreasing to 0.8888. When conventional skip connections are used, the model performance deteriorates further, with the average Dice coefficient decreasing to 0.8829, the average HD95 increasing to 8.64 mm, and the average sensitivity decreasing to 0.8888. However, when the coding layer is replaced with a standard Transformer module, conventional skip connections are used, and conventional decoding layers are used simultaneously, the average Dice coefficient drops significantly to 0.8694, the average HD95 increases to 9.43 mm, and the average sensitivity is only 0.8758.

[0080] Ablation experiments show that the unique encoding module, skip connection method, and decoding module of this scheme complement each other and together form the performance foundation of DAV-Former. Among them, the encoding module, as the core feature extraction module, significantly enhances the model's spatial context awareness ability with its dynamic anisotropic sampling mechanism; the skip connection method effectively improves the accuracy of detail recovery and boundary segmentation through cross-frame feature alignment and fusion; and the decoding module ensures the smoothness and stability of the output results in the spatial dimension by applying spatial consistency constraints, thus suppressing inter-frame jitter.

[0081] Example 2 Based on the same concept, referencing Figure 6 This application also proposes a prostate Gleason grading device based on improved deformable attention, comprising: The stacking module is used to stack multiple consecutive image frames corresponding to the same case to obtain pseudo-3D volume data, wherein the image frames are pathological slide images; The module is based on the U-net architecture to build a prostate Gleason classification model, which includes an encoding module, a bottleneck module, a decoding module, and a classification head. The grading module inputs pseudo-3D volumetric data into the prostate Gleason grading model. The encoding module extracts features from the pseudo-3D volumetric data using multiple encoding layers. Each encoding layer includes an encoder and a deformable attention module. The encoder extracts features from the input of the current layer to obtain a feature map sequence corresponding to each image frame. The deformable attention module constructs a spatial fusion input for each feature map in the feature map sequence and performs deformable attention calculation based on the spatial fusion input to obtain a pathological slice feature sequence corresponding to each image frame. The spatial fusion input is the fusion result of the current feature map with the previous and next adjacent feature maps. The bottleneck module processes the pathological slice feature sequence output from the last encoding layer and inputs the corresponding pathological slice features to the decoding module frame by frame. The decoding module decodes the received current pathological slice features to obtain the decoding result corresponding to the current image frame. Each decoding layer in the decoding module is connected to the corresponding encoding layer through skip connections. The decoding result corresponding to each image frame is input to the classification head to obtain the Gleason grading result for each image frame of the corresponding case.

[0082] Example 3 This embodiment also provides an electronic device, see reference. Figure 7 It includes a memory 402 and a processor 401, the memory 402 storing a computer program and the processor 401 being configured to run the computer program to perform the steps in any of the above method embodiments.

[0083] Specifically, the processor 401 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0084] The memory 402 may include a mass storage device for data or instructions. For example, and not limitingly, the memory 402 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 402 may include removable or non-removable (or fixed) media. Where appropriate, the memory 402 may be internal or external to a data processing device. In a particular embodiment, the memory 402 is non-volatile memory. In a particular embodiment, the memory 402 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0085] The memory 402 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 401.

[0086] The processor 401 implements any of the prostate Gleason grading methods based on improved deformable attention in the above embodiments by reading and executing computer program instructions stored in the memory 402.

[0087] Optionally, the electronic device may further include a transmission device 403 and an input / output device 404, wherein the transmission device 403 is connected to the processor 401 and the input / output device 404 is connected to the processor 401.

[0088] The transmission device 403 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 403 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0089] The input / output device 404 is used to input or output information. In this embodiment, the input information may be pathological slide images, etc., and the output information may be Gleason grading results, etc.

[0090] Optionally, in this embodiment, the processor 401 can be configured to perform the following steps via a computer program: Pseudo-3D volume data is obtained by stacking multiple consecutive image frames corresponding to the same case, wherein the image frames are pathological slide images; A prostate Gleason classification model is constructed based on the U-net architecture. The prostate Gleason classification model includes an encoding module, a bottleneck module, a decoding module, and a classification head. The pseudo-3D volume data is input into the prostate Gleason grading model. The encoding module extracts features from the pseudo-3D volume data using multiple encoding layers. Each encoding layer includes an encoder and a deformable attention module. The encoder extracts features from the input of the current layer to obtain feature maps that correspond one-to-one with the image frames, forming a feature map sequence. The deformable attention module constructs a spatial fusion input for each feature map in the feature map sequence and performs deformable attention calculation based on the spatial fusion input to obtain pathological slice features that correspond one-to-one with the image frames, forming a pathological slice feature sequence. The spatial fusion input is the fusion result of the current feature map with the previous adjacent feature map and the next adjacent feature map. The bottleneck module processes the pathological slice feature sequence output by the last coding layer and inputs the corresponding pathological slice features to the decoding module frame by frame. The decoding module decodes the received current pathological slice features to obtain the decoding result corresponding to the current image frame. Each decoding layer in the decoding module is connected to the corresponding coding layer through a skip connection. The decoding result corresponding to each image frame is input into the classification head to obtain the Gleason classification result for each image frame of the corresponding case.

[0091] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0092] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0093] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 7 Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0094] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A prostate Gleason grading method based on improved deformable attention, characterized in that, Includes the following steps: Pseudo-3D volume data is obtained by stacking multiple consecutive image frames corresponding to the same case, wherein the image frames are pathological slide images; A prostate Gleason classification model is constructed based on the U-net architecture. The prostate Gleason classification model includes an encoding module, a bottleneck module, a decoding module, and a classification head. The pseudo-3D volume data is input into the prostate Gleason grading model. The encoding module extracts features from the pseudo-3D volume data using multiple encoding layers. Each encoding layer includes an encoder and a deformable attention module. The encoder extracts features from the input of the current layer to obtain feature maps that correspond one-to-one with the image frames, forming a feature map sequence. The deformable attention module constructs a spatial fusion input for each feature map in the feature map sequence and performs deformable attention calculation based on the spatial fusion input to obtain pathological slice features that correspond one-to-one with the image frames, forming a pathological slice feature sequence. The spatial fusion input is the fusion result of the current feature map with the previous adjacent feature map and the next adjacent feature map. The bottleneck module processes the pathological slice feature sequence output from the last encoding layer and inputs the corresponding pathological slice features to the decoding module frame by frame. The decoding module decodes the received current pathological slice features to obtain the decoding result corresponding to the current image frame. Each decoding layer in the decoding module is connected to the corresponding encoding layer through skip connections. The pathological slice features input to the decoding layer are used as the features to be decoded. During the skip connection process, each decoding layer obtains the pathological slice features corresponding to the features to be decoded in the corresponding encoding layer as the skip connection features. The decoding layer decodes the joint decoding features to obtain the decoding result of the current decoding layer. The joint decoding features are the concatenation result of the features to be decoded and the skip connection features. When the image frame corresponding to the skip connection feature is the feature corresponding to the first image frame, the skip connection feature and the feature to be decoded are concatenated to obtain the joint decoding feature. When the skip connection feature is not the feature corresponding to the first image frame, the previous adjacent pathological slice feature in the pathological slice feature sequence of the skip connection feature is obtained as the preceding feature, and the regression light of the preceding feature transformed into the skip connection feature is predicted. The flow field is then used to map the preceding features to the coordinate system of the skip connection features to obtain aligned preceding features. The aligned preceding features are fused with the skip connection features to obtain new skip connection features. The new skip connection features are then concatenated with the features to be decoded to obtain joint decoding features. If the image frame corresponding to the joint decoding features of the current decoding layer is not the first frame, the Gleason classification result corresponding to the previous adjacent image frame is obtained as the prior image frame segmentation result. The joint decoding features are fused with the corresponding prior image frame segmentation result to obtain new joint decoding features. The decoding layer decodes the new joint decoding features. If the image frame corresponding to the joint decoding features of the current decoding layer is not the first frame, the prior image frame segmentation result is aligned to the channel dimension of the joint decoding features to obtain historical segmentation information. The global translation vector is predicted based on the concatenation result of the historical segmentation information and the joint decoding features. The historical segmentation information is aligned to the coordinate system of the joint decoding features based on the global translation vector to obtain aligned segmentation information. The aligned segmentation information is fused with the joint decoding features and output through residual connections to obtain new joint decoding features. The decoding result corresponding to each image frame is input into the classification head to obtain the Gleason classification result for each image frame of the corresponding case.

2. The prostate Gleason grading method based on improved deformable attention according to claim 1, characterized in that, In the step of constructing spatial fusion input for each feature map, if the image frame corresponding to the current feature map is the first frame, then the current feature map is used as the previous adjacent feature map; if the image frame corresponding to the current feature map is the last frame, then the current feature map is used as the next adjacent feature map.

3. The prostate Gleason grading method based on improved deformable attention according to claim 1, characterized in that, The spatial fusion input is projected into a query vector using a query projection matrix. The query vector is then input into the offset prediction subnetwork to obtain the regression offset and scale factor. The initial anchor coordinates of the spatial fusion input are obtained. The initial anchor coordinates are then deformed based on the regression offset and scale factor to obtain the deformed point coordinates. Key vectors and value vectors are constructed based on the deformed point coordinates. Multi-head attention is then performed based on the query vector, key vector, and value vector to obtain the corresponding pathological slice features.

4. A prostate Gleason grading device based on improved deformable attention, characterized in that, include: The stacking module is used to stack multiple consecutive image frames corresponding to the same case to obtain pseudo-3D volume data, wherein the image frames are pathological slide images; The module is based on the U-net architecture to build a prostate Gleason classification model, which includes an encoding module, a bottleneck module, a decoding module, and a classification head. The grading module inputs pseudo-3D volumetric data into the prostate Gleason grading model. The encoding module extracts features from the pseudo-3D volumetric data using multiple encoding layers. Each encoding layer includes an encoder and a deformable attention module. The encoder extracts features from the input of the current level to obtain a feature map sequence that corresponds one-to-one with each image frame. The deformable attention module constructs a spatial fusion input for each feature map in the feature map sequence and performs deformable attention calculations based on the spatial fusion input to obtain a pathological slice feature sequence that corresponds one-to-one with each image frame. The spatial fusion input is the fusion result of the current feature map with the previous and next adjacent feature maps. The bottleneck module then processes the last... The pathological slide feature sequences output by each encoding layer are processed, and the corresponding pathological slide features are input to the decoding module frame by frame. The decoding module decodes the received current pathological slide features to obtain the decoding result corresponding to the current image frame. Each decoding layer in the decoding module is connected to the corresponding encoding layer through skip connections. The pathological slide features input to the decoding layer are taken as the features to be decoded. During the skip connection process, each decoding layer obtains the pathological slide features corresponding to the features to be decoded in the corresponding encoding layer as the skip connection features. The decoding layer decodes the joint decoding features to obtain the decoding result of the current decoding layer. The joint decoding features are the concatenation result of the features to be decoded and the skip connection features. When the skip connection features correspond to... When the image frame corresponds to the first image frame, the jump connection feature and the feature to be decoded are concatenated to obtain the joint decoding feature. When the jump connection feature does not correspond to the first image frame, the previous adjacent pathological slice feature in the pathological slice feature sequence is obtained as the preceding feature. The regression optical flow field of the preceding feature transformed into the jump connection feature is predicted. Then, based on the regression optical flow field, the preceding feature is mapped to the coordinate system of the jump connection feature to obtain the aligned preceding feature. The aligned preceding feature and the jump connection feature are fused to obtain a new jump connection feature. The new jump connection feature is concatenated with the feature to be decoded to obtain the joint decoding feature. If the image frame corresponding to the joint decoding feature of the current decoding layer is not the first frame, the previous adjacent pathological slice feature in the pathological slice feature sequence is obtained as the preceding feature. The Gleason classification results corresponding to adjacent image frames are used as prior image frame segmentation results. The joint decoding features are fused with the corresponding prior image frame segmentation results to obtain new joint decoding features. The decoding layer decodes the new joint decoding features. If the image frame corresponding to the joint decoding features of the current decoding layer is not the first frame, the prior image frame segmentation results are aligned to the channel dimension of the joint decoding features to obtain historical segmentation information. Based on the concatenation result of the historical segmentation information and the joint decoding features, a global translation vector is predicted. Based on the global translation vector, the historical segmentation information is aligned to the coordinate system of the joint decoding features to obtain aligned segmentation information. The aligned segmentation information and the joint decoding features are fused and output through residual connection to obtain new joint decoding features.The decoding result corresponding to each image frame is input into the classification head to obtain the Gleason classification result for each image frame of the corresponding case.

5. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform a prostate Gleason grading method based on improved deformable attention as described in any one of claims 1-3.

6. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements a prostate Gleason grading method based on improved deformable attention as described in any one of claims 1-3.

Images