Pathological image splicing method, system and model training method based on deep learning

Abstract

The application discloses a pathological image splicing method, system and model training method based on deep learning. The method stacks multiple pathological image blocks along a channel dimension into a multi-channel input tensor, uses an alignment network to one-time predict transformation parameters with the minimum sum of re-projection errors as the target, completes registration through a differentiable spatial transformation network, performs feature re-calibration through an adaptive weighted fusion module, and finally outputs a panoramic pathological image through a multi-scale fusion network of a U-Net architecture. The system comprises an input module, an alignment network module, a differentiable spatial transformation network module, an adaptive weighted fusion module and a multi-scale fusion network module which are sequentially connected in signal. The model training method adopts a phased training strategy. The application improves the pathological image registration accuracy and fusion effect, can efficiently complete the accurate splicing of multiple image blocks, and is suitable for the rapid reconstruction and clinical analysis of digital pathological sections.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and medical image processing technology, specifically to a deep learning-based method, system, and model training method for stitching together pathological images. Background Technology

[0002] Pathological analysis is the gold standard for disease diagnosis. Full-field digital slide (WSI) technology uses a fully automated microscope to acquire high-resolution digital images of pathological tissue samples. After high-precision multi-field stitching processing by computer, complete digital slide images are obtained, which helps doctors to observe quickly and comprehensively, and significantly improves the efficiency and accuracy of pathological diagnosis.

[0003] Due to the physical limitations of optical imaging systems, it is impossible to simultaneously acquire images with high magnification and a wide field of view. In practice, it is necessary to first acquire a large number of local image tiles, and then synthesize them into a panoramic pathological image through image stitching technology. However, pathological image stitching faces many special technical challenges due to its inherent characteristics: First, after hematoxylin-eosin (HE) staining, the cell nuclei of pathological tissue sections appear blue-purple, while the cytoplasm and stroma appear pink, presenting a highly repetitive texture pattern. Traditional feature point detection-based methods (such as SIFT and SURF) struggle to extract stable features in such scenarios, resulting in a high feature matching error rate. Second, the need for microscopic feature observation in pathological diagnosis requires stitching errors to be controlled at the sub-pixel level (<1μm), which conventional methods cannot meet. Third, the time complexity of traditional pairwise registration strategies is O(n^2). 2 Furthermore, it is prone to error accumulation leading to global inconsistency, making it unsuitable for actual WSI scanning needs involving hundreds to thousands of image blocks; fourth, factors such as fluctuations in light source brightness and uneven dye concentration during the scanning process cause differences in brightness, contrast, and color between adjacent image blocks, making direct fusion prone to producing obvious seams.

[0004] Existing deep learning-based image stitching methods are mostly designed for natural scene images, and have significant limitations when applied to pathological image stitching: DeTone et al. [1] The proposed end-to-end homography matrix estimation method only supports two image inputs and cannot be extended to multi-image patch collaborative stitching. Furthermore, it is not optimized for the texture characteristics of pathological images. In pathological image stitching tests, when processing 25 image patches, its PSNR was 29.85 dB, SSIM was 0.798, and the stitching time was 0.856 seconds. Shen et al. [2]The proposed RANSAC-Flow two-stage registration method relies on feature point detection, and the matching error rate is as high as 15% or more in the case of repetitive textures in pathological images. Moreover, the staged processing will exacerbate the accumulation of errors. The pathological image registration or fusion schemes proposed in relevant domestic patents still adopt pairwise registration strategies or have not solved the global collaborative optimization problem. The stitching error accumulates with the increase of the number of image blocks. The simple combination of general U-Net, VGG and other network architectures does not consider the multi-scale features of pathological images and lacks a global consistency constraint mechanism for multiple image blocks.

[0005] In summary, existing technologies cannot meet the requirements of high precision, high efficiency, and global collaborative optimization in pathological image stitching. There is an urgent need for an intelligent stitching method, system, and model training method that is specifically designed for the characteristics of pathological images, supports end-to-end collaborative stitching of multiple image blocks, and has global optimization capabilities.

[0006] References: [1]DeTone D, Malisiewicz T, Rabinovich A. Deep image homographyestimation[J]. arXiv preprint arXiv:1606.03798, 2016 [2]Shen X, Darmon F, Zhou C, et al. RANSAC-Flow: generic two-stageimage alignment[C]. European Conference on Computer Vision. Springer, Cham,2020: 618-637 Summary of the Invention

[0007] In view of the shortcomings of existing technologies in pathological image stitching, such as poor global coordination, low registration accuracy, poor fusion effect, and low processing efficiency, this invention addresses at least the following technical problems: 1. Solve the high time complexity (O(n)) caused by the pairwise registration strategy used in traditional pathological image stitching. 2 The core problem of global stitching inconsistency caused by error accumulation is that it cannot achieve global collaborative optimization of multiple image blocks. It is adapted to the stitching requirements of hundreds to thousands of image blocks in full-view digital slicing scanning. 2. To address the problem that pathological images have complex textures and high background repetition, traditional feature point matching methods are difficult to extract stable features, and existing deep learning solutions have not been specifically optimized, resulting in registration accuracy that cannot reach the sub-pixel level (<1μm), thus failing to meet the needs of microscopic feature observation in pathological diagnosis; 3. To address the issue that differences in imaging conditions such as light source, staining, and objective lens during the scanning process cause brightness, contrast, and color deviations between adjacent image blocks, and that existing methods of direct fusion easily produce obvious stitching seams, affecting the consistency of panoramic images; 4. To address the problem that existing methods cannot effectively extract and retain multi-scale details (differences in cell nucleus, cytoplasm, and tissue structure scale) of pathological images when fusing multiple image patches, resulting in blurred fusion results and loss of key pathological diagnostic features; 5. This addresses the challenge that existing deep learning-based stitching methods are mostly designed for natural scene images and cannot be directly adapted to the characteristics of pathological images, or are simply combinations of general network architectures, lacking a global consistency constraint mechanism, and making it difficult to balance stitching accuracy, efficiency and image quality.

[0008] This invention discloses a deep learning-based method for stitching together pathological images, which includes the following steps: S1: Stack multiple pathological image blocks along the channel dimension to form a multi-channel input tensor; S2: The multi-channel input tensor is input into the alignment network, with the objective function being to minimize the sum of reprojection errors of all pathological image blocks relative to the central reference block. The transformation parameters of all pathological image blocks are predicted at once through the interaction of features within the network. S3: Input the transformation parameters into a differentiable spatial transformation network, and perform spatial transformation registration on the pathological image blocks through bilinear interpolation sampling to obtain the registered image; S4: Input the registered image into the adaptive weighted fusion module, and obtain the weighted fusion feature map through channel statistical feature extraction, weight generation and feature recalibration; S5: Input the weighted fusion feature map into the multi-scale fusion network, and after encoder downsampling, decoder upsampling and multi-scale feature stitching reconstruction, output a panoramic pathological image with RGB three channels.

[0009] Furthermore, the alignment network is provided with a cross-layer feature fusion unit, which includes two specific cross-layer skip connections. The first skip connection connects the second set of convolutional layers in the first feature extraction stage of the alignment network with the first set of convolutional layers in the second feature extraction stage, and the second skip connection connects the first set of convolutional layers in the second feature extraction stage with the second set of convolutional layers in the second feature extraction stage, thus fusing shallow high-resolution spatial information with deep abstract semantic information.

[0010] Furthermore, the processing procedure of the adaptive weighted fusion module is as follows: Global average pooling and global max pooling are performed on the feature maps of the registered images respectively to obtain two channel description vectors, which are then added element by element. The fused vector is input into a two-layer multilayer perceptron to generate channel weight coefficients, the channel weight coefficients being in the range of (0,1). The channel weight coefficients are multiplied one by one with the original input feature map to complete feature recalibration and output a weighted fused feature map.

[0011] Furthermore, the multi-scale fusion network is an encoder-decoder structure based on the U-Net architecture.

[0012] Furthermore, the multi-scale fusion network is equipped with four skip connection units, which concatenate the outputs of each stage of the encoder with the inputs of the corresponding stages of the decoder in the channel dimension, thus fusing deep semantic information and shallow detail information.

[0013] Furthermore, the output layer of the multi-scale fusion network is processed by 1×1 convolution and activation function to output an RGB three-channel panoramic pathological image.

[0014] Furthermore, the transformation parameters include rotation angle, horizontal translation amount, and vertical translation amount.

[0015] The present invention also provides a pathological image stitching system based on deep learning, for implementing the above-mentioned pathological image stitching method based on deep learning. The system includes an input module, an alignment network module, a differentiable spatial transformation network module, an adaptive weighted fusion module, and a multi-scale fusion network module connected in sequence. The input module is used to stack multiple pathological image blocks along the channel dimension to form a multi-channel input tensor; The alignment network module is used to receive multi-channel input tensors and output transformation parameters for all pathological image blocks with the goal of minimizing the sum of reprojection errors. The differentiable spatial transformation network module is used to perform bilinear interpolation sampling and registration on pathological image blocks based on transformation parameters, and output the registered image. The adaptive weighted fusion module is used to extract channel statistical features, generate weights, and recalibrate features of the registered image, and output a weighted fusion feature map. The multi-scale fusion network module is used to extract, stitch and reconstruct multi-scale features from the weighted fusion feature map, and output a panoramic pathological image with RGB three channels.

[0016] This invention also provides a deep learning-based training method for a pathological image stitching model, used to train the alignment network, adaptive weighted fusion module, and multi-scale fusion network in the aforementioned deep learning-based pathological image stitching method, specifically including: A phased training strategy is adopted: first, the alignment network is trained independently, and then the adaptive weighted fusion module and the multi-scale fusion network are trained together with the alignment network parameters fixed.

[0017] Furthermore, during training, a weighted combination of mean squared error loss and perceptual loss is used as the loss function, and the perceptual loss is calculated based on the feature output of the pre-trained VGG network. During training, a gradient backpropagation path is constructed based on the output panoramic pathological image. The gradient backpropagation path propagates back along the entire link from the multi-scale fusion network, the adaptive weighted fusion module, the differentiable spatial transformation network to the alignment network, thereby achieving end-to-end training of all network modules.

[0018] This invention achieves high-precision, high-efficiency, and high-consistency panoramic pathological image stitching through a multi-dimensional innovative approach, resulting in the following beneficial technical effects: 1. Stack multiple image patches along the channel dimension into a multi-channel tensor, and predict all transformation parameters at once through an alignment network, reducing the stitching time complexity from the traditional O(n) to O(n) channel. 2 The processing speed is reduced to O(n), which is about 25 times faster than AutoStitch and about 7 times faster than DeTone. The global optimization strategy effectively avoids the problem of error accumulation in pairwise registration. The stitching result has a PSNR of over 31.26dB and an SSIM of over 0.834, achieving global collaborative registration of multiple image blocks and global consistent registration effect. 2. An alignment network with specific cross-layer skip connections integrates shallow high-resolution spatial information with deep abstract semantic information. It can still extract stable features against the background of repetitive textures in pathological images, with a registration accuracy of 0.42±0.08μm, which meets the sub-pixel level (<1μm) accuracy requirements for pathological diagnosis. It effectively avoids problems such as cell structure breakage and ghosting, and is suitable for the microscopic observation needs of pathological diagnosis. 3. The adaptive weighted fusion module adaptively generates weight coefficients based on the statistical characteristics of each channel, accurately compensating for differences in brightness, contrast, and color caused by light source, staining, and objective lens during the scanning process. It completely eliminates stitching seams, improves the overall consistency of panoramic images, and achieves a subjective score of 4.6 out of 5 for pathological diagnosis usability. 4. The multi-scale fusion network based on the U-Net architecture fuses the deep semantic information of the encoder and the shallow detail information of the decoder through four skip connections, effectively preserving key pathological diagnostic features such as cell nucleus morphology, cell edges, and tissue structure, avoiding the problem of blurred fusion results, and the output RGB panoramic pathological image quality meets the clinical diagnostic standards. 5. The phased training strategy first optimizes the alignment network independently, and then trains the fusion module together. With the weighted combination of mean squared error loss and perceptual loss, each module of the network can be accurately adapted to the pathological image stitching task. The model has better convergence, more stable performance, and better stitching effect than a simple combination of general network architecture. 6. Deploying this invention on entry-level scanners can compensate for mechanical positioning errors, and the generated image quality is statistically indistinguishable from that of high-end equipment, significantly lowering the hardware threshold for digital pathology slide technology. Applying it to high-end scanners can reduce image block overlap to 15%, shortening scanning time by 40% while maintaining stitching quality. By replacing the original image blocks with transmission transformation parameters, bandwidth requirements are reduced by 99.996%, perfectly meeting the real-time data transmission needs of remote pathology consultations and possessing strong clinical practical value. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the deep learning-based pathological image stitching method of the present invention; Figure 2 This is a schematic diagram of the alignment network structure of the present invention. In the figure, "Conv3×3,s1" represents a 3×3 convolutional layer with a stride of 1, "MaxPool2×2,s2" represents a 2×2 max pooling layer with a stride of 2, and "FC" represents a fully connected layer. Figure 3 This is a schematic diagram of the adaptive weighted fusion module structure of the present invention. In the figure, "GAP" represents global average pooling, "GMP" represents global max pooling, "FC" represents a fully connected layer, "ReLU" represents a linear rectified activation function, and "Sigmoid" represents a sigmoid activation function. Figure 4 This is a schematic diagram of the multi-scale fusion network structure of the present invention; Figure 5 This is a schematic diagram of the training dataset construction process of this invention; Figure 6 This is a schematic diagram comparing the splicing effect of the present invention with other methods, where 6(a) is a comparison diagram of peak signal-to-noise ratio, 6(b) is a comparison diagram of structural similarity, 6(c) is a comparison diagram of processing time, 6(d) is a comparison diagram of pathologist subjective scores, 6(e) is a comparison diagram of normalized comprehensive performance, and 6(f) is a comparison diagram of speed. Figure 7 This is a schematic diagram comparing the ablation experimental results of the present invention, wherein 7(a) is a bar chart comparing the peak signal-to-noise ratio of the splicing results of different experimental groups, and 7(b) is a bar chart comparing the structural similarity of the splicing results of different experimental groups. Detailed Implementation

[0020] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0021] For clarity, the English abbreviations and symbols appearing in the specification and drawings of this invention are explained as follows: Conv (Convolution Layer): Convolutional layer; s (Stride): Step size; MaxPool (Max Pooling Layer): Max pooling layer; FC (Fully Connected Layer): A fully connected layer; GAP (Global Average Pooling): Global average pooling; GMP (Global Max Pooling): Global max pooling; ReLU (Rectified Linear Unit): Linear rectified activation function; Sigmoid: Sigmoid activation function; Tanh (Hyperbolic Tangent): Hyperbolic tangent activation function.

[0022] This embodiment provides a deep learning-based pathological image stitching method, system, and model training method, which are described in detail below: like Figure 1 As shown, the deep convolutional neural network used in this embodiment includes an alignment network, a differentiable spatial transformation network, an adaptive weighted fusion module, and a multi-scale fusion network.

[0023] like Figure 2 As shown, the alignment network serves as the core registration module, used to predict the transformation parameters of all input image patches at once. Its structure includes: a first feature extraction stage, consisting of two sets of cascaded 3×3 convolutional layers with 64 and 128 kernels respectively, followed by a 2×2 max pooling layer with a stride of 2; and a second feature extraction stage, consisting of three sets of cascaded 3×3 convolutional layers with 256, 512, and 512 kernels respectively, each containing three 3×3 convolutional layers with a stride of 1 and padding of 1, followed by a 2×2 max pooling layer with a stride of 2.

[0024] To integrate shallow spatial information with deep semantic information, the network has two specific cross-layer skip connections: the first skip connection connects the output of the second set of convolutional layers (128 channels, 256×256 resolution) in the first feature extraction stage with the input of the first set of convolutional layers (256 channels, 128×128 resolution) in the second feature extraction stage; the second skip connection connects the output of the first set of convolutional layers in the second feature extraction stage with the input of the second set of convolutional layers (512 channels, 64×64 resolution) in the second feature extraction stage.

[0025] Finally, three parallel fully connected layers are used to output the rotation angle θ and horizontal translation Δs of multiple image patches, respectively. x and vertical translation Δs y Preferably, the number of image blocks is at least four.

[0026] After the transformation parameter prediction is completed, the registration operation of the pathological image patches is performed by the differentiable spatial transformation network: Differentiable spatial transformation network receives the transformation parameters output by the alignment network, performs spatial transformation registration on pathological image blocks through bilinear interpolation sampling, outputs the registered image and extracts multi-channel feature maps.

[0027] After image registration is completed, an adaptive weighted fusion module is used to eliminate stitching seams between image patches and optimize feature representation: such as Figure 3 As shown, the adaptive weighted fusion module is used to eliminate seams between image patches. The registered multi-channel feature map (dimensions H, W, C, where C = N × 3) is input into the adaptive weighted fusion module. First, global average pooling (GAP) and global max pooling (GMP) are performed on the input feature map to obtain two channel description vectors with dimensions 1, 1, C. Then, the two vectors are added element-wise, and the fused vector is input into a two-layer multi-layer perceptron (MLP). The first layer of the MLP compresses the dimension from C to C / 16 using the ReLU activation function, and the second layer restores the dimension from C / 16 to C using the Sigmoid activation function, generating channel weight coefficients ranging from (0, 1). Those skilled in the art will understand that the compression ratio can be adjusted according to the number of channels C, for example, set to C / 8 or C / 32. Finally, the generated channel weight coefficients are multiplied channel-by-channel by the original input feature map to output the weighted fused feature map.

[0028] The weighted and fused feature maps are input into a multi-scale fusion network to achieve high-quality reconstruction of panoramic pathological images. like Figure 4As shown, the multi-scale fusion network adopts the U-Net architecture for reconstructing high-quality panoramic images. Its specific configuration is as follows: The encoder consists of four downsampling stages, each containing two 3×3 convolutional layers (stride 1, padding 1), a batch normalization layer, and a ReLU activation layer, followed by a 2×2 max-pooling layer, with channel numbers of 64, 128, 256, and 512 respectively; the decoder consists of four upsampling stages, each containing a 2×2 deconvolutional layer, two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation layer, with channel numbers of 512, 256, 128, and 64 respectively; four skip connections are set up to concatenate the outputs of each stage of the encoder with the corresponding inputs of the decoder in the channel dimension. The output layer is a 1×1 convolutional layer followed by a Tanh activation function, outputting the stitched panoramic pathological image (RGB three channels).

[0029] Before training the model, a dedicated training dataset adapted for pathological image stitching tasks needs to be constructed: like Figure 5 As shown, high-resolution panoramic images (WSI) of tissue sections from breast cancer, lung cancer, and gastric cancer were acquired using a digital tissue slicer. With an overlap factor of 20%, each panoramic image was divided into a 5×5 grid (25 image blocks), each block measuring 1024×1024 pixels. To simulate mechanical errors and imaging differences during real-world scanning, geometric transformations (rotation angle range -4° to 4°, translation range ±10 pixels), flipping, 90° rotation, and HSV color space dithering (H±10%, S±20%, V±15%) were randomly applied to each image block. The transformation parameters (rotation, translation) for each image block were recorded as labels for supervised learning. This method was used to construct 6000 training pairs and 1500 validation pairs.

[0030] To achieve the aforementioned image fusion effect, the overall network model of this invention is trained using a strategy combining phased independent training and joint training. The specific training process is as follows: First, with the parameters of the adaptive weighted fusion module and the multi-scale fusion network fixed, the alignment network is trained independently using the Adam optimizer to construct an accurate transformation parameter prediction model. The initial learning rate is 1×10⁻⁶. -4 Cosine annealing attenuation is used, and the loss function is the sum of the mean squared errors (MSE) of the transformation parameters. The training is performed for 300 epochs.

[0031] Then, with the already trained alignment network parameters fixed, the adaptive weighted fusion module and the multi-scale fusion network are jointly trained to construct a high-quality image fusion model. The optimizer is Adam, with an initial learning rate of 5×10⁻⁶. -5The loss function is halved every 100 epochs. It is a weighted combination of pixel-level MSE loss and feature-level perceptual loss, with weights λ1=1.0 and λ2=2.0. The perceptual loss is calculated based on the feature output of layer 16 of the pre-trained VGG-19 network. The entire training process lasts 200 epochs. Gradients are backpropagated along the entire chain from the multi-scale fusion network, adaptive weighted fusion module, differentiable space transformation network to alignment network, completing the full-chain optimization training of the overall model.

[0032] To verify the splicing accuracy and processing efficiency of the method of the present invention, multiple sets of quantitative performance tests and comparative experiments were conducted: like Figure 6 As shown, the performance of the method of this invention was verified on a test set containing 2000 test samples, and compared with existing methods such as direct splicing, AutoStitch, Lucas-Kanade, and DeTone. The performance of each method is shown in Table 1: Table 1. Comparison of quantitative performance of this invention with other splicing methods

[0033] The experimental data above show that the present invention is significantly superior to other comparative methods in terms of splicing quality (PSNR, SSIM) and processing speed.

[0034] Using a USAF1951 resolution test plate and 0.5μm fluorescent microspheres, the splicing error was measured by laser confocal microscopy. On 50 test samples, the registration accuracy of this invention was 0.42±0.08μm, which meets the sub-pixel level (<0.5μm) accuracy requirement.

[0035] like Figure 7 As shown, Figure 7 (a) is a bar chart comparing the peak signal-to-noise ratio (PSNR) of the splicing results of different experimental groups. Figure 7 (b) A bar chart comparing the structural similarity (SSIM) of the splicing results from different experimental groups. The module configurations and corresponding performance indicators of the four experimental groups are shown in Table 2: Table 2 Comparison of module configurations and splicing performance in different experimental groups of the present invention

[0036] Experimental group A uses the traditional SIFT feature matching method, combined with direct averaging fusion and Poisson fusion; experimental group B uses the alignment network of this invention, combined with direct averaging fusion and the multi-scale fusion network of this invention; experimental group C uses the alignment network of this invention, combined with the adaptive weighted fusion module of this invention and direct convolutional fusion; experimental group D uses the complete technical solution of this invention, integrating the alignment network, the adaptive weighted fusion module, and the multi-scale fusion network. Ablation experiments show that experimental group D (complete), i.e., the complete technical solution of this invention, achieves a PSNR of 31.26 dB and an SSIM of 0.834, both being the best results among all groups, fully verifying the superior technical performance of the synergistic cooperation of the various modules of this invention.

[0037] The method of this invention exhibits excellent practical application results in pathological scanning equipment with different configurations: Deploying the method of this invention on an entry-level scanner (≤100,000 RMB) can effectively compensate for its ±50μm mechanical positioning error, and the quality of the generated panoramic image is not statistically different from that of high-end equipment (≥200,000 RMB) (p>0.05). On high-end scanners, the method of this invention can reduce the image block overlap rate to 15%, while ensuring stitching quality (PSNR>29.5dB) and shortening the scanning time by 40%. By transmitting transformation parameters (approximately 0.29KB) to replace the original image blocks (approximately 75MB), the bandwidth requirement is reduced by 99.996%, and high-definition images can be quickly reconstructed at the receiving end based on the parameters.

[0038] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A deep learning-based method for stitching together pathological images, characterized in that, The method includes the following steps: S1: Stack multiple pathological image blocks along the channel dimension to form a multi-channel input tensor; S2: The multi-channel input tensor is input into the alignment network, with the objective function being to minimize the sum of reprojection errors of all pathological image blocks relative to the central reference block. The transformation parameters of all pathological image blocks are predicted at once through the interaction of features within the network. S3: Input the transformation parameters into a differentiable spatial transformation network, and perform spatial transformation registration on the pathological image blocks through bilinear interpolation sampling to obtain the registered image; S4: Input the registered image into the adaptive weighted fusion module, and obtain the weighted fusion feature map through channel statistical feature extraction, weight generation and feature recalibration; S5: Input the weighted fusion feature map into the multi-scale fusion network, and after encoder downsampling, decoder upsampling and multi-scale feature stitching reconstruction, output a panoramic pathological image with RGB three channels.

2. The pathological image stitching method based on deep learning as described in claim 1, characterized in that, The alignment network is provided with a cross-layer feature fusion unit, which includes two specific cross-layer skip connections. The first skip connection connects the second set of convolutional layers in the first feature extraction stage of the alignment network with the first set of convolutional layers in the second feature extraction stage. The second skip connection connects the first set of convolutional layers in the second feature extraction stage with the second set of convolutional layers in the second feature extraction stage.

3. The pathological image stitching method based on deep learning as described in claim 1, characterized in that, The processing procedure of the adaptive weighted fusion module is as follows: Global average pooling and global max pooling are performed on the feature maps of the registered images respectively to obtain two channel description vectors, which are then added element by element. The fused vector is input into a two-layer multilayer perceptron to generate channel weight coefficients, the channel weight coefficients being in the range of (0,1). The channel weight coefficients are multiplied one by one with the original input feature map to complete feature recalibration and output a weighted fused feature map.

4. The pathological image stitching method based on deep learning as described in claim 1, characterized in that, The multi-scale fusion network is an encoder-decoder structure based on the U-Net architecture.

5. The deep learning-based pathological image stitching method as described in claim 4, characterized in that, The multi-scale fusion network is configured with four skip connection units, which concatenate the outputs of each stage of the encoder with the inputs of the corresponding stage of the decoder in the channel dimension.

6. The pathological image stitching method based on deep learning as described in claim 4, characterized in that, The output layer of the multi-scale fusion network is processed by 1×1 convolution and activation function to output a panoramic pathological image with RGB three channels.

7. The pathological image stitching method based on deep learning as described in claim 1, characterized in that, The transformation parameters include rotation angle, horizontal translation amount, and vertical translation amount.

8. A pathological image stitching system based on deep learning, characterized in that, The method for implementing the deep learning-based pathological image stitching method as described in any one of claims 1 to 7 includes an input module, an alignment network module, a differentiable spatial transformation network module, an adaptive weighted fusion module, and a multi-scale fusion network module connected in sequence. The input module is used to stack multiple pathological image blocks along the channel dimension to form a multi-channel input tensor; The alignment network module is used to receive multi-channel input tensors and output transformation parameters for all pathological image blocks with the goal of minimizing the sum of reprojection errors. The differentiable spatial transformation network module is used to perform bilinear interpolation sampling and registration on pathological image blocks based on transformation parameters, and output the registered image. The adaptive weighted fusion module is used to extract channel statistical features, generate weights, and recalibrate features of the registered image, and output a weighted fusion feature map. The multi-scale fusion network module is used to extract, stitch and reconstruct multi-scale features from the weighted fusion feature map, and output a panoramic pathological image with RGB three channels.

9. A method for training a pathological image stitching model based on deep learning, characterized in that, The alignment network, adaptive weighted fusion module, and multi-scale fusion network used to train the deep learning-based pathological image stitching method according to any one of claims 1-7 specifically include: A phased training strategy is adopted: first, the alignment network is trained independently, and then the adaptive weighted fusion module and the multi-scale fusion network are trained together with the alignment network parameters fixed.

10. The method for training a pathological image stitching model based on deep learning as described in claim 9, characterized in that, During training, a weighted combination of mean squared error loss and perceptual loss is used as the loss function, and the perceptual loss is calculated based on the feature output of the pre-trained VGG network. During training, a gradient backpropagation path is constructed based on the output panoramic pathological image. The gradient backpropagation path propagates back along the entire link from the multi-scale fusion network, the adaptive weighted fusion module, the differentiable spatial transformation network to the alignment network.

Images