Pathological image multi-layer fusion method and system based on self-supervised training and continuous depth field

Abstract

This invention proposes a multi-layer fusion method and system for pathological images based on self-supervised training and continuous depth field. This method addresses the problems of label dependence and loss of high-frequency biological details caused by neural network spectral bias in existing fluorescence microscopy image fusion. It uses a three-dimensional convolutional neural network to encode spatial and depth joint features of multi-layer image sequences, and decouples the predicted continuous depth field topographic map and foreground confidence mask in parallel. A physically differentiable rendering layer calculates fusion weights based on the continuous depth field to complete weighted fusion, and combines mask soft suppression to output a full depth-of-field image. A self-supervised joint loss function, including high-frequency focusing loss, depth smoothing loss, and mask constraint loss, is constructed based on physical optics priors. The physical correlation between the input sequence and the fused image is used as a supervision signal for closed-loop optimization. High-fidelity fusion can be achieved without manual annotation, effectively preserving the details of minute biological structures.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and medical image processing technology, and in particular to a method and system for multi-layer fusion of pathological images based on self-supervised training and continuous depth field. Background Technology

[0002] In the field of modern biomedical fluorescence microscopy, researchers typically employ microscope objectives with extremely high numerical apertures to achieve higher optical resolution. However, according to optical diffraction theory, a high numerical aperture inevitably results in an extremely shallow effective depth of field (DoF) for the microscopy imaging system, usually only at the micrometer or sub-micrometer level. Since samples such as biological tissues, cell nuclei, or organoids have a certain physical thickness in three-dimensional space, a single exposure can only clearly capture a very thin focal plane region, while other biological structures outside the focal plane will be severely blurred by the point spread function (PSF) of the objective.

[0003] To obtain high-resolution images of complete cells or tissues with full depth of field, the commonly used technical solution is to drive the stage or objective lens along the optical axis with a fixed mechanical step size to scan layer by layer, capturing a series of locally focused image stacks (Z-stacks). Then, relying on computer image fusion algorithms, the effective focusing information of each layer is extracted from the multi-layer Z-stack images and merged into a globally clear two-dimensional image with full depth of field.

[0004] In existing image fusion algorithms, deep learning-based fusion methods mainly fall into two categories: supervised learning methods, which train neural networks by constructing a "perfect focus-out-of-focus" paired dataset; and traditional image processing methods, which perform fusion based on multi-scale decomposition strategies such as Laplacian pyramids and wavelet transforms. Supervised learning methods typically employ an encoder-decoder architecture, encoding multi-layered input images into latent space features and then predicting the optimal focusing layer or pixel-level fusion weights through regression or classification. While these methods improve fusion quality to some extent, they still have technical limitations.

[0005] Therefore, there is an urgent need for a method and system for multilayer fusion of pathological images based on self-supervised training and continuous depth field to solve the problems existing in the current technology. Summary of the Invention

[0006] This invention provides a method and system for multi-layer fusion of pathological images based on self-supervised training and continuous depth field. It addresses the problems of existing deep learning fusion technology, which heavily relies on hard-to-obtain perfect focus pairing labels and is limited by the inherent spectral bias of neural networks, which easily erases minute high-frequency biological structural features during forward propagation, resulting in physical distortion and loss of detail in the final image.

[0007] The core technology of this invention is to decouple the prediction of continuous depth field and foreground mask by performing joint spatial and depth feature encoding on multidimensional tensors, inputting them into physically differentiable rendering layers for weighted fusion and soft suppression, and constructing a joint prior loss function in combination with physical optics laws to achieve self-supervised closed-loop update without manual labeling.

[0008] In a first aspect, the present invention provides a method for multilayer fusion of pathological images based on self-supervised training and continuous depth field, the method comprising the following steps:

[0009] Obtain discrete multilayer image sequences of pathological samples and construct multidimensional image tensors from these discrete multilayer image sequences; A three-dimensional convolutional neural network is used to perform joint spatial and depth feature encoding on multidimensional image tensors to extract high-dimensional latent space features. The high-dimensional latent space features are reduced in dimension and reshaped into two-dimensional feature maps. The two-dimensional feature maps are then input into the decoupled prediction branches in parallel to predict continuous depth field topographic maps and foreground confidence masks, respectively. A physically differentiable rendering layer is constructed, and a continuous depth field topographic map, a foreground confidence mask, and a discrete multi-layer image sequence are input into the physically differentiable rendering layer. Inside the physically differentiable rendering layer, the continuous differentiable fusion weights corresponding to each physical imaging layer are calculated based on the depth values ​​contained in the continuous depth field topographic map. The discrete multi-layer image sequence is then forward-weighted fused using the continuous differentiable fusion weights to obtain an initial panoramic depth image. The initial panoramic depth image is then subjected to soft suppression processing using the foreground confidence mask, and the soft-suppressed image is output as the panoramic depth fusion image. Based on physical optics priors, a self-supervised joint loss function is constructed, which includes high-frequency focusing loss and depth terrain smoothing loss. The high-frequency focusing features of the panoramic depth fusion image and the depth continuity features of the continuous depth field topographic map are used as supervision signals without manual labels. The weights of the 3D convolutional neural network and the prediction branch are updated in a self-supervised closed loop through error backpropagation.

[0010] Furthermore, before performing joint feature encoding on the multidimensional image tensor, the method also includes: performing an axis transposition operation on the multidimensional image tensor to replace the channel dimension with the physical depth dimension to reconstruct the three-dimensional tensor; the three-dimensional convolutional neural network includes a feature encoder composed of multiple layers of three-dimensional convolutional layers connected in series, and each three-dimensional convolutional layer is cascaded with a three-dimensional batch normalization layer and an activation function.

[0011] Furthermore, the high-dimensional latent space features are dimensionality reduced and reshaped into a two-dimensional feature map. Specifically, this includes: vertically flattening and horizontally merging the high-dimensional latent space features in the feature channel dimension and physical depth dimension to obtain a two-dimensional feature map; predicting a continuous depth field terrain map specifically includes: performing a step-wise dimensionality reduction mapping on the two-dimensional feature map through cascaded two-dimensional convolutional layers, and constraining the value of the output matrix within a continuous open interval through an activation function in the last layer, and then multiplying the whole by a depth scalar to output a continuous depth field terrain map.

[0012] Furthermore, based on the depth values ​​contained in the continuous depth field topographic map, the continuously differentiable fusion weights corresponding to each physical imaging layer are calculated, specifically including: Based on the sub-pixel physical focal plane predicted by the continuous depth field topographic map, the relative distance between each physical imaging layer and the sub-pixel physical focal plane is calculated; based on the relative distance, the fusion weights are assigned using the Gaussian distribution formula, and the fusion weights are normalized along the depth axis to obtain continuously differentiable fusion weights.

[0013] Furthermore, a foreground confidence mask is used to perform soft suppression processing on the initial full-view depth image, specifically including: The initial full-depth image is linearly reduced based on a preset retention ratio and protected by baseline overlay using a foreground confidence mask to smoothly transition background noise areas and block the generation of dead black noise.

[0014] Furthermore, the high-frequency focusing loss in the self-supervised joint loss function extracts the second derivative information by performing spatial convolution on the panoramic depth fusion image, and introduces a temperature scaling factor to amplify the gradient response of edge features, and is evaluated within the activation region of the foreground confidence mask; the depth terrain smoothing loss uses the total variation of the first-order spatial difference calculated on the continuous depth field terrain map as a regularization constraint.

[0015] Furthermore, the self-supervised joint loss function also includes mask decoupling physical constraint loss; the mask decoupling physical constraint loss includes a sparsity penalty term and a direction guidance loss; wherein, the sparsity penalty term is constructed by applying a norm penalty to the foreground confidence mask; the direction guidance loss is constructed by extracting the maximum projection matrix of the discrete multi-layer image sequence in the depth direction, generating coarse-grained physical soft labels through threshold offset, magnification and clamping processing, and calculating the mean square error between the foreground confidence mask and the coarse-grained physical soft labels.

[0016] Secondly, the present invention provides a pathological image multilayer fusion device based on self-supervised training and continuous depth field, comprising: The multidimensional tensor construction module is used to obtain discrete multi-layer image sequences of pathological samples and construct multidimensional image tensors from these sequences. The joint feature encoding module is used to perform joint spatial and depth feature encoding on multidimensional image tensors using a three-dimensional convolutional neural network to extract high-dimensional latent space features. The dual-branch prediction module is used to reduce the dimensionality of high-dimensional latent space features and reshape them into two-dimensional feature maps. The two-dimensional feature maps are then input into the decoupled prediction branches in parallel to predict the continuous depth field topographic map and the foreground confidence mask, respectively. The physically differentiable rendering module is used to construct the physically differentiable rendering layer. It inputs a continuous depth field topographic map, a foreground confidence mask, and a discrete multi-layer image sequence into the physically differentiable rendering layer. Inside the physically differentiable rendering layer, it calculates the continuous differentiable fusion weights corresponding to each physical imaging layer based on the depth values ​​contained in the continuous depth field topographic map. It then uses the continuous differentiable fusion weights to perform forward weighted fusion on the discrete multi-layer image sequence to obtain an initial full-depth image. Finally, it uses the foreground confidence mask to perform soft suppression processing on the initial full-depth image and outputs the soft-suppressed image as the full-depth fusion image. The self-supervised optimization module is used to construct a self-supervised joint loss function based on physical optics priors, which includes high-frequency focusing loss and depth terrain smoothing loss. The high-frequency focusing features of the panoramic depth fusion image and the depth continuity features of the continuous depth field topographic map are used as supervision signals without manual labels. The weights of the 3D convolutional neural network and the prediction branch are updated in a self-supervised closed loop through error backpropagation.

[0017] Thirdly, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to execute the above-described method for multilayer fusion of pathological images based on self-supervised training and continuous depth field.

[0018] Fourthly, the present invention provides a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including the above-described method for multilayer fusion of pathological images based on self-supervised training and continuous depth field.

[0019] The main contributions and innovations of this invention are as follows: 1. This invention encapsulates the classic optical Gaussian fusion process into a fully differentiable mathematical rendering layer, enabling the neural network to achieve self-supervised convergence without any manually labeled focus reference images, relying solely on the physical correlation between the final output panoramic depth fusion image and the input image sequence. This not only significantly reduces the cost of dataset construction but also ensures the physical fidelity of the algorithm in real microscopic imaging scenarios.

[0020] 2. This invention constructs a joint physical prior loss function that introduces high-frequency focusing loss and performs penalty assessment within the activation region of the foreground mask, forcing the model to follow the laws of physical optics to capture the highest frequency sub-pixel level focusing edges, thereby perfectly preserving extremely fine high-frequency biological structural information such as organelles and dendritic spines, and significantly improving the resolution and microscopic feature extraction capability of the fused image.

[0021] 3. On the one hand, this invention uses depth terrain smoothing loss to explicitly regularize the predicted continuous depth map, strongly penalizing spatial discontinuities in the depth map caused by local noise, and completely eliminating mosaic artifacts in the fused image; on the other hand, in the fusion stage, it combines the predicted foreground confidence mask with soft suppression processing of a preset retention ratio, making the transition of background noise areas extremely smooth, fundamentally blocking the generation of dead black noise points, and comprehensively improving the visual observation quality of the global image.

[0022] Details of one or more embodiments of the present invention are set forth in the following drawings and description, so that other features, objects and advantages of the invention will be more readily understood. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a pathological image multilayer fusion method based on self-supervised training and continuous depth field according to an embodiment of the present invention; Figure 2 These are comparison diagrams showing the effects of embodiments of the present invention; Figure 3 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0025] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0026] This invention relates to a multi-layer fusion method for pathological images based on self-supervised training and continuous depth fields. For example... Figure 1 As shown, the overall algorithm flow includes, in sequence: multidimensional image tensor construction, 3D convolutional space-depth joint feature encoding, dual-branch decoupled prediction (continuous depth field prediction and foreground mask prediction), physically differentiable rendering fusion, and closed-loop optimization of the self-supervised joint loss function based on physical optics priors. The following detailed explanations of each step are provided with specific examples.

[0027] Example 1 This embodiment provides a multi-layer fusion method for pathological images based on self-supervised training and continuous depth fields. This method runs on a heterogeneous parallel computing platform equipped with Intel Xeon series CPUs and GPUs such as NVIDIA A100 series GPUs, and is used for panoramic deep fusion of fluorescence microscopy Z-stack images of mouse brain slices. Figure 1 As shown, the complete execution flow of this embodiment includes the following steps: Step S1: Obtain discrete multilayer image sequences of pathological samples and construct multidimensional image tensors from the discrete multilayer image sequences.

[0028] Specifically, the stage of the laser confocal microscope is driven by the control system to perform layer-by-layer scanning along the optical axis Z with a fixed step size of 1 μm, acquiring a single-channel fluorescence image sequence (Z-stack) of mouse brain slices. This sequence comprises N=10 discrete slices, each with a resolution of 512×512 pixels. At the hardware execution level, the CPU is responsible for low-level file reading, normalized data preprocessing, and multi-process data flow control. The CPU normalizes the pixel grayscale values ​​of the input single-channel discrete image sequence to the [-1,1] range to improve the convergence stability of the subsequent deep learning network. The normalized data is then pushed in batches into the high-speed memory space of the GPU via the PCIe bus. In the GPU memory, this image stack is constructed as a high-order multidimensional image tensor conforming to the deep learning framework, with its initial dimensions represented as:

[0029] Where B represents the batch size of a single parallel processing operation. In this embodiment, since the three-dimensional convolution operation consumes a lot of computing resources, the batch size B is set to a fixed value of 2 to avoid GPU memory overflow. N represents the total number of discrete layers in the Z-axis physical imaging (N=10 in this embodiment). C represents the number of feature channels of the image (C=1 for a single-channel fluorescence microscopy image). H and W represent the matrix height and width of a single-layer two-dimensional image, respectively (H=512, W=512 in this embodiment).

[0030] Step S2: Use a three-dimensional convolutional neural network to perform joint spatial and depth feature encoding on the multidimensional image tensor to extract high-dimensional latent space features.

[0031] Before performing convolution calculations, the GPU scheduling core first performs an axis swap operation on the multidimensional image tensor output from step S1, replacing the feature channel dimension C with the physical depth dimension N, thereby reconstructing a stereo tensor adapted to the 3D convolution input format:

[0032] In this embodiment, the specific dimensions of the 3D tensor are: .

[0033] Subsequently, the reconstructed 3D tensor is fed into a feature encoder consisting of three cascaded 3D convolutional layers. The specific configuration of this encoder is as follows: The first 3D convolutional layer (Conv3D_1): The input feature channel number is 1, and the output feature channel number is mapped to the basic feature dimension of 16. The spatial size of the convolutional kernel is set to... To prevent resolution degradation at spatial and depth edges, the padding and stride in the 3D space are set to 1. After convolution, a 3D batch normalization layer (BatchNorm3d) is applied to stabilize the latent space feature distribution. The activation function is a LeakyReLU function with a fixed negative slope of 0.2. The output feature dimension of this layer is... .

[0034] The second 3D convolutional layer (Conv3D_2) has 16 input feature channels and doubles the number of output feature channels to 32. The spatial size of the convolutional kernel is also set to... The padding is 1, and the stride is 1. This is followed by a BatchNorm3d layer and a LeakyReLU activation function with a negative slope of 0.2. The output feature dimension of this layer is... .

[0035] The third 3D convolutional layer (Conv3D_3): The input feature channels are 32. To control the number of model parameters and achieve feature compression and fusion, the output feature channels are reduced to 16. The padding is 1, and the stride is 1. This is followed by a BatchNorm3d layer and a LeakyReLU activation function with a negative slope of 0.2. The output feature dimension of this layer is... .

[0036] The physical mechanism of this step is as follows: During the sliding process, the 3D convolutional kernel not only extracts the morphological and topological features of cell nuclei or luminescent structures in the two-dimensional spatial plane, but more importantly, it captures the light intensity diffusion and defocusing degradation gradient along the Z-axis dimension at the same spatial coordinate point (x,y). Mathematically, this is equivalent to implicitly fitting the 3D point spread function (PSF) of the optical imaging system, thus providing a spatial-depth correlation feature description with a strong physical background for subsequent continuous focal plane prediction.

[0037] After three layers of 3D-CNN encoding, the output high-dimensional latent space feature tensor has dimensions of [B, 16, N, H, W], specifically [2, 16, 10, 512, 512] in this embodiment.

[0038] Step S3: Reduce the dimensionality of the high-dimensional latent space features and reshape them into a two-dimensional feature map. Input the two-dimensional feature map into the decoupled prediction branch in parallel to predict the continuous depth field topographic map and the foreground confidence mask respectively.

[0039] The GPU computing core vertically flattens the feature channel dimension (size 16) and physical depth dimension (size N=10) of the five-dimensional latent space feature tensor output in step S2 and then horizontally merges them, thereby reducing the dimensionality of the tensor and reshaping it into a standard two-dimensional feature map, whose dimensions are represented as follows:

[0040] In this embodiment, the specific dimensions of the two-dimensional feature map are: .

[0041] Subsequently, the reshaped 2D feature map is input in parallel into two decoupled prediction head branches: (a) Continuous Depth Prediction Branch: This branch consists of cascaded two-dimensional convolutional layers, with the number of feature channels decreasing sequentially in a stepwise manner. The mapping path is as follows: (In this embodiment, i.e.) The kernel size of each 2D convolutional layer is . The padding is set to 1, and the intermediate layers use LeakyReLU (negative slope 0.2) for non-linear mapping. After the final convolutional output, the Sigmoid activation function is used to strictly constrain the values ​​of the output matrix to a continuous open interval (0,1). Then, the matrix is ​​multiplied by a scalar (N-1) (in this embodiment, ...). The final output is a single-channel continuous depth field topographic map. (In this embodiment, i.e.) Each floating-point value in this matrix (In this embodiment, [0, 9]) directly represents the theoretically optimal sub-pixel physical focal plane at that spatial coordinate point. The meaning of "sub-pixel level" here is that the predicted value can be any continuous floating-point number between 0 and N-1 (e.g., 4.3), rather than being limited to discrete integer indices (e.g., 4 or 5), thereby achieving a more refined focal plane estimation than the traditional layer-by-layer selection.

[0042] (ii) Foreground confidence mask prediction branch: This branch runs in parallel with the depth branch and has no parameter sharing or information interaction. The feature channel mapping path is ( (In this embodiment, i.e.) The final convolutional layer uses the Sigmoid activation function to output a two-dimensional continuous soft probability mask. (In this embodiment, i.e.) The value of each pixel in the matrix. This represents the network's confidence that a valid fluorescent signal (foreground cell structure) exists at that location.

[0043] There is no parameter sharing or information exchange between the two branches, achieving decoupled independent prediction, so that depth estimation and foreground determination can be optimized independently without interfering with each other.

[0044] Step S4: Input the continuous depth field topographic map, foreground confidence mask, and discrete multi-layer image sequence into the physically differentiable rendering layer. Calculate the continuous differentiable fusion weights based on the continuous depth field topographic map to perform forward weighted fusion, and combine the foreground confidence mask for soft suppression processing to output a panoramic depth fusion image.

[0045] This embodiment constructs the fusion process of multifocal images as a fully continuously differentiable mathematical rendering layer, so that the network can perform end-to-end self-supervised learning through the loss function.

[0046] The forward propagation calculation process is as follows: The GPU generates a one-dimensional discrete integer grid sequence in video memory. (In this embodiment, i.e.) (corresponding to the index of 10 physical imaging layers). For each spatial pixel coordinate (x, y) in the image, the rendering layer uses the continuous floating-point depth value predicted in step S3. The relative distance between each physical imaging layer z and the theoretically optimal focal plane is calculated, and then Gaussian fusion weights conforming to a normal distribution are assigned. :

[0047] Regarding the standard deviation of key hyperparameters The setting is based on: In this embodiment, the standard deviation of the Gaussian fusion weights is used. The standard deviation is set to a fixed physical empirical constant of 0.5. The mathematical basis for this is that, since the discrete layer indices along the Z-axis have been normalized to a grid with a step size of 1.0, setting the standard deviation to 0.5 ensures that the effective cutoff boundary of the Gaussian kernel just covers the adjacent physical layers before and after the current predicted focus. This provides sufficient continuity interpolation bandwidth for smooth aliasing while avoiding excessive participation of out-of-focus, blurred pixels far from the focal plane in the fusion process due to an excessively large standard deviation, thus guaranteeing the sharpness of the reconstruction from a physical optics perspective.

[0048] Subsequently, Softmax normalization is performed on the above weights along the depth axis to obtain normalized weights:

[0049] Among them, in the denominator This is a numerically stable term used to prevent numerical overflow caused by a denominator of zero. This normalized weight is used to normalize the original input sequence. By performing weighted summation, the full depth-of-field image is obtained:

[0050] Finally, soft suppression is performed using the foreground confidence mask predicted in step S3, resulting in the final fused image:

[0051] in, To preserve the preset signal ratio constant, this embodiment preferably uses 0.8, which means preserving 20% ​​of the baseline color. By preserving 20% ​​of the baseline color, the transition in the background noise area is extremely smooth, fundamentally preventing the generation of dead black noise (black spots). In the pure background area ( The output value tends to A uniform background color of 0.2 instead of 0 avoids the unnatural black block artifacts that appear in the background area in traditional mask suppression methods.

[0052] Backpropagation gradient flow derivation: This rendering layer is "fully differentiable" because the entire forward chain consists of continuously differentiable basic functions (exponential functions, weighted summations, linear multiplication, and affine transformations are all continuously differentiable). When the backend loss function calculates the error gradient with respect to the output image... Then, according to the chain rule for differentiating multivariable composite functions, the gradient flow will propagate forward losslessly to all learnable parameters. The complete gradient propagation chain is as follows: Among them, key depth predictor variables The differentiating operator has an explicit analytical expression:

[0053] In the above chain rule, For linear constant terms, The original input image values, For the Jacobian terms normalized by Softmax, Checked by Gauss The partial derivatives are given. Because the partial derivatives... With all the aforementioned intermediate terms continuously non-zero, the network can accurately perceive whether the currently predicted depth value should be fine-tuned upwards or downwards along the Z-axis to further reduce the total loss. Similarly, regarding the mask prediction variables... There are also no non-differentiable nodes on the gradient path. This differentiability design is the mathematical foundation for achieving self-supervised training.

[0054] Step S5: Construct a self-supervised joint loss function based on physical optics priors. Use the physical correlation between panoramic depth fusion images, continuous depth field topographic maps, foreground confidence masks, and discrete multi-layer image sequences as supervision signals. Update the network weights in a self-supervised closed loop through error backpropagation.

[0055] This embodiment drives model self-evolution by applying a joint loss function (PIC-Loss, Physical Information Constrained Loss) that conforms to physical optics priors during backpropagation, without manual labeling. The model is trained using the AdamW optimizer, with the initial learning rate strictly set to a fixed value. The total number of training iterations is determined based on the model's convergence. In this embodiment, training stops when the loss function value stabilizes. Total loss function... The specific mathematical structure and weighting coefficient allocation are as follows:

[0056] The following sections will provide a detailed explanation of each loss function.

[0057] (a) High-frequency focusing loss This loss term is based on the physical prior that "the physical surface of the object being focused possesses the maximum gradient energy." Specifically, the GPU calls a fixed... The Laplacian kernel is used to process the output image. Spatial convolution is performed to extract second derivative information, and its kernel matrix is:

[0058] To transform the optimization problem into a minimization problem, the Laplace response is negative, and a fixed temperature scaling factor is introduced. = 0.05 to amplify the gradient response of edge features. This loss term is evaluated only in the foreground mask activation region, and its mathematical expression is:

[0059] in, This indicates the Laplace operator kernel mentioned above. Spatial convolution operation, This represents the square of the pixel-wise absolute value of the Laplacian operator convolution output. This represents the average value over all spatial pixels. Temperature scaling factor. The value of 0.05 amplifies the magnitude of the Laplacian response, preventing gradient signals at minute edges from being submerged in the overall loss. This ensures the network can effectively perceive and preserve sub-pixel-level high-frequency biological structural details. The design of evaluating only the mask activation region avoids imposing meaningless high-frequency constraints on the non-signal background region.

[0060] (ii) Depth Smoothness Loss The loss term is based on the physical prior that "the surface depth of a physical sample is spatially continuous." The algorithm predicts the two-dimensional depth map. Calculate the total variation loss of the first-order spatial differences and assign fixed weights. =0.3. Its mathematical expression is:

[0061] This loss term, as an explicit regularization, strongly penalizes spatial discontinuities in the depth map caused by local noise, forcing the depth predictions of adjacent pixels to tend to transition smoothly, thereby eliminating mosaic artifacts or blocky depth jumps that may appear in the fused map.

[0062] (iii) Mask decoupling physical constraint loss and ): To prevent the network from passing through a mask To maliciously reduce the prediction to 1 The value of the mask constraint loss is introduced in this embodiment because a full 1 mask will cause all pixels to participate in the high-frequency loss assessment, which will cause the optimization direction to deviate from the actual focus requirement.

[0063] Sparsity penalty term The foreground mask is sparsified using the L1 norm, with fixed weight coefficients. = 0.1 control. Its mathematical expression is:

[0064] This penalty encourages the mask to tend towards sparseness, that is, to output high confidence only in regions where fluorescence signals actually exist, preventing the mask from degenerating into a trivial solution of all 1s.

[0065] Directional guidance loss Intensity extraction is performed using the two-dimensional maximum projection matrix of the input Z-stack in the depth direction. Specifically, this involves processing the input multi-layer image sequence. By taking the maximum value pixel by pixel along the Z-axis, we obtain the projection matrix P(x,y) that reflects the maximum fluorescence intensity at each spatial coordinate point:

[0066] After normalizing P(x,y) to the interval [0,1], we get Then subtract a threshold offset. (In this embodiment) The offset is adaptively determined based on the statistical distribution of the projection matrix P(x,y), for example, by taking the mean of P(x,y) as the offset reference, multiplying it by an amplification factor of 5, and finally clamping the result within the interval [0,1] to generate a coarse-grained physical soft label. Its mathematical expression is:

[0067] in This indicates that the values ​​are truncated to the [0,1] interval. Subsequently, the mean squared error (MSE) loss is used to guide the macroscopic direction of the foreground mask:

[0068] This loss term is calculated directly from the input data itself, requiring no manual annotation, and is part of the self-supervised signal. Its physical meaning is: pixels with the highest fluorescence intensity in the depth direction likely correspond to real biological structures and should be marked as foreground. This loss term provides a reasonable initialization direction and macroscopic constraints for mask prediction.

[0069] Under the joint constraint of PIC-Loss, the network performs error backpropagation through the AdamW optimizer, achieving self-supervised closed-loop updates of the 3D-CNN encoder and the weights of each prediction head. During training, the loss terms work collaboratively: The driver network retains high-frequency details. Constraining the spatial continuity of the depth field, To prevent mask degradation, It provides a macroscopic optimization direction. The combined effect of these four factors enables the network to converge to a physically reasonable fusion result under label-free conditions.

[0070] In this embodiment, the execution flow for the inference phase is as follows: It should be noted that steps S1 to S5 above fully describe the execution logic of the method in the training phase of this invention. After training is completed, the network parameters (including the 3D-CNN encoder weights, depth prediction branch weights, and mask prediction branch weights) are permanently saved. In the inference (deployment) phase, for a new input Z-stack image sequence, only steps S1 (tensor construction), S2 (feature encoding), S3 (two-branch prediction), and S4 (physically differentiable rendering fusion) need to be executed sequentially, i.e., the forward propagation path, without calculating the loss function or performing backpropagation. The output of the inference phase is the final panoramic depth-fused image. .

[0071] Alternative embodiments are also included: This embodiment illustrates the adaptability of the method of the present invention to different numbers of Z-stack layers. When the number of Z-stack layers is changed from N=10 to N=20, the overall algorithm flow remains unchanged, and only the following parameter adjustments are required: Because the number of layers N increases, the volume of the single-sample input tensor increases from... To double the original size (relative to N=10), the maximum batch size needs to be re-determined based on the GPU memory capacity; for example, the batch size B can be set to 4. In step S3, the final output scalar of the continuous depth prediction branch is adjusted from (N-1)=9 to (N-1)=19. In step S4, the discrete integer grid sequence is adjusted to... [0, 19], Gaussian kernel standard deviation The value remains at 0.5; the weight coefficients of each loss function in step S5 remain unchanged. These adjustments demonstrate that the core technical concept of this invention is not limited to a specific number of Z-stack layers, and those skilled in the art can flexibly adapt it according to actual imaging needs.

[0072] Furthermore, the core technical concept of this invention is not limited to a specific 3D convolutional encoder configuration. Those skilled in the art can adjust the number of 3D convolutional layers from three to two or four, or change the channel mapping path, based on actual computing resources and fusion accuracy requirements. Adjust to other configurations (e.g.) All of these should fall within the scope of protection of this invention.

[0073] See Figure 2The left side shows the processing result of the current industrial-grade fusion algorithm, and the right side shows the fusion result of the algorithm in this embodiment. Compared with the patch blurring and dead black noise effect caused by the conventional method on the left, the algorithm in this embodiment completely preserves the submicron-level fine fluorescent structures such as neuronal synapses and dendritic spines, with no obvious noise in the background area and significantly improved edge sharpness, which can be directly used for subsequent pathological quantitative analysis.

[0074] Example 2 Based on the same inventive concept as Embodiment 1, this embodiment provides a pathological image multilayer fusion device based on self-supervised training and continuous depth field, the device comprising: The multidimensional tensor construction module is used to obtain discrete multi-layer image sequences of pathological samples and construct multidimensional image tensors from these sequences. The joint feature encoding module is used to perform joint spatial and depth feature encoding on multidimensional image tensors using a three-dimensional convolutional neural network to extract high-dimensional latent space features. The dual-branch prediction module is used to reduce the dimensionality of high-dimensional latent space features and reshape them into two-dimensional feature maps. The two-dimensional feature maps are then input into the decoupled prediction branches in parallel to predict the continuous depth field topographic map and the foreground confidence mask, respectively. The physically differentiable rendering module is used to input the continuous depth field topographic map, the foreground confidence mask and the discrete multi-layer image sequence into the physically differentiable rendering layer. It calculates the continuous differentiable fusion weights based on the continuous depth field topographic map to perform forward weighted fusion, and performs soft suppression processing in combination with the foreground confidence mask to output a panoramic depth fusion image. The self-supervised optimization module is used to construct a self-supervised joint loss function based on physical optics priors. It uses the physical correlation between panoramic depth fusion images, continuous depth field topographic maps, foreground confidence masks, and discrete multi-layer image sequences as supervision signals, and performs self-supervised closed-loop updates of the weights of the 3D convolutional neural network and prediction branches through error backpropagation.

[0075] The specific execution logic of each of the above functional modules is completely consistent with steps S1 to S5 in Example 1. The parameter configuration, tensor dimension, and loss function composition of each module are the same as in Example 1, and will not be repeated here.

[0076] Example 3 This embodiment also provides an electronic device, see reference. Figure 3 It includes a memory 404 and a processor 402, wherein the memory 404 stores a computer program and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0077] Specifically, the processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement embodiments of the present invention.

[0078] Memory 404 may include a mass storage device for data or instructions. For example, and not limitingly, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to a data processing device. In a particular embodiment, memory 404 is non-volatile memory. In a particular embodiment, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0079] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0080] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the pathological image multilayer fusion methods based on self-supervised training and continuous depth field in the above embodiments.

[0081] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0082] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0083] Input / output device 408 is used to input or output information.

[0084] Example 4 This embodiment also provides a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including the pathological image multilayer fusion method based on self-supervised training and continuous depth field according to Embodiment 1.

[0085] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0086] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0087] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 1 Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0088] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0089] The above embodiments are merely illustrative of several implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A method for multilayer fusion of pathological images based on self-supervised training and continuous depth field, characterized in that, Includes the following steps: Obtain discrete multilayer image sequences of pathological samples, and construct the discrete multilayer image sequences into multidimensional image tensors; A three-dimensional convolutional neural network is used to perform joint spatial and depth feature encoding on the multidimensional image tensor to extract high-dimensional latent space features. The high-dimensional latent space features are reduced in dimension and reshaped into a two-dimensional feature map, and the two-dimensional feature map is input in parallel into the decoupled prediction branch to predict the continuous depth field topographic map and the foreground confidence mask respectively. Construct a physically differentiable rendering layer by inputting the continuous depth field topographic map, the foreground confidence mask, and the discrete multi-layer image sequence into the physically differentiable rendering layer; Inside the physically differentiable rendering layer, the continuously differentiable fusion weights corresponding to each physical imaging layer are calculated based on the depth values ​​contained in the continuous depth field topographic map. The continuously differentiable fusion weights are used to perform forward weighted fusion on the discrete multi-layer image sequence to obtain an initial panoramic depth image. The foreground confidence mask is used to perform soft suppression processing on the initial panoramic depth image. The soft-suppressed image is then output as the panoramic depth fusion image. Based on physical optics priors, a self-supervised joint loss function is constructed, which includes high-frequency focusing loss and depth terrain smoothing loss. The high-frequency focusing features of the panoramic depth fusion image and the depth continuity features of the continuous depth field terrain map are used as supervision signals without manual labels. The weights of the three-dimensional convolutional neural network and the prediction branch are updated in a self-supervised closed loop through error backpropagation.

2. The pathological image multilayer fusion method as described in claim 1, characterized in that, Before performing joint feature encoding on the multidimensional image tensor, the method further includes: performing an axis transposition operation on the multidimensional image tensor to replace the channel dimension with the physical depth dimension to reconstruct the three-dimensional tensor; the three-dimensional convolutional neural network includes a feature encoder composed of multiple layers of three-dimensional convolutional layers connected in series, and each three-dimensional convolutional layer is cascaded with a three-dimensional batch normalization layer and an activation function.

3. The pathological image multilayer fusion method as described in claim 1, characterized in that, The process of reducing and reshaping the high-dimensional latent space features into a two-dimensional feature map specifically includes: vertically flattening and horizontally merging the high-dimensional latent space features in the feature channel dimension and physical depth dimension to obtain the two-dimensional feature map; the process of predicting the continuous depth field terrain map specifically includes: performing a step-wise dimensionality reduction mapping on the two-dimensional feature map through cascaded two-dimensional convolutional layers, and constraining the value of the output matrix within a continuous open interval through an activation function in the last layer, and then multiplying the whole by a depth scalar to output the continuous depth field terrain map.

4. The pathological image multilayer fusion method as described in claim 1, characterized in that, The continuously differentiable fusion weights corresponding to each physical imaging layer are calculated based on the depth values ​​contained in the continuous depth field topographic map, specifically including: Based on the sub-pixel physical focal plane predicted by the continuous depth field topographic map, the relative distance between each physical imaging layer and the sub-pixel physical focal plane is calculated; based on the relative distance, fusion weights are assigned using a Gaussian distribution formula, and the fusion weights are normalized along the depth axis to obtain the continuously differentiable fusion weights.

5. The pathological image multilayer fusion method as described in claim 1, characterized in that, The initial full-depth image is subjected to soft suppression processing using the foreground confidence mask, specifically including: using the foreground confidence mask to perform linear reduction and baseline background color overlay protection on the initial full-depth image based on a preset retention ratio, so as to smoothly transition background noise areas and block the generation of dead black noise points.

6. The pathological image multilayer fusion method as described in claim 1, characterized in that, The high-frequency focusing loss in the self-supervised joint loss function extracts second-order derivative information by performing spatial convolution on the panoramic depth fusion image, and introduces a temperature scaling factor to amplify the gradient response of edge features, and is evaluated within the activation region of the foreground confidence mask; the depth terrain smoothing loss uses the total variation of the first-order spatial difference calculated on the continuous depth field terrain map as a regularization constraint.

7. The pathological image multilayer fusion method as described in claim 1, characterized in that, The self-supervised joint loss function further includes a mask decoupling physical constraint loss; the mask decoupling physical constraint loss includes a sparsity penalty term and a direction guidance loss; wherein, the sparsity penalty term is constructed by applying a norm penalty to the foreground confidence mask; the direction guidance loss is constructed by extracting the maximum projection matrix of the discrete multi-layer image sequence in the depth direction, generating coarse-grained physical soft labels through threshold offset, magnification and clamping processing, and calculating the mean square error between the foreground confidence mask and the coarse-grained physical soft labels.

8. A multilayer fusion device for pathological images based on self-supervised training and continuous depth field, characterized in that, include: A multidimensional tensor construction module is used to obtain discrete multi-layer image sequences of pathological samples and construct the discrete multi-layer image sequences into multidimensional image tensors; The joint feature encoding module is used to perform spatial and depth joint feature encoding on the multidimensional image tensor using a three-dimensional convolutional neural network to extract high-dimensional latent space features. The dual-branch prediction module is used to reduce the dimensionality of the high-dimensional latent space features and reshape them into a two-dimensional feature map, and input the two-dimensional feature map into the decoupled prediction branches in parallel to predict the continuous depth field topographic map and the foreground confidence mask, respectively. The physically differentiable rendering module is used to construct a physically differentiable rendering layer, and inputs the continuous depth field topographic map, the foreground confidence mask and the discrete multi-layer image sequence into the physically differentiable rendering layer; Inside the physically differentiable rendering layer, the continuously differentiable fusion weights corresponding to each physical imaging layer are calculated based on the depth values ​​contained in the continuous depth field topographic map. The continuously differentiable fusion weights are used to perform forward weighted fusion on the discrete multi-layer image sequence to obtain an initial panoramic depth image. The foreground confidence mask is used to perform soft suppression processing on the initial panoramic depth image. The soft-suppressed image is then output as the panoramic depth fusion image. The self-supervised optimization module is used to construct a self-supervised joint loss function based on physical optics priors, which includes high-frequency focusing loss and depth terrain smoothing loss. The high-frequency focusing features of the panoramic depth fusion image and the depth continuity features of the continuous depth field terrain map are used as supervision signals without manual labels. The weights of the three-dimensional convolutional neural network and the prediction branch are updated in a self-supervised closed loop through error backpropagation.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the pathological image multilayer fusion method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The readable storage medium stores a computer program, the computer program including program code for controlling a process to execute the process, the process including the pathological image multilayer fusion method according to any one of claims 1 to 7.

Images