Three-dimensional biological image missing region completion method and system based on deep learning
By constructing a mask autoencoder model for 3D data, and using deep learning technology to predict and reconstruct the structural consistency of missing regions in 3D biological images, the problem of difficulty in repairing complex and irregular missing regions in existing technologies is solved, and the spatial continuity and integrity of biological tissues are restored.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
- Filing Date
- 2026-03-09
- Publication Date
- 2026-06-23
Smart Images
Figure CN122265100A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and more specifically, to a method and system for completing missing regions in three-dimensional biological images based on deep learning. Background Technology
[0002] As life science research advances towards larger scales and higher resolutions, three-dimensional biological images are widely used in tissue structure analysis and pathological analysis. Due to the limitations of the microscopic imaging field of view, large biological samples are typically segmented and stitched together to reconstruct a complete three-dimensional structure. During the actual stitching process, factors such as sample preparation errors, imaging noise, and non-rigid deformation of biological tissues can easily lead to localized missing regions and impaired structural continuity in the stitched results. These missing regions are irregular in shape, spatially complex, and strongly correlated with surrounding tissue structures. These missing regions not only affect the integrity of the three-dimensional image but also adversely impact subsequent quantitative analysis and structural reconstruction. Therefore, effectively repairing missing regions generated during the stitching process of three-dimensional biological images has become an urgent technical problem to be solved.
[0003] On the one hand, traditional image processing methods typically use interpolation, smoothing, or rule-based filling to repair missing regions. For example, linear interpolation of neighboring pixels or filling based on local statistical features can achieve a continuous transition in image grayscale or intensity. These methods are simple to implement and have low computational cost, but they mainly rely on local information and are difficult to effectively characterize the morphological features of complex biological tissues in three-dimensional space. For biological image scenes with high requirements for structural continuity, they can easily cause structural blurring or breakage.
[0004] On the other hand, with the development of deep learning technology, some research has begun to focus on the automatic repair of missing regions in images, and has attempted to use neural networks to predict and reconstruct missing content. Existing related research is mostly based on convolutional neural networks, autoencoders, or generative models, which learn image context information to fill in missing regions, achieving some success in two-dimensional natural images or regular occlusion scenarios. However, these methods are mostly designed for two-dimensional images, or assume that the missing regions are regularly shaped and randomly distributed, making them difficult to adapt to the complex and irregular missing regions in three-dimensional biological images caused by slicing loss, imaging gaps, etc. Meanwhile, some research has introduced deep learning methods into the three-dimensional image stitching or registration process to improve the alignment accuracy between image blocks. However, these methods mainly focus on geometric alignment itself, and for pixel loss and structural interruptions that already exist after stitching, they usually still rely on traditional interpolation or simple filling methods, and have not yet formed a dedicated completion scheme for missing regions in three-dimensional biological images.
[0005] Existing technologies for repairing missing regions in 3D biological images mainly rely on traditional interpolation filling methods or deep learning models that borrow from 2D image completion. These technologies cannot fully utilize the spatial context information of 3D biological images and lack effective means to deal with complex and irregular missing regions caused by slice loss, imaging gaps, etc. They are difficult to restore the structural continuity and integrity of biological tissues in 3D space and cannot meet the actual needs of high-quality 3D biological image analysis and research.
[0006] In summary, with the development of deep learning technology in recent years, relatively mature research results have been achieved in image processing and analysis tasks in the fields of two-dimensional natural images and medical images. Related methods have performed well in tasks such as registration and segmentation of two-dimensional images. Some studies have also attempted to introduce deep learning methods into the three-dimensional biological image stitching process to improve the registration accuracy between image patches. However, in the scenario of three-dimensional biological image stitching, the data modality expands from two-dimensional to three-dimensional, and the missing regions generated in the stitching results have real three-dimensional spatial structural features. Existing technologies are difficult to apply directly, mainly in the following aspects: 1. Many existing deep learning methods are designed for two-dimensional images. Their network structure, feature representation and training methods are all based on two-dimensional data modalities, which are difficult to directly transfer to three-dimensional biological image stitching scenarios. They are also insufficient in modeling the continuity and structural relationships in three-dimensional space. 2. Existing deep learning-based 3D image processing methods mostly focus on the registration stage between image patches, paying attention to geometric alignment accuracy, but do not specifically model the 3D missing regions generated after stitching, and cannot effectively repair spatial holes and structural interruptions in the stitching results; 3. Existing methods for repairing missing regions mostly employ traditional techniques such as interpolation and smoothing, relying primarily on two-dimensional slicing or local neighborhood information for processing. When dealing with biological tissues with complex three-dimensional topological structures, these methods are prone to disrupting structural continuity and spatial consistency. Summary of the Invention
[0007] This invention provides a method and system for completing missing regions in three-dimensional biological images based on deep learning, which can effectively predict and complete missing regions to restore the structural continuity and integrity of biological tissues in three-dimensional space.
[0008] According to an embodiment of the present invention, a method for filling in missing regions of three-dimensional biological images based on deep learning is provided, comprising the following steps: Acquire three-dimensional biological images containing missing regions; The missing regions in 3D biological images are identified, and the 3D context information around the missing regions is extracted. A deep learning completion model for 3D data is constructed based on 3D context information. The deep learning completion model is used to predict and reconstruct the structural consistency of missing regions. The reconstructed 3D image data is output to obtain a continuous and complete 3D biological image in terms of spatial structure.
[0009] Furthermore, a deep learning-based completion model for 3D data is constructed based on 3D context information. This model is then used to predict and reconstruct the structural consistency of missing regions, including: A deep learning-based completion model is used to jointly model the missing region and its surrounding 3D context. The missing pixel values are predicted through self-supervised learning, thereby achieving structural consistency reconstruction.
[0010] Furthermore, the missing regions in the 3D biological images are identified, and the surrounding 3D context information is extracted, including: Missing regions are automatically identified using a threshold segmentation algorithm; Using the missing region as the center, extract the surrounding 3D context information from the 3D image.
[0011] Furthermore, the missing regions are automatically identified using a threshold segmentation algorithm, including: Calculate the grayscale gradient of the image. If the gradient is lower than a preset threshold and the volume of a continuous region exceeds the minimum threshold, it is marked as a missing mask.
[0012] Furthermore, extracting surrounding 3D context information from the 3D image, centered on the missing region, includes: Extract a pre-defined fixed-size image block, ensuring that the missing area is located at the center of the block. This extraction operation is achieved through coordinate offset, slicing sub-regions in a three-dimensional array.
[0013] Furthermore, the three-dimensional contextual information includes the grayscale of neighboring pixels, texture features, and prior knowledge of biological structures.
[0014] Furthermore, using deep learning completion models to predict and reconstruct structural consistency in missing regions includes: Based on the prediction results, a legality verification is performed. If the verification passes, the completion target is checked. If not, the context block is updated, and new blocks are extracted in a loop to continue the completion. If the verification fails or the target is achieved, the algorithm is terminated, and the completed pixels are fused into the original image to output a complete 3D biological image.
[0015] Furthermore, it also includes: A batch of training data was generated using a threshold- and morphologically based simulated damage algorithm as the training set for the deep learning completion model. The missing regions in the training set images were fixed as irregular shapes near the cross-section, and each missing block had an original ground truth mask.
[0016] Furthermore, it also includes: The completion output of the deep learning completion model is verified. At each completion step, the gray value of the current completed pixel is checked to see if it is reasonable. Inverse completion is used for verification: the model is used to complete the neighborhood from the completed pixel in reverse to judge whether the error between the reverse completion result and the forward completion is reasonable, so that the completion structure is always consistent with the original image context.
[0017] According to another embodiment of the present invention, a deep learning-based three-dimensional biological image missing region completion system is provided, comprising: The data input module is used to acquire three-dimensional biological images containing missing regions; The missing region identification module is used to identify missing regions in 3D biological images and extract the 3D context information around the missing regions. The completion module is used to build a deep learning completion model for 3D data based on 3D context information, and to use the deep learning completion model to predict and reconstruct the structural consistency of missing regions. The output module is used to output the reconstructed 3D image data to obtain a continuous and complete 3D biological image in spatial structure.
[0018] A storage medium storing program files capable of implementing any of the above-mentioned deep learning-based methods for filling in missing regions of 3D biological images.
[0019] A processor for running a program, wherein the program executes any of the above-mentioned deep learning-based method for filling in missing regions of 3D biological images.
[0020] The method and system for filling in missing regions of three-dimensional biological images based on deep learning in this embodiment of the invention target missing regions in three-dimensional biological images caused by imaging and slicing losses, and use the surrounding three-dimensional context information to achieve effective prediction and filling of missing regions, so as to restore the structural continuity and integrity of biological tissues in three-dimensional space. Attached Figure Description
[0021] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart of the overall solution. Figure 2 Examples of training set images; Figure 3 To complete the module's principle block diagram; Figure 4This is a schematic diagram to verify the mechanism. Detailed Implementation
[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0024] Three-dimensional imaging of large biological samples is limited by the field of view of microscopic imaging, usually requiring the sample to be segmented and the complete three-dimensional structure to be reconstructed. In the actual imaging and reconstruction process, due to factors such as losses during the slicing process, imaging noise, and the complexity of the sample's own structure, the reconstructed three-dimensional biological images are prone to problems such as local pixel loss and structural interruption, which disrupt the continuity and integrity of biological tissues in three-dimensional space, thus affecting subsequent quantitative analysis and biological research.
[0025] For the aforementioned missing regions, existing methods mostly employ interpolation or simple filling methods based on local rules for repair, which are insufficient to effectively restore the structural features of complex biological tissues in three-dimensional space. Furthermore, existing deep learning image completion methods are primarily designed for two-dimensional natural images or regular occlusion scenarios, lacking specificity for the spatial structure and contextual relevance of missing regions in three-dimensional biological images, making them difficult to directly apply to three-dimensional biological image missing repair tasks.
[0026] Therefore, in view of the above-mentioned shortcomings, the technical problem to be solved by the present invention is: how to effectively predict and complete the missing regions in three-dimensional biological images caused by imaging and slicing losses, so as to restore the structural continuity and integrity of biological tissues in three-dimensional space by utilizing the surrounding three-dimensional context information.
[0027] In view of the above-mentioned shortcomings, the object of the present invention is: This paper proposes a method and system for filling missing regions in 3D biological image stitching scenarios. It clearly states that 3D image data is the processing object, rather than a simple extension of 2D slices. It is native 3D and non-2D is extended to 3D. By using deep learning models to jointly model the missing regions in a 3D image section and their surrounding 3D context information, the structural continuity of biological tissues in 3D space can be restored. Without altering the existing 3D image stitching and registration process, this method effectively repairs damaged and missing areas before stitching, thereby enhancing the integrity and application value of 3D biological image stitching results.
[0028] Example 1 This invention provides a method for completing missing regions in 3D biological images based on deep learning. The technical solution includes: acquiring a 3D biological image containing missing regions; identifying the missing regions in the 3D biological image and extracting the surrounding 3D context information; constructing a deep learning completion model for 3D data, namely a mask autoencoder model (3D-MAE model), based on the 3D context information; using the deep learning completion model to predict and reconstruct the structural consistency of the missing regions; and outputting the reconstructed 3D image data, thereby obtaining a spatially continuous and complete 3D biological image.
[0029] This invention utilizes the spatial context information and structural continuity features of 3D biological images to construct a masked autoencoder (MAE) model for 3D data. This model effectively predicts and completes missing regions caused by slicing loss, imaging noise, etc., thereby restoring the continuity and integrity of biological tissues in 3D space. Specifically, this invention uses 3D biological images containing missing regions as input data, employs a 3D-MAE model to jointly model the missing regions and their surrounding 3D context, and predicts the missing pixel values through self-supervised learning to achieve structural consistency reconstruction. This method is independent of the specific shape or cause of the missing regions and is applicable to 3D imaging scenarios of various biological samples, such as neural tissue or neuronal cell body slices.
[0030] For ease of understanding, the technical solution of the present invention will be described in detail with reference to the accompanying drawings. The drawings include: Figure 1 This is a flowchart of the overall solution. Figure 2 Examples of training set images; Figure 3 To complete the module's principle block diagram; Figure 4 To verify the principle of the mechanism, the technical implementation of each part will be explained in detail below, with corresponding text descriptions for each figure.
[0031] The process of this invention is as follows: Figure 1 As shown. Figure 1 This is a flowchart, from top to bottom, including the following boxes: "Input 3D biological image data", "Missing region identification and marking", "Extract 3D context information", "Input model prediction and completion pixels", "Verify completion result validity" (if the branch is "Yes", the output is fused; if "No", the parameters are adjusted and completion is restarted), "Completion target achieved" (if the branch is "No", the upper and lower pixels are updated and the loop continues; if "Yes", the loop terminates), and "Output completed 3D image". This flowchart uses a loop structure to represent the completion process, ensuring that the algorithm iterates until the termination condition is met.
[0032] The detailed technical implementation is as follows: First, acquire 3D biological image data containing missing regions, typically stored in TIFF or NIfTI format with pixel resolution at the micrometer level. Then, automatically identify missing regions using a threshold segmentation algorithm: calculate the image grayscale gradient; if the gradient is lower than a preset threshold (empirical value 0.05 times the average grayscale) and the volume of a continuous region exceeds a minimum threshold (set to 64 pixels), it is identified as a missing mask. A mask value of 1 indicates a missing pixel. Using the missing region as the center, extract the surrounding 3D context information from the 3D image: crop a fixed-size image block, such as a cube with sides of 4 pixels, ensuring the missing region is centered within the block. This cropping operation is achieved through coordinate offset, i.e., slicing a sub-region within the 3D array. , where d is the half-side length (usually fixed at 2), and the context information includes the grayscale of neighboring pixels, texture features, and biological structural priors (such as tissue continuity).
[0033] The extracted 3D context block and missing mask are input into a pre-trained mask autoencoder model, which outputs the predicted completed pixel values (see the completion module principle for details). Based on the prediction results, a validity verification is performed (see the verification mechanism section for details). If the verification passes, it checks whether the completion target has been achieved (e.g., the change in peak signal-to-noise ratio before and after completion is less than a threshold of 0.5dB or the number of iterations exceeds 5). If not, the context block is updated (merging the partial completion results), and new blocks are extracted repeatedly to continue completion. If the verification fails or the target is achieved, the algorithm terminates, the completed pixels are merged into the original image, and a complete 3D biological image is output (implemented by pixel replacement). The technical means employed in this process ensure the accuracy and robustness of the completion, and it achieves gradual optimization through iterative iteration, avoiding excessive computational overhead caused by processing the entire image at once.
[0034] Three-dimensional biological images have a unique characteristic: biological tissues (such as nerve fibers or cell bodies) exhibit spatial continuity and local texture consistency. Therefore, the completed image must maintain structural consistency with the original image; ensuring structural consistency is equivalent to guaranteeing the correctness of the completed image. This invention utilizes this feature, employing a threshold- and morphologically based simulated damage algorithm to generate a batch of training data (including 128 sets of mouse brain nerve fiber data and 128 sets of mouse brain neuron cell body data) as the algorithm's training set. Figure 2 As shown, the left image represents a nerve fiber, and the right image represents a neuron cell body. The missing regions in the training set images are fixed as irregular shapes near the cut surface (25% coverage), and each missing block has an original ground truth mask. The quality of the labeled training set (complete, missing images) is ensured after manual inspection and verification.
[0035] The detailed technical implementation is as follows: Training data preparation begins with multiple sets of 3D slice images of mouse brains, obtained from optical microscopy imaging with a resolution of 1 micrometer / pixel. A fixed mask is used to simulate missing areas near the slice: a 25% region near the slice is selected, the mask value is set to 1, and the corresponding pixels are set to 0 or background values to form a damaged image. Each sample pair consists of (damaged image + image patch to be completed = complete target image), with a total of approximately 3000 pairs. Data augmentation includes random rotation (-10° to 10°), scaling (0.95-1.05x), and Gaussian noise addition (standard deviation 0.02) to improve model generalization. Data preprocessing includes normalization. This ensures the input range is [0,1]. The data preparation techniques employed guarantee the model's adaptability to irregular missing values, making it suitable for highly heterogeneous biological tissue scenarios.
[0036] This invention trains a three-dimensional mask autoencoder network (3D-MAE) that receives damaged image patches and masks as inputs to predict missing pixel values.
[0037] like Figure 3As shown, the detailed technical implementation is as follows: The network model used in this method is 3D-MAE, which consists of two main parts: an encoder and a decoder. The input image block [512, 512, 32] is segmented into [32, 32, 32] blocks and input to the encoder. The encoder is a variant of the 3D Vision Transformer (3D-ViT). First, the block is divided into 4x4x4 patches using a 3D Patch embedding layer. Each patch is flattened into a vector (64 dimensions), and positional encoding is added before being input into the Transformer block (4 layers, 4 self-attention heads, 128 feedforward dimensions). A 25% patch is fixed with a mask, encoding only visible patches to reduce computational overhead. The decoder receives the latent representation from the encoder and placeholders for the mask patches, reconstructing all patches using a lightweight Transformer (2 layers) to reassemble a complete image block. After model output, the mean squared error is used as the loss function, calculating only the loss for the masked region: L = MSE(mask*target, mask* ), where target is the complete target pixel. Training uses the Adam optimizer with a learning rate of 2e-4, a batch size of 8, and 500 training rounds. This invention designs an adaptive learning rate strategy as shown in formula (3): The decay factor (0.01) is used to prevent overfitting. Once the parameters are determined, the completed image patch can be calculated. The original image is then updated using the new patch, and the preceding process is repeated to complete the missing data. This module's technique fully utilizes self-supervised learning, avoiding the need for extensive annotation and making it suitable for complex 3D biological structures.
[0038] like Figure 4 As shown, in practical use, the algorithm verifies the 3D-MAE completion output. At each completion step, it checks whether the grayscale value of the currently completed pixel is reasonable and uses reverse completion for verification: it uses the model to complete the neighborhood from the completed pixel in reverse, judging whether the error between the reverse completion result and the forward completion is reasonable, to ensure that the completed structure is always consistent with the original image context. Simultaneously, this invention also judges the continuity between the completed region and the neighborhood structure in real time, determining consistency based on factors such as whether the grayscale gradient is smooth and whether the texture matching degree is high.
[0039] The complete and damaged images in the training set of this invention have structural consistency. This invention utilizes this data to prepare the supervision information required by the algorithm, which is then used for self-supervised learning training. This invention also introduces noise points that deviate from the structure into the dataset, and makes the noise points have a large reconstruction error pointing to the original structure. With structurally consistent training data and appropriate noise data, the model learns the error correction strategy of "structural proximity" during training.
[0040] The detailed technical implementation is as follows: The verification mechanism includes a brightness reasonableness check: Calculate the local average grayscale of the padded pixel; if the deviation from the neighborhood exceeds a threshold (empirical value 10%), it is deemed unreasonable. Reverse padded pixel verification: Starting from the padded pixel as a new starting point, reversely extract the context block and input it into the model to generate a reverse prediction, and calculate the Euclidean distance error between the forward and reverse predictions: error = If the error exceeds 0.1, the completion is redone. Simultaneously, continuity is verified using the Structure Similarity Index Measure (SSIM): SSIM > 0.85. This mechanism employs multiple checks to ensure the biological validity of the completion results and avoid artifact generation.
[0041] The key points of this invention are: 1. This invention is aimed at three-dimensional biological image data modalities and proposes a missing region completion method based on deep learning. The completion process uses the original three-dimensional image as the processing object, rather than simply extending the two-dimensional image method to three dimensions; 2. This invention addresses the characteristics of missing regions in three-dimensional biological images, such as irregular shapes, complex spatial distribution, and high correlation with surrounding structures. It introduces three-dimensional contextual information to jointly model the missing regions, thereby achieving prediction and reconstruction of structural consistency. 3. This invention designs missing region completion as an independent and universal 3D image processing task, which does not depend on the specific generation method of the missing region and is applicable to various 3D biological image missing scenarios. 4. This invention constructs a complete technical solution for filling in missing regions of three-dimensional biological images, which can output three-dimensional biological image results that are continuous and complete in spatial structure.
[0042] The key points and areas to be protected in this invention are as follows: Areas to be protected: 1. The missing region completion method for three-dimensional biological image data proposed in this invention and its overall technical concept include a process of jointly modeling the missing region and its surrounding three-dimensional context information and completing the completion; 2. The technical solution for predicting and reconstructing the structural consistency of missing regions based on three-dimensional context information in this invention is not limited to specific network structures or model forms; 3. The processing method used in this invention is the automatic identification, marking, and completion of missing regions in three-dimensional biological images, and the fusion output of the results. 4. The system implementation scheme corresponding to the present invention includes the functional module division and combination form for implementing the method of filling missing regions in three-dimensional biological images.
[0043] Compared with existing three-dimensional image processing methods that mainly target two-dimensional images or focus on registration, this invention has the following advantages in completing missing regions of three-dimensional biological images: 1. This invention uses native three-dimensional biological image data as the processing object and directly models the three-dimensional spatial structure, avoiding the problem of missing structural information caused by simply extending two-dimensional methods to three dimensions in the prior art, and is more conducive to maintaining the continuity and consistency of biological tissues in space; 2. This invention addresses the irregular shape and complex spatial distribution of missing regions in 3D biological images by fully utilizing the 3D context information surrounding the missing regions for joint modeling. Compared to traditional methods that rely on local neighborhood information, this invention can more effectively recover complex biological structures. 3. This invention designs missing region completion as an independent 3D image processing task, which does not depend on the specific generation method of the missing region. Compared with existing deep learning methods that only focus on registration accuracy, it has stronger versatility and applicability. 4. This invention emphasizes structural consistency rather than merely pursuing grayscale or intensity continuity during the completion process, making the completion result more reasonable in spatial morphology, which is beneficial for subsequent three-dimensional analysis and biological research applications.
[0044] This invention has been experimentally verified and tested in the task of completing missing regions in 3D biological images. Experimental results show that by completing 3D biological images containing missing regions, this invention can effectively restore the spatial structural continuity of the missing regions, and the completion results are significantly improved in terms of overall integrity and structural rationality. Related experimental and application results demonstrate that the 3D biological image missing region completion method proposed in this invention has good feasibility and practical application value.
[0045] Example 2 The system corresponding to this invention includes: a data input module for acquiring three-dimensional biological image data; a missing region identification module for identifying missing regions and extracting three-dimensional contextual information; a completion module for predicting and reconstructing missing pixels based on a mask autoencoder model; a verification module for checking the legality and continuity of the completion results; and an output module for fusing the completion results and outputting a complete three-dimensional image. This system can be deployed on computing devices (such as servers equipped with graphics processors) and its functions are invoked through a software interface, ensuring that those skilled in the art can use it directly.
[0046] This invention provides a deep learning-based system for completing missing regions in 3D biological images. The technical solution includes: acquiring a 3D biological image containing missing regions; identifying the missing regions in the 3D biological image and extracting the surrounding 3D context information; constructing a deep learning-based completion model for 3D data, namely a mask autoencoder model (3D-MAE model), based on the 3D context information; using the deep learning completion model to predict and reconstruct the structural consistency of the missing regions; and outputting the reconstructed 3D image data, thereby obtaining a spatially continuous and complete 3D biological image.
[0047] This invention utilizes the spatial context information and structural continuity features of 3D biological images to construct a masked autoencoder (MAE) model for 3D data. This model effectively predicts and completes missing regions caused by slicing loss, imaging noise, etc., thereby restoring the continuity and integrity of biological tissues in 3D space. Specifically, this invention uses 3D biological images containing missing regions as input data, employs a 3D-MAE model to jointly model the missing regions and their surrounding 3D context, and predicts the missing pixel values through self-supervised learning to achieve structural consistency reconstruction. This method is independent of the specific shape or cause of the missing regions and is applicable to 3D imaging scenarios of various biological samples, such as neural tissue or neuronal cell body slices.
[0048] For ease of understanding, the technical solution of the present invention will be described in detail with reference to the accompanying drawings. The drawings include: Figure 1 This is a flowchart of the overall solution. Figure 2 Examples of training set images; Figure 3 To complete the module's principle block diagram; Figure 4 To verify the principle of the mechanism, the technical implementation of each part will be explained in detail below, with corresponding text descriptions for each figure.
[0049] The process of this invention is as follows: Figure 1 As shown. Figure 1 This is a flowchart, from top to bottom, including the following boxes: "Input 3D biological image data", "Missing region identification and marking", "Extract 3D context information", "Input model prediction and completion pixels", "Verify completion result validity" (if the branch is "Yes", the output is fused; if "No", the parameters are adjusted and completion is restarted), "Completion target achieved" (if the branch is "No", the upper and lower pixels are updated and the loop continues; if "Yes", the loop terminates), and "Output completed 3D image". This flowchart uses a loop structure to represent the completion process, ensuring that the algorithm iterates until the termination condition is met.
[0050] The detailed technical implementation is as follows: First, acquire 3D biological image data containing missing regions, typically stored in TIFF or NIfTI format with pixel resolution at the micrometer level. Then, automatically identify missing regions using a threshold segmentation algorithm: calculate the image grayscale gradient; if the gradient is lower than a preset threshold (empirical value 0.05 times the average grayscale) and the volume of a continuous region exceeds a minimum threshold (set to 64 pixels), it is identified as a missing mask. A mask value of 1 indicates a missing pixel. Using the missing region as the center, extract the surrounding 3D context information from the 3D image: crop a fixed-size image block, such as a cube with sides of 4 pixels, ensuring the missing region is centered within the block. This cropping operation is achieved through coordinate offset, i.e., slicing a sub-region within the 3D array. , where d is the half-side length (usually fixed at 2), and the context information includes the grayscale of neighboring pixels, texture features, and biological structural priors (such as tissue continuity).
[0051] The extracted 3D context block and missing mask are input into a pre-trained mask autoencoder model, which outputs the predicted completed pixel values (see the completion module principle for details). Based on the prediction results, a validity verification is performed (see the verification mechanism section for details). If the verification passes, it checks whether the completion target has been achieved (e.g., the change in peak signal-to-noise ratio before and after completion is less than a threshold of 0.5dB or the number of iterations exceeds 5). If not, the context block is updated (merging the partial completion results), and new blocks are extracted repeatedly to continue completion. If the verification fails or the target is achieved, the algorithm terminates, the completed pixels are merged into the original image, and a complete 3D biological image is output (implemented by pixel replacement). The technical means employed in this process ensure the accuracy and robustness of the completion, and it achieves gradual optimization through iterative iteration, avoiding excessive computational overhead caused by processing the entire image at once.
[0052] Three-dimensional biological images have a unique characteristic: biological tissues (such as nerve fibers or cell bodies) exhibit spatial continuity and local texture consistency. Therefore, the completed image must maintain structural consistency with the original image; ensuring structural consistency is equivalent to guaranteeing the correctness of the completed image. This invention utilizes this feature, employing a threshold- and morphologically based simulated damage algorithm to generate a batch of training data (including 128 sets of mouse brain nerve fiber data and 128 sets of mouse brain neuron cell body data) as the algorithm's training set. Figure 2 As shown, the left image represents a nerve fiber, and the right image represents a neuron cell body. The missing regions in the training set images are fixed as irregular shapes near the cut surface (25% coverage), and each missing block has an original ground truth mask. The quality of the labeled training set (complete, missing images) is ensured after manual inspection and verification.
[0053] The detailed technical implementation is as follows: Training data preparation begins with multiple sets of 3D slice images of mouse brains, obtained from optical microscopy imaging with a resolution of 1 micrometer / pixel. A fixed mask is used to simulate missing areas near the slice: a 25% region near the slice is selected, the mask value is set to 1, and the corresponding pixels are set to 0 or background values to form a damaged image. Each sample pair consists of (damaged image + image patch to be completed = complete target image), with a total of approximately 3000 pairs. Data augmentation includes random rotation (-10° to 10°), scaling (0.95-1.05x), and Gaussian noise addition (standard deviation 0.02) to improve model generalization. Data preprocessing includes normalization. This ensures the input range is [0,1]. The data preparation techniques employed guarantee the model's adaptability to irregular missing values, making it suitable for highly heterogeneous biological tissue scenarios.
[0054] This invention trains a three-dimensional mask autoencoder network (3D-MAE) that receives damaged image patches and masks as inputs to predict missing pixel values.
[0055] like Figure 3 As shown, the detailed technical implementation is as follows: The network model used in this method is 3D-MAE, which consists of two main parts: an encoder and a decoder. The input image block [512, 512, 32] is segmented into [32, 32, 32] blocks and input to the encoder. The encoder is a variant of the 3D Vision Transformer (3D-ViT). First, the block is divided into 4x4x4 patches using a 3D Patch embedding layer. Each patch is flattened into a vector (64 dimensions), and positional encoding is added before being input into the Transformer block (4 layers, 4 self-attention heads, 128 feedforward dimensions). A 25% patch is fixed with a mask, encoding only visible patches to reduce computational overhead. The decoder receives the latent representation from the encoder and placeholders for the mask patches, reconstructing all patches using a lightweight Transformer (2 layers) to reassemble a complete image block. After model output, the mean squared error is used as the loss function, calculating only the loss for the masked region: L = MSE(mask*target, mask* ), where target is the complete target pixel. Training uses the Adam optimizer with a learning rate of 2e-4, a batch size of 8, and 500 training rounds. This invention designs an adaptive learning rate strategy as shown in formula (3): The decay factor (0.01) is used to prevent overfitting. Once the parameters are determined, the completed image patch can be calculated. The original image is then updated using the new patch, and the preceding process is repeated to complete the missing data. This module's technique fully utilizes self-supervised learning, avoiding the need for extensive annotation and making it suitable for complex 3D biological structures.
[0056] like Figure 4 As shown, in practical use, the algorithm verifies the 3D-MAE completion output. At each completion step, it checks whether the grayscale value of the currently completed pixel is reasonable and uses reverse completion for verification: it uses the model to complete the neighborhood from the completed pixel in reverse, judging whether the error between the reverse completion result and the forward completion is reasonable, to ensure that the completed structure is always consistent with the original image context. Simultaneously, this invention also judges the continuity between the completed region and the neighborhood structure in real time, determining consistency based on factors such as whether the grayscale gradient is smooth and whether the texture matching degree is high.
[0057] The complete and damaged images in the training set of this invention have structural consistency. This invention utilizes this data to prepare the supervision information required by the algorithm, which is then used for self-supervised learning training. This invention also introduces noise points that deviate from the structure into the dataset, and makes the noise points have a large reconstruction error pointing to the original structure. With structurally consistent training data and appropriate noise data, the model learns the error correction strategy of "structural proximity" during training.
[0058] The detailed technical implementation is as follows: The verification mechanism includes a brightness reasonableness check: Calculate the local average grayscale of the padded pixel; if the deviation from the neighborhood exceeds a threshold (empirical value 10%), it is deemed unreasonable. Reverse padded pixel verification: Starting from the padded pixel as a new starting point, reversely extract the context block and input it into the model to generate a reverse prediction, and calculate the Euclidean distance error between the forward and reverse predictions: error = If the error exceeds 0.1, the completion is redone. Simultaneously, continuity is verified using the Structure Similarity Index Measure (SSIM): SSIM > 0.85. This mechanism employs multiple checks to ensure the biological validity of the completion results and avoid artifact generation.
[0059] The key points of this invention are: 1. This invention is aimed at three-dimensional biological image data modalities and proposes a missing region completion method based on deep learning. The completion process uses the original three-dimensional image as the processing object, rather than simply extending the two-dimensional image method to three dimensions; 2. This invention addresses the characteristics of missing regions in three-dimensional biological images, such as irregular shapes, complex spatial distribution, and high correlation with surrounding structures. It introduces three-dimensional contextual information to jointly model the missing regions, thereby achieving prediction and reconstruction of structural consistency. 3. This invention designs missing region completion as an independent and universal 3D image processing task, which does not depend on the specific generation method of the missing region and is applicable to various 3D biological image missing scenarios. 4. This invention constructs a complete technical solution for filling in missing regions of three-dimensional biological images, which can output three-dimensional biological image results that are continuous and complete in spatial structure.
[0060] The key points and areas to be protected in this invention are as follows: Areas to be protected: 1. The missing region completion method for three-dimensional biological image data proposed in this invention and its overall technical concept include a process of jointly modeling the missing region and its surrounding three-dimensional context information and completing the completion; 2. The technical solution for predicting and reconstructing the structural consistency of missing regions based on three-dimensional context information in this invention is not limited to specific network structures or model forms; 3. The processing method used in this invention is the automatic identification, marking, and completion of missing regions in three-dimensional biological images, and the fusion output of the results. 4. The system implementation scheme corresponding to the present invention includes the functional module division and combination form for implementing the method of filling missing regions in three-dimensional biological images.
[0061] Compared with existing three-dimensional image processing methods that mainly target two-dimensional images or focus on registration, this invention has the following advantages in completing missing regions of three-dimensional biological images: 1. This invention uses native three-dimensional biological image data as the processing object and directly models the three-dimensional spatial structure, avoiding the problem of missing structural information caused by simply extending two-dimensional methods to three dimensions in the prior art, and is more conducive to maintaining the continuity and consistency of biological tissues in space; 2. This invention addresses the irregular shape and complex spatial distribution of missing regions in 3D biological images by fully utilizing the 3D context information surrounding the missing regions for joint modeling. Compared to traditional methods that rely on local neighborhood information, this invention can more effectively recover complex biological structures. 3. This invention designs missing region completion as an independent 3D image processing task, which does not depend on the specific generation method of the missing region. Compared with existing deep learning methods that only focus on registration accuracy, it has stronger versatility and applicability. 4. This invention emphasizes structural consistency rather than merely pursuing grayscale or intensity continuity during the completion process, making the completion result more reasonable in spatial morphology, which is beneficial for subsequent three-dimensional analysis and biological research applications.
[0062] This invention has been experimentally verified and tested in the task of completing missing regions in 3D biological images. Experimental results show that by completing 3D biological images containing missing regions, this invention can effectively restore the spatial structural continuity of the missing regions, and the completion results are significantly improved in terms of overall integrity and structural rationality. Related experimental and application results demonstrate that the 3D biological image missing region completion method proposed in this invention has good feasibility and practical application value.
[0063] Example 3 A storage medium storing program files capable of implementing any of the above-mentioned deep learning-based methods for filling in missing regions of 3D biological images.
[0064] Example 4 A processor for running a program, wherein the program executes any of the above-mentioned deep learning-based method for filling in missing regions of 3D biological images.
[0065] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0066] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0067] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The system embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.
[0068] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0069] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0070] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0071] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for filling in missing regions in three-dimensional biological images based on deep learning, characterized in that, include: Acquire three-dimensional biological images containing missing regions; The missing regions in 3D biological images are identified, and the 3D context information around the missing regions is extracted. A deep learning completion model for 3D data is constructed based on 3D context information. The deep learning completion model is used to predict and reconstruct the structural consistency of missing regions. The reconstructed 3D image data is output to obtain a continuous and complete 3D biological image in terms of spatial structure.
2. The method for completing missing regions in a three-dimensional biological image based on deep learning according to claim 1, characterized in that, A deep learning-based completion model for 3D data is constructed based on 3D context information. This model is then used to predict and reconstruct the structural consistency of missing regions, including: A deep learning-based completion model is used to jointly model the missing region and its surrounding 3D context. The missing pixel values are predicted through self-supervised learning, thereby achieving structural consistency reconstruction.
3. The method for completing missing regions in three-dimensional biological images based on deep learning according to claim 1, characterized in that, Identifying missing regions in 3D biological images and extracting the surrounding 3D contextual information includes: Missing regions are automatically identified using a threshold segmentation algorithm; Using the missing region as the center, extract the surrounding 3D context information from the 3D image.
4. The method for completing missing regions in three-dimensional biological images based on deep learning according to claim 3, characterized in that, The missing regions are automatically identified using a threshold segmentation algorithm, including: Calculate the grayscale gradient of the image. If the gradient is lower than a preset threshold and the volume of a continuous region exceeds the minimum threshold, it is marked as a missing mask.
5. The method for completing missing regions in three-dimensional biological images based on deep learning according to claim 3, characterized in that, Centered on the missing region, extracting surrounding 3D context information from the 3D image includes: Extract a pre-defined fixed-size image block, ensuring that the missing area is located at the center of the block. This extraction operation is achieved through coordinate offset, slicing sub-regions in a three-dimensional array.
6. The method for completing missing regions in three-dimensional biological images based on deep learning according to claim 1, characterized in that, The 3D context information includes the grayscale of neighboring pixels, texture features, and prior biological structures.
7. The method for completing missing regions in three-dimensional biological images based on deep learning according to claim 1, characterized in that, Using deep learning completion models to predict and reconstruct structural consistency in missing regions includes: Based on the prediction results, a legality verification is performed. If the verification passes, the completion target is checked. If not, the context block is updated, and new blocks are extracted in a loop to continue the completion. If the verification fails or the target is achieved, the algorithm is terminated, and the completed pixels are fused into the original image to output a complete 3D biological image.
8. The method for completing missing regions in a three-dimensional biological image based on deep learning according to claim 1, characterized in that, Also includes: A batch of training data was generated using a threshold- and morphologically based simulated damage algorithm as the training set for the deep learning completion model. The missing regions in the training set images were fixed as irregular shapes near the cross-section, and each missing block had an original ground truth mask.
9. The method for completing missing regions in a three-dimensional biological image based on deep learning according to claim 1, characterized in that, Also includes: The completion output of the deep learning completion model is verified. At each completion step, the gray value of the current completed pixel is checked to see if it is reasonable. Inverse completion is used for verification: the model is used to complete the neighborhood from the completed pixel in reverse to judge whether the error between the reverse completion result and the forward completion is reasonable, so that the completion structure is always consistent with the original image context.
10. A deep learning-based system for completing missing regions in 3D biological images, characterized in that, include: The data input module is used to acquire three-dimensional biological images containing missing regions; The missing region identification module is used to identify missing regions in 3D biological images and extract the 3D context information around the missing regions. The completion module is used to build a deep learning completion model for 3D data based on 3D context information, and to use the deep learning completion model to predict and reconstruct the structural consistency of missing regions. The output module is used to output the reconstructed 3D image data to obtain a continuous and complete 3D biological image in spatial structure.