Microscopy image stream fusion methods, apparatuses, storage media, and program products

Abstract

This invention discloses a method, device, storage medium, and program product for streaming fusion of microscopic images. The invention constructs a deep learning model based on recursive updates of a memory state and integrates this model as a whole into a microscope acquisition system. As high-resolution microscopic images arrive one by one, the internal fusion state is maintained, and the fusion result is gradually improved through recursive updates of the memory state. This achieves the goals of arbitrary number of image inputs, continuous state updates, online fusion output, and on-demand fusion stopping in high-resolution microscopic imaging scenarios.

Description

Technical Field

[0001] This invention belongs to the fields of image processing and deep learning, and specifically relates to a microscopic image fusion technique. Background Technology

[0002] During microscope imaging, due to the limited depth of field of microscope objectives, uneven sample thickness, undulating sample surfaces, and the difficulty of the focal plane covering the entire observation area at once, a single image typically only achieves high clarity in a localized area, while other areas are prone to defocusing, blurring, loss of detail, or incomplete structure. To obtain a clearer and more complete overall microscopic image, it is usually necessary to acquire multiple partially focused images along the Z-axis or at different focal planes, and then fuse these images to form a single output image with complete and clear structural information. An example of fusion is shown below. Figure 1 As shown.

[0003] Existing multifocal image fusion methods mainly fall into two categories: traditional multifocal image fusion methods and deep learning-based multifocal image fusion methods. Traditional multifocal image fusion methods primarily rely on manually designed image processing workflows, sharpness evaluation metrics, and rules (gradient, Laplacian, energy, transform domain coefficients, etc.) to complete the multifocal image fusion. Although traditional multifocal image fusion methods are theoretically mature and do not depend on large-scale training data, they still suffer from the following problems: 1. Relying on manually designed clarity evaluation metrics and rules, its generalization ability is limited; 2. Insufficient adaptability to areas with weak texture, low contrast, and complex blurry conditions.

[0004] Deep learning-based multifocal image fusion methods utilize convolutional neural networks, GAN networks, and Transformers to extract and fuse features from input multifocal images, thereby achieving multifocal image fusion. Compared to traditional multifocal image fusion methods, deep learning-based methods have made further progress in image fusion quality and generalization, but the following problems still exist: 1. Models are often designed with a limited number of input images (number of model channels), making them difficult to adapt to scenarios where the number of images to be fused is not fixed in real-world applications; 2. The intermediate results of the pairwise fusion method are difficult to systematically retain all the effective structural information in the historical images, and the error may gradually accumulate as the number of recursions increases; 3. For high-resolution microscopic images, the memory and computing power overhead of multi-image parallel processing is relatively large. Summary of the Invention

[0005] To address the technical problems mentioned in the background section, this invention proposes a method, apparatus, storage medium, and program product for streaming microscopic image fusion.

[0006] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows: A streaming fusion method for microscopic images based on recursive updates of memory states is proposed. A deep learning model based on recursive updates of memory states is constructed and trained. Whenever a new image is acquired by the microscope, it is input into the deep learning model. The model extracts current image feature information and focus guidance information, and combines this with the historical state saved from the previous moment to perform a recursive update of the microscope's internal system state. The focus guidance information guides which regions in the current image have higher sharpness. The historical state includes a fusion memory state and a hidden state. The fusion memory state represents the internal fusion features formed by recursively fusing previous input images up to the current moment, including structural information, texture information, and sharp region information of the fused image content. The hidden state represents the contextual information, state evolution trend, and auxiliary update information accumulated during the recursive update process to guide how to update the existing fusion memory state when a new image arrives. As images are continuously input, the microscope's internal system state continuously evolves, gradually accumulating effective structural information from different focal planes or different sharp regions, ultimately forming the final fused image.

[0007] Furthermore, the deep learning model includes a focus guidance module, an encoder feature encoding module, a memory update unit, and a decoder feature decoding module. Images acquired by the microscope are input to the focus guidance module and the encoder feature encoding module, respectively. The focus guidance module extracts focus guidance information related to local focus details, texture integrity, and high-frequency structures from the current input image. The encoder feature encoding module performs feature extraction on the current input image. The memory update unit receives the feature information and focus guidance information of the current input image at the current moment and updates the internal state of the system by combining it with the historical state generated and saved at the previous moment. The decoder feature decoding module restores the phased fusion result at the current moment based on the updated internal state of the system.

[0008] Furthermore, the deep learning model also includes a saturation sensing module. The saturation sensing module extracts saturation feedback information and global stopping determination information from the phased fusion results output by the Decoder feature decoding module. The saturation feedback information indicates whether each position in the current phase fusion result has become stable and participates in the state update control of the memory update unit as feedback information for the next moment. The global stopping determination information is used to determine whether the current phase fusion result has reached the preset imaging requirements, so as to control whether the microscope continues to acquire the next image.

[0009] Furthermore, the saturation sensing module adopts a structure based on convolution and pooling.

[0010] Furthermore, the focus guidance module adopts a structure based on convolution and splicing fusion.

[0011] Furthermore, the memory update unit adopts a ConvLSTM-based structure.

[0012] Secondly, the present invention also provides a computer device, which includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described microscopic image streaming fusion method based on memory state recursive update.

[0013] Furthermore, the present invention also provides a computer storage medium storing a computer program, which, when executed on a processor, implements the aforementioned microscopic image streaming fusion method based on memory state recursive updates.

[0014] Finally, the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the above-described microscopic image streaming fusion method based on memory state recursive updates.

[0015] The beneficial effects of adopting the above technical solution are as follows: This invention maintains the internal fusion state as each microscopic image arrives and gradually improves the fusion result through a recursive update of the memory state. Compared with existing technologies, it has the following advantages: 1. It can continuously fuse any number of input images without limiting or predicting the number of images; 2. The recursive update of the memory state and the hidden state is integrated to continuously retain the effective historical information of the previous input image; 3. Streaming processing of high-resolution microscopic images reduces memory usage and computational pressure; 4. Supports simultaneous shooting and fusion, and can stop and output the current fusion result at any time. Attached Figure Description

[0016] Figure 1 This is an example diagram of traditional multi-focus image fusion; Figure 2 This is a diagram of the deep learning model structure in the embodiment; Figure 3 This is a flowchart of the model training process in the embodiment; Figure 4 This is a flowchart of the model application in the embodiment. Detailed Implementation

[0017] The basic idea of ​​this embodiment is to solve the problem of microscopic image fusion using deep learning methods. However, current common deep learning image fusion models are usually designed for a fixed number of inputs. That is, after the model is built, the number of input images needs to match the preset input format, making it difficult to directly adapt to actual application scenarios with variable input numbers. In actual microscopic imaging, due to differences in sample type, sample thickness, depth of field range, and imaging strategy, the number of images obtained in each ultra-depth-of-field imaging is often uncertain and usually cannot be accurately predicted before the start of imaging. Therefore, a major technical problem to be solved in this embodiment is how to build a deep learning model that can achieve the fusion of any number (variable number) of images. Based on this, this embodiment designs a streaming fusion method for any number of high-resolution microscopic images based on recursive updates of memory states.

[0018] This embodiment does not treat multiple images to be fused as fixed-length image data that must be input simultaneously at once, but rather as a variable-length image stream arriving one by one. In this image stream, whenever the microscope acquires a new image, the system extracts the current image's feature information and focus guidance information, and combines this with the historical state saved from the previous moment and the saturation feedback information from the previous moment to perform a recursive update of the system's internal state. As images continue to be input, the system's internal state continuously evolves, gradually accumulating effective structural information from different focal planes or different areas of sharpness, ultimately forming the fused image. In the above process, the historical state maintained by the system includes a fusion memory state and a hidden state. The fusion memory state represents the internal fusion features formed by the recursive fusion of previous input images up to the current moment, including structural information, texture information, and sharpness information of the fused image content. The hidden state represents the context information, state evolution trend, and auxiliary update information accumulated during the recursive update process; it is mainly used to guide how the system should update the existing fusion memory state when the current input image arrives. By continuously maintaining the aforementioned historical state, the system can continuously correct, supplement, and improve the existing fusion results as subsequent images arrive, thereby establishing a recursive fusion mechanism applicable to any number of input images.

[0019] This embodiment requires constructing a deep learning model based on recursive updates of memory states, the structure of which is as follows: Figure 2As shown, the system includes an input image, a focus guidance module, an Encoder feature encoding module, a memory update unit, a Decoder feature decoding module, a staged fusion result node, a saturation perception module, and a final output image node. The input image is fed into the focus guidance module and the Encoder feature encoding module. The focus guidance module can adopt a convolutional and concatenation-based fusion structure to extract focus guidance information related to local focus details, texture integrity, and high-frequency structures from the current input image, characterizing which regions in the current image are clearer and more worthy of being written into the existing fusion memory. This module preferably uses a lightweight structure to reduce computational complexity and minimize the impact on subsequent recursive update processes. The Encoder feature encoding module performs feature extraction on the current image to obtain current image representation information suitable for recursive updates, and can use network structures such as ResNet-50 or MobileNetV3. The memory update unit generates and retains the fusion memory state and hidden state in the previous time step, and receives the current image representation information, current focus guidance information, and saturation feedback information from the previous time step in the current time step. It combines this with the historical state retained from the previous time step to complete the state update. This unit can be implemented using a ConvLSTM-based structure. The Decoder feature decoding module is used to recover the current stage fusion result based on the updated internal state. It can employ a U-Net-style decoder, a Transformer-style decoder, or other decoding structures. The saturation sensing module can adopt a convolutional and pooling-based structure to extract saturation feedback information and global stopping determination information from the stage fusion result. The saturation feedback information is used to characterize whether each position in the current stage fusion result has stabilized and serves as feedback information for the next time step in the state update control of the memory update unit. The global stopping determination information is used to determine whether the current stage fusion result has met the preset imaging requirements, thereby controlling whether the microscope continues to acquire the next image.

[0020] In this embodiment, the recursive update of the memory state is not a simple mechanical overlay of the current image with the previous result, nor is it a normal pairwise fusion of the current image with the fused image from the previous moment. Instead, it uses a ConvLSTM unit with temporal memory capabilities to selectively update the internal state of the system based on the historical state saved in the previous moment, while maintaining the spatial structure, by combining the current image feature information, the current focus guidance information, and the saturation feedback information from the previous moment. Specifically, for the t-th input image, after receiving the input consisting of the current image feature information, the current focus guidance information, and the saturation feedback information from the previous moment, the memory update unit combines the hidden state and the fused memory state from the previous moment, and controls the combination of the current image information and the historical state through input gates, forget gates, and output gates, thereby updating the fused memory state and hidden state at the current moment. As images are continuously input, the internal fusion state continues to evolve, gradually accumulating effective structural information from different focal planes or different areas of sharpness, enabling the system to gradually form a more complete and clearer fusion result during continuous shooting.

[0021] In this embodiment, the current stage fusion result is not necessarily obtained only after all images have been acquired. Instead, it is continuously refreshed with each new image, forming a streaming processing mechanism of "acquire one image, update the state once, and refresh the result once," enabling image acquisition and image fusion to proceed synchronously. For practical microscope applications, this mechanism is particularly suitable for scenarios such as Z-axis layer-by-layer scanning, autofocus image-by-image shooting, and high-resolution step-by-step scanning, because the system holds a usable stage fusion result at any given time and can determine in real time whether it is necessary to continue acquiring the next image based on this result.

[0022] Before applying the aforementioned deep learning model, training data needs to be prepared to train the model. The training process is as follows: Figure 3 As shown: S1. Acquire image sequences of various samples at different magnifications and in varying quantities; S2. The ground truth value of the ground truth for each image sequence is synthesized using the traditional hyper-depth-of-field fusion method, and a training dataset is formed with the corresponding image sequence. The traditional hyper-depth-of-field fusion method uses image sharpness, gradient and other indicators for image fusion. S3, according to Figure 2 The network structure shown constructs a deep learning model based on recursive updates of memory states; S4. Use the training dataset to train the constructed deep learning model.

[0023] After the above process, a trained deep learning model is obtained. The subsequent application process can be referenced from [the above steps]. Figure 4 : S1. Control the microscope to perform real-time image acquisition along the preset Z-axis scanning path, and obtain an input image of the sample at the current focal plane; S2. The system determines whether the current input image is the first image: if it is the first image, the state of the model's internal memory unit is initialized; if it is not the first image, the internal state that was updated and retained in the previous time step is read and used as the historical basis for the current time step's recursive processing. S3. Feed the current input image into the deep learning model, perform image fusion with memory on the current image, and output the fusion result of the current stage and the stopping decision information; S4. Based on the stopping judgment information output by the model, determine whether the fusion result of the current stage has met the preset stopping condition. The preset stopping condition can be set by the model according to the image's clarity, contrast, texture integrity and other evaluation indicators. S5. If the fusion result of the current stage meets the preset stop condition: output the fusion result of the current stage and send a stop acquisition command to the microscope to terminate subsequent imaging; if the fusion result of the current stage does not meet the preset stop condition, control the microscope to move to the next focal plane position, continue to acquire the next image, and repeat the above process.

[0024] This embodiment also relates to a computer device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the above-described contents. This embodiment also relates to a computer storage medium storing a computer program, which, when executed on a processor, implements the above-described contents. This embodiment also relates to a computer program product, including a computer program or instructions, which, when executed by a processor, implements the above-described contents.

[0025] The embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A microscopic image streaming fusion method based on memory state recursive update, characterized in that: A deep learning model based on recursive updates of memory states is constructed and trained. Whenever the microscope acquires a new image, the image is input into the deep learning model to extract the current image feature information and focus guidance information. Combined with the historical state saved at the previous moment, a recursive update is performed on the internal state of the microscope system. The focus guidance information is used to guide which areas in the current image have higher clarity. The historical state includes a fusion memory state and a hidden state. The fusion memory state represents the internal fusion features formed by recursively fusing the previous input images up to the current moment, including the structural information, texture information, and clear area information of the fused image content. The hidden state represents the context information, state evolution trend, and auxiliary update information accumulated during the recursive update process to guide how to update the existing fusion memory state when a new image arrives. As images are continuously input, the internal state of the microscope system evolves, gradually accumulating effective structural information from different focal planes or different areas of sharpness, and forming the final fused image.

2. The microscopic image streaming fusion method based on memory state recursive update according to claim 1, characterized in that: The deep learning model includes a focus guidance module, an Encoder feature encoding module, a memory update unit, and a Decoder feature decoding module; Images acquired by the microscope are input to the focus guidance module and the Encoder feature encoding module, respectively. The focus guidance module extracts focus guidance information related to local focus details, texture integrity, and high-frequency structures from the current input image. The Encoder feature encoding module performs feature extraction on the current input image. The memory update unit receives the feature information and focus guidance information of the current input image at the current moment and updates the internal state of the system by combining it with the historical state generated and saved at the previous moment. The Decoder feature decoding module restores the stage fusion result of the current moment based on the updated internal state of the system.

3. The microscopic image streaming fusion method based on memory state recursive update according to claim 2, characterized in that: The deep learning model also includes a saturation sensing module. The saturation sensing module extracts saturation feedback information and global stopping determination information from the phased fusion results output by the Decoder feature decoding module. The saturation feedback information indicates whether each position in the current phase fusion result has become stable and participates in the state update control of the memory update unit as feedback information for the next moment. The global stopping determination information is used to determine whether the current phase fusion result has reached the preset imaging requirements, so as to control whether the microscope continues to acquire the next image.

4. The microscopic image streaming fusion method based on memory state recursive update according to claim 3, characterized in that: The saturation sensing module adopts a structure based on convolution and pooling.

5. The microscopic image streaming fusion method based on memory state recursive update according to claim 2, characterized in that: The focus guidance module adopts a structure based on convolution and splicing fusion.

6. The microscopic image streaming fusion method based on memory state recursive update according to claim 2, characterized in that: The memory update unit adopts a ConvLSTM-based structure.

7. A computer device comprising a processor and a memory, wherein the memory stores a computer program, characterized in that: The processor is used to execute the computer program to implement the microscopic image streaming fusion method based on memory state recursive update as described in any one of claims 1-6.

8. A computer storage medium storing a computer program, characterized in that: When the computer program is executed on the processor, it implements the microscopic image streaming fusion method based on memory state recursive update as described in any one of claims 1-6.

9. A computer program product, comprising a computer program or instructions, characterized in that: When the computer program or instructions are executed by the processor, they implement the microscopic image streaming fusion method based on memory state recursive update as described in any one of claims 1-6.

Images