Chromosome three-dimensional structure light field mapping and space-time alignment multi-modal fusion segmentation system

Abstract

The invention relates to the technical field of biological microscopic imaging and computational analysis, and discloses a chromosome three-dimensional structure light field mapping and space-time alignment multi-modal fusion segmentation system, which comprises a hybrid asynchronous bimodal imaging module, a space-time state analysis and uncertainty evaluation module and a closed-loop feedback and probe strategy generation module. The system continuously obtains global low-resolution data by using low-phototoxicity light field imaging, and predicts a high-resolution structure and quantifies the uncertainty of the high-resolution structure in real time by using a deep learning model. When the uncertainty is locally too high, the system actively generates a probe strategy and drives structured light imaging to perform primary high-resolution acquisition on the area. And the acquired image is used as a reference for online correction of the deep learning model. According to the invention, through intelligent closed-loop feedback of prediction verification correction, disturbance of high-intensity illumination is greatly reduced, and long-time high-resolution four-dimensional dynamic tracking of a fine structure in a living cell can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of biological microscopic imaging and computational analysis, in particular to a chromosome three-dimensional structure light field mapping and spatiotemporal alignment multi-modal fusion segmentation system. BACKGROUND

[0002] In life science research, real-time and three-dimensional observation of dynamic processes of fine structures such as chromosomes and organelles in living cells is the key to understanding the basic laws of life activities. In order to obtain high-resolution images, researchers have developed a variety of advanced microscopic imaging techniques, such as confocal microscopy and super-resolution techniques such as structured illumination microscopy (SIM) and stimulated emission depletion microscopy (STED). Although these techniques can provide nanometer-level spatial resolution, their imaging principles usually rely on high-intensity laser illumination. This high-photon-dose irradiation can cause serious phototoxicity to living cells, damage cell structures, interfere with their normal physiological activities, and also cause rapid bleaching of fluorescent probes, thereby preventing long-term continuous observation.

[0003] In contrast, in order to reduce phototoxicity and improve imaging speed, researchers have also used low-photon-flux imaging methods such as light field microscopy (LFM). Light field microscopy can capture three-dimensional spatial information through a single snapshot, has extremely high imaging speed and extremely low phototoxicity, and is very suitable for long-term dynamic recording. However, this type of technology sacrifices spatial resolution at the cost of limited quality of the reconstructed three-dimensional image, making it difficult to clearly distinguish fine subcellular structures such as chromosomes.

[0004] Therefore, the existing technology generally faces a fundamental contradiction that is difficult to reconcile. Pursuing high spatial resolution inevitably brings serious phototoxicity and photobleaching, and in order to achieve long-term mild imaging, the clarity of the image must be sacrificed. Researchers have to make a difficult choice between observation time and analytical ability, which greatly limits the in-depth exploration of long-term, fine life processes such as cell division and DNA replication repair. Although attempts have been made to use deep learning and other computational methods to restore high-resolution information from low-resolution images, these methods usually rely on fixed, offline trained models. For the dynamic processes of living cells that change rapidly and unpredictably, this open-loop prediction method lacks feedback and verification mechanisms, and its accuracy and reliability cannot be guaranteed, and it cannot adapt to new conditions that may arise during the experiment. SUMMARY

[0005] The technical problem to be solved by the present application is that the existing live cell four-dimensional imaging technology has inherent contradictions when trying to simultaneously obtain high spatial resolution, high temporal resolution and low phototoxicity, and the existing image segmentation and reconstruction methods based on deep learning usually rely on large-scale and high-quality pixel-level annotation data sets, which have high acquisition costs, and the results may not meet the biological topological constraints.

[0006] To solve the above technical problems, the present application provides a chromosome three-dimensional structure light field mapping and spatio-temporal alignment multi-modal fusion segmentation system in the first aspect.

[0007] The system comprises a hybrid asynchronous dual-modal imaging module, a spatio-temporal state analysis and uncertainty evaluation module, a closed-loop feedback and probe strategy generation module, and a four-dimensional high-resolution structure fusion output module.

[0008] The hybrid asynchronous dual-modal imaging module is used to obtain low-resolution global dynamic data and high-resolution anchor point images of a live cell sample. The spatio-temporal state analysis and uncertainty evaluation module is used to generate a high-resolution three-dimensional structure prediction based on the low-resolution global dynamic data and simultaneously quantify the uncertainty of the high-resolution three-dimensional structure prediction to form an uncertainty atlas, and to correct the model online using the high-resolution anchor point images. The closed-loop feedback and probe strategy generation module is used to generate a probe strategy for guiding the acquisition of the high-resolution anchor point images based on the uncertainty atlas. The four-dimensional high-resolution structure fusion output module is used to collect and integrate the high-resolution three-dimensional structure prediction output by the spatio-temporal state analysis and uncertainty evaluation module after online correction to form four-dimensional dynamic structure data that is continuous in space and time.

[0009] In one specific embodiment, the hybrid asynchronous dual-modal imaging module comprises a global dynamic light field acquisition unit and an adaptive structured light probe unit. The global dynamic light field acquisition unit is used to continuously acquire and reconstruct the low-resolution global dynamic data using a light field microscope. The adaptive structured light probe unit is used to generate a customized structured light illumination pattern and acquire the high-resolution anchor point images according to the probe strategy.

[0010] In one specific embodiment, the spatio-temporal state analysis and uncertainty evaluation module comprises a high-resolution state generation unit, a differentiable physical prior embedding unit, a prediction uncertainty quantification unit, and a multi-source loss driven model training unit. The high-resolution state generation unit is used to generate the high-resolution three-dimensional structure prediction V pred based on the low-resolution global dynamic data through a generator network G. The differentiable physical prior embedding unit is used to define a differentiable physical prior that describes the physical properties of chromosomes and outputs a physical constraint loss L phy .

[0011] In one embodiment, the physical constraint loss is calculated based on a chromosome abstraction as a flexible polymer chain model composed of particles and springs, and the physical constraint loss includes continuity constraint L cont and volume repulsion constraint L excl The calculation formula is as follows:

[0012]

[0013] L phy = a cont L cont + a excl L excl ;

[0014] where p i is the three-dimensional coordinate of the i-th particle extracted from the high-resolution three-dimensional structure prediction, M is the total number of particles, l0is the ideal distance between adjacent particles, r0is the effective radius of the particle, a cont and a excl are preset weight coefficients. The prediction uncertainty quantification unit is used to perform multiple random forward propagation on the high-resolution three-dimensional structure prediction, calculate and output the uncertainty map U(t).

[0015] In one embodiment, the uncertainty map is the pixel-by-pixel variance of multiple high-resolution three-dimensional structure prediction results generated by the multiple random forward propagation. The calculation formula is as follows:

[0016]

[0017] where K is the total number of forward propagation, is the prediction result generated by the k-th forward propagation, is the average value of the k-th prediction result. The multi-source loss driven model training unit is used to correct the model parameters of the high-resolution state generation unit based on the high-resolution anchor point image and the physical constraint loss.

[0018] In one embodiment, the multi-source loss driven model training unit is configured to receive the high-resolution anchor point image collected by the hybrid asynchronous dual-modality imaging module, and calculate anchor point loss L anchor , which is a key component for correcting the model parameters of the high-resolution state generation unit.

[0019] In one specific embodiment, the closed-loop feedback and probe strategy generation module comprises a probe trigger decision unit and an optimal probe pattern calculation unit. The probe trigger decision unit directly receives the uncertainty map output by the prediction uncertainty quantification unit of the spatio-temporal state resolution and uncertainty assessment module as the only basis for its decision. When the uncertainty value in the uncertainty map exceeds a preset threshold, a probe is triggered and a target spatial region is locked. The optimal probe pattern calculation unit is used to calculate an optimal illumination pattern I opt for the target spatial region, and the optimal illumination pattern and the target spatial region together constitute the probe strategy. In one embodiment, the optimal probe pattern calculation unit calculates the optimal illumination pattern to maximize the expected information gain, so as to maximize the reduction of the uncertainty of the high-resolution three-dimensional structure prediction in the target spatial region with limited photon budget. Its goal can be described by the following optimization problem:

[0020]

[0021] where I is the illumination pattern, R probe is the target spatial region, V new is the newly collected data, θ G is the model parameter of the high-resolution state generation unit, is the historical data, and H(·) is the calculation of entropy.

[0022] In one specific embodiment, the adaptive structured light probe unit in the hybrid asynchronous dual-modality imaging module receives the optimal illumination pattern output by the optimal probe pattern calculation unit of the closed-loop feedback and probe strategy generation module to perform one directional high-resolution imaging, thereby generating the high-resolution anchor point image.

[0023] The present application provides a chromosome three-dimensional structure light field mapping and spatio-temporal alignment multi-modality fusion segmentation system.

[0024] Has the following beneficial effects:

[0025] 1、The present application sets up a hybrid asynchronous dual-modality imaging module, uses a global dynamic light field acquisition unit to continuously and dynamically capture the sample under low light, and at the same time uses a closed-loop feedback and probe strategy generation module to drive an adaptive structured light probe unit to perform one-time high-resolution imaging on a key region only when necessary. This working mode avoids continuous high-intensity light irradiation on the entire sample, ensures the acquisition of high temporal resolution global dynamics and high spatial resolution local structures, and significantly reduces the damage of phototoxicity and photobleaching effect to living cell samples, thereby ensuring the authenticity of biological process observation.

[0026] 2、The application realizes online correction and adaptive optimization of the model by constructing a self-supervised closed loop composed of a space-time state analysis and uncertainty evaluation module and a closed-loop feedback and probe strategy generation module. The system uses high-resolution anchor images collected by itself as internal supervision signals, and continuously fine-tunes model parameters through a model training unit driven by multi-source loss, thereby eliminating the dependence on large-scale, pre-manual labeled training data sets. This design greatly reduces the cost and cycle of technical implementation, and improves the adaptability of the system to different samples and experimental conditions.

[0027] 3、The application integrates physical constraint losses describing characteristics such as chromosome continuity and volume exclusion into model training by setting a differentiable physical prior embedding unit in the space-time state analysis and uncertainty evaluation module, forcing high-resolution three-dimensional structure prediction to follow basic biophysical laws. In addition, the closed-loop feedback mechanism can actively identify and utilize adaptive structure light probe units to explore areas with high model prediction uncertainty, further ensuring the physical authenticity and structural accuracy of the final output four-dimensional dynamic structure data in areas with complex topology or severe dynamic changes. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The structural schematic diagram of a chromosome three-dimensional structure light field mapping and space-time alignment multi-modal fusion segmentation system of an embodiment of the application;

[0029] Figure 2 The functional block diagram of a hybrid asynchronous dual-modal imaging module of an embodiment of the application;

[0030] Figure 3 The functional block diagram of a space-time state analysis and uncertainty evaluation module of an embodiment of the application;

[0031] Figure 4 The functional block diagram of a closed-loop feedback and probe strategy generation module of an embodiment of the application;

[0032] Figure 5 The workflow diagram of a chromosome three-dimensional structure light field mapping and space-time alignment multi-modal fusion segmentation system of an embodiment of the application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the specification of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the application.

[0034] Please refer to the drawings in the specification of the application.Figure 1 , attached Figure 1 is a structural schematic diagram of a chromosome three-dimensional structure light field mapping and spatiotemporal alignment multi-modal fusion segmentation system according to an embodiment of the present application. The present application provides a chromosome three-dimensional structure light field mapping and spatiotemporal alignment multi-modal fusion segmentation system, which can include a hybrid asynchronous dual-modal imaging module 10, a spatiotemporal state analysis and uncertainty evaluation module 20, a closed-loop feedback and probe strategy generation module 30, and a four-dimensional high-resolution structure fusion output module 40.

[0035] The system is built on a set of inverted microscope bodies and is configured with a culture chamber for maintaining the physiological state of living cell samples. All hardware components of the system are uniformly time-controlled and data-processed by a central computing server. The hybrid asynchronous dual-modal imaging module 10 is responsible for the acquisition of physical image data. The functions of the spatiotemporal state analysis and uncertainty evaluation module 20, the closed-loop feedback and probe strategy generation module 30, and the four-dimensional high-resolution structure fusion output module 40 are realized by software programs in the central computing server.

[0036] Please refer to attached Figure 2 , attached Figure 1 is a functional block diagram of a hybrid asynchronous dual-modal imaging module according to an embodiment of the present application. The hybrid asynchronous dual-modal imaging module 10 physically integrates two independent and collaborative light paths to complete imaging of different modes of the sample. The hybrid asynchronous dual-modal imaging module 10 includes a global dynamic light field acquisition unit 11 and an adaptive structure light probe unit 12.

[0037] The hardware composition of the global dynamic light field acquisition unit 11 includes a main objective lens, an imaging lens barrel, a microlens array placed on the imaging surface of the main objective lens, and a first image sensor located at the back focal plane of the microlens array. The function of the global dynamic light field acquisition unit 11 is to acquire raw four-dimensional light field image data from the first image sensor through a single exposure at consecutive time points. This raw data is then transmitted to the central computing server for calculation to reconstruct low spatial resolution three-dimensional volume data.

[0038] The hardware composition of the adaptive structure light probe unit 12 includes one or more laser light sources, a set of illumination light path optical elements, a two-dimensional spatial light modulator, and a second image sensor. The two-dimensional spatial light modulator is located in the illumination light path and is used to modulate the spatial phase or amplitude of the light beams emitted by the laser light sources to form a specific structure light illumination pattern on the focal plane of the main objective lens. The second image sensor is used to collect the fluorescence signals emitted by the sample under the specific structure light illumination pattern.

[0039] To ensure the spatial alignment of the image data acquired by the global dynamic light field acquisition unit 11 and the adaptive structured light probe unit 12, the system uses a calibration sample containing a plurality of sub-resolution fluorescent microspheres at the initialization. The calibration sample in the same field of view is imaged by the two units respectively, and two sets of image data are obtained. The central computing server realizes the accurate registration of the three-dimensional coordinate systems of the two units by calculating the affine transformation matrix of the fluorescent microsphere centroid coordinates in the two sets of image data. This transformation matrix is used to unify the spatial coordinates of all image data in subsequent system operation.

[0040] The working mode of the hybrid asynchronous dual-mode imaging module 10 is asynchronous. The central computing server sends continuous acquisition trigger signals to the first image sensor of the global dynamic light field acquisition unit 11 to realize high-speed and uninterrupted imaging of the sample. At the same time, the central computing server only sends single and coordinated trigger signals to the laser light source, the two-dimensional spatial light modulator and the second image sensor of the adaptive structured light probe unit 12 to complete a high-resolution anchor point image acquisition once when receiving the probe instruction from the closed-loop feedback and probe strategy generation module 30. The asynchronous working mode of the two units is precisely managed by the timing control program of the central computing server.

[0041] Please refer to the accompanying Figure 3 , accompanying Figure 3 is a functional block diagram of the spatiotemporal state analysis and uncertainty evaluation module according to an embodiment of the present application. The spatiotemporal state analysis and uncertainty evaluation module 20 is the computing processing core of the system, and its functions are realized by a set of interrelated software units deployed on the central computing server. The module 20 receives the low-resolution global dynamic data output by the hybrid asynchronous dual-mode imaging module 10, and is responsible for generating high-resolution prediction, quantifying uncertainty, and online correcting model parameters according to high-resolution anchor point images.

[0042] In one embodiment, the spatiotemporal state analysis and uncertainty evaluation module 20 includes a high-resolution state generation unit 21, a micro-physical prior embedding unit 22, a prediction uncertainty quantification unit 23, and a multi-source loss driven model training unit 24.

[0043] The high-resolution state generation unit 21 includes a generator network G and a discriminator network D. The generator network G adopts a U-Net network structure containing time sequence information processing capability, and its input is a plurality of low-resolution three-dimensional volume data sequences {V LFM (t-k),…,V LFM (t)} from the global dynamic light field acquisition unit 11 in time sequence, and its output is a high-resolution three-dimensional structure prediction V pred(t). The discriminator network D adopts a three-dimensional PatchGAN network structure, whose input is three-dimensional image data and output is a three-dimensional probability map, each value in the probability map corresponds to the probability of a local region in the input image being a real image.

[0044] The differentiable physical prior embedding unit 22 is used to convert the physical properties of the chromosome into a loss term that can be used for backpropagation of the neural network. This unit first generates a high-resolution three-dimensional structure prediction V pred from the output of the high-resolution state generation unit 21 in (t), and parameterizes this skeleton as a flexible polymer chain model connected by springs with M particles. Then, this unit calculates a physical constraint loss L phy , which is a weighted sum of the continuity constraint L phy , the volume repulsion constraint L cont , and the volume repulsion constraint L excl . The continuity constraint L cont is used to limit the distance between adjacent particles, and its calculation formula is:

[0045]

[0046] where p i is the three-dimensional space coordinate of the i-th particle, M is the total number of particles, and l0 is the preset ideal distance between adjacent particles. The volume repulsion constraint L excl is used to limit the minimum distance between non-adjacent particles, and its calculation formula is:

[0047]

[0048] where r0 is the preset effective radius of the particle. The complete calculation formula of the physical constraint loss L phy is:

[0049] L phy = α cont L cont + α excl L excl ;

[0050] where α cont and α excl are preset weight coefficients.

[0051] The prediction uncertainty quantification unit 23 is used to evaluate the reliability of the prediction results of the high-resolution state generation unit 21. In the reasoning phase of the system, this unit performs K independent forward propagations by keeping the random dropout layer open in the generator network G for the same input data, thereby obtaining a set of K high-resolution three-dimensional structure prediction results V Subsequently, the unit calculates the variance of the K prediction results at each three-dimensional pixel position, generating an uncertainty map U(t) of the same dimension as the prediction results. The calculation formula is:

[0052]

[0053] wherein, is the prediction result of the kth forward propagation, is the arithmetic mean of all k prediction results.

[0054] The multi-source loss-driven model training unit 24 is used to update the parameters of the generator network G and the discriminator network D in the high-resolution state generation unit 21 by optimizing a composite loss function. The total loss function L of the generator network G is G composed of four parts:

[0055] L G = λ adv L adv + λ content L content + λ anchor L anchor + λ phy L phy ;

[0056] wherein, λ adv , λ content , λ anchor and λ phy are the weight coefficients of each loss. The adversarial loss λ adv is used to drive the output of the generator network G to be statistically close to the real data distribution. The content consistency loss λ content is used to ensure that the generated result after downsampling is consistent with the input low-resolution global dynamic data. The anchor loss λ anchor is used to limit the generated result to be consistent with the high-resolution anchor image within the target space region specified by the probe strategy at the probe acquisition moment. The physical prior loss LphyLphy is calculated by the differentiable physical prior embedding unit 22. The loss function L of the discriminator network D D is used to improve its ability to distinguish real images from generated images. During the system operation, when a new high-resolution anchor image is obtained, the anchor loss L anchor is used to update the network parameters online.

[0057] Please refer to the attached Figure 4 , attached Figure 4is a functional block diagram of the closed-loop feedback and probe strategy generation module according to one embodiment of the present application. The closed-loop feedback and probe strategy generation module 30 is responsible for translating the abstract uncertainty information output by the spatio-temporal state resolution and uncertainty assessment module 20 into concrete hardware control instructions for the hybrid asynchronous dual-modality imaging module 10. The functions of the module 30 are implemented by software programs deployed on the central computing server.

[0058] In one embodiment, the closed-loop feedback and probe strategy generation module 30 includes a probe trigger decision unit 31 and an optimal probe pattern computation unit 32.

[0059] The input of the probe trigger decision unit 31 is the uncertainty map U(t) output by the predictive uncertainty quantification unit 23. The function of this unit is to monitor the numerical value of the uncertainty map in real time and decide whether to initiate a high-resolution probe imaging according to a pre-set rule. Specifically, this unit computes the maximum value max(U(t)) in the uncertainty map U(t) at each time point t and compares it with a pre-set uncertainty threshold τ. When max(U(t)) > τ, this unit generates a probe trigger instruction and records the 3D pixel coordinates of the maximum uncertainty value and its neighborhood, which defines the target spatial region R probe The uncertainty threshold τ can be a fixed value set according to prior experimental data or an adaptive value dynamically adjusted according to the statistical distribution of the uncertainty maps output by the system in the past period of time.

[0060] The optimal probe pattern computation unit 32 is invoked after receiving the probe trigger instruction and the target spatial region R probe from the probe trigger decision unit 31. The function of this unit is to compute an optimal structured illumination pattern I probe for the uncertainty distribution characteristics within the target spatial region R opt . The optimal goal is to maximize the expected information gain that can be obtained by this probe imaging, i.e., to minimize the uncertainty of the high-resolution 3D structure prediction in the target spatial region to the greatest extent with a limited photon budget. This optimization goal can be described by the following formula:

[0061]

[0062] where I opt is the optimal illumination pattern to be solved, I is any candidate illumination pattern, [ ] represents expectation, V new is the newly acquired image data within the region R probe under the illumination pattern I, P(V|I, R probe ) is the probability distribution of acquiring specific data under given conditions, and H(·) represents the calculation of information entropy, θG are the model parameters of the generator network G in the high-resolution state generation unit 21, while are the existing data of the system before the current probe imaging.

[0063] In the specific algorithm implementation, in order to ensure the real-time of the calculation, the above optimization problem is simplified as a look-up table and synthesis process. The optimal probe pattern calculation unit 32 first performs Fourier transform or gradient analysis on the uncertainty map in the target spatial region R probe to extract the main spatial frequency and direction features of the uncertainty distribution. Subsequently, the unit selects one or more basis function patterns that best match the extracted features from a pre-set orthogonal basis function library composed of different spatial frequency, direction and phase sinusoidal fringe patterns. The selected basis function pattern or its linear combination constitutes the optimal illumination pattern I opt . Finally, the probe strategy composed of the target spatial region R ptobe and the optimal illumination pattern I opt is output to the adaptive structured light probe unit 12 to perform a physical imaging operation.

[0064] Please refer to the accompanying Figure 5 , the accompanying Figure 5 is a workflow diagram of a chromosome three-dimensional structure light field mapping and spatiotemporal alignment multi-modal fusion segmentation system according to an embodiment of the present application. The overall workflow of the present application includes an initialization phase, a closed-loop iteration phase, and an end and output phase.

[0065] In the initialization phase, the system first prepares the hardware and software. This process includes placing the observed living cell sample in the incubation chamber of the microscope, completing the spatial coordinate system registration between the global dynamic light field acquisition unit 11 and the adaptive structured light probe unit 12 in the hybrid asynchronous dual-modal imaging module 10. At the same time, the central computing server loads a pre-trained model into the spatiotemporal state analysis and uncertainty evaluation module 20. This pre-trained model provides an initial parameter basis for subsequent online correction.

[0066] In the closed-loop iteration phase, the system is executed in a loop according to the following steps. First, the global dynamic light field acquisition unit 11 performs continuous low-resolution three-dimensional imaging of the sample to generate a time sequence of low-resolution global dynamic data V LFM (t), and transmits this data stream to the spatiotemporal state analysis and uncertainty evaluation module 20 in real time.

[0067] Next, the spatiotemporal state analysis and uncertainty evaluation module 20 receives and processes the data. The high-resolution state generation unit 21 inside it generates a high-resolution three-dimensional structure prediction V pred(t). Meanwhile, the prediction uncertainty quantification unit 23 performs uncertainty evaluation on the prediction result, generating a pixel-wise uncertainty map U(t).

[0068] Subsequently, the uncertainty map U(t) is transmitted to the closed-loop feedback and probe strategy generation module 30. The probe trigger decision unit 31 analyzes the uncertainty map U(t) to determine whether its maximum value exceeds a preset threshold τ. If it does not exceed, the system does not perform a probe operation, and directly processes the data at the next time point. If it exceeds, the probe trigger decision unit 31 outputs a probe trigger instruction, and determines the target spatial region R probe .

[0069] After the probe is triggered, the optimal probe pattern calculation unit 32 immediately calculates an optimal illumination pattern I probe for the target spatial region R opt . The probe strategy composed of the target spatial region R probe and the optimal illumination pattern I opt is sent to the adaptive structured light probe unit 12. The adaptive structured light probe unit 12 performs a high-resolution imaging according to the probe strategy, and acquires a local, high-resolution anchor point image V anchor .

[0070] Then, the high-resolution anchor point image V anchor is fed back to the spatiotemporal state analysis and uncertainty evaluation module 20. The multi-source loss-driven model training unit 24 calculates an anchor point loss L anchor using the anchor point image, and performs several cycles of online fine-tuning of the model parameters in the high-resolution state generation unit 21 in combination with other loss terms. This step completes a modification of the model.

[0071] The above prediction, quantification, decision, detection, and modification steps constitute a complete closed-loop iteration cycle of the system. This closed-loop iteration runs throughout the observation period of the dynamic process of living cells, enabling the system to continuously optimize itself.

[0072] In the end and output stage, when the entire dynamic process observation ends, the four-dimensional high-resolution structure fusion output module 40 is responsible for collecting the high-resolution three-dimensional structure predictions output by the spatiotemporal state analysis and uncertainty evaluation module 20 at all time points and after complete closed-loop iteration modification. This module integrates the three-dimensional data at these continuous time points, and finally outputs a complete, spatiotemporally continuous four-dimensional dynamic structure data set, which can be used for subsequent data analysis and visualization.

[0073] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. A system for multi-modal fusion segmentation of chromosome 3D structure light field mapping and spatio-temporal alignment, characterized in that, The method comprises the following steps: a hybrid asynchronous dual-modality imaging module is used to acquire low-resolution global dynamic data and high-resolution anchor point images of a living cell sample; a spatiotemporal state analysis and uncertainty evaluation module is used to generate high-resolution three-dimensional structure prediction based on the low-resolution global dynamic data, and to quantify the uncertainty of the high-resolution three-dimensional structure prediction synchronously, forming an uncertainty atlas, and to correct the model of the spatiotemporal state analysis and uncertainty evaluation module online using the high-resolution anchor point images; a closed-loop feedback and probe strategy generation module is used to generate a probe strategy for guiding the acquisition of the high-resolution anchor point images based on the uncertainty atlas; a four-dimensional high-resolution structure fusion output module is used to collect and integrate the high-resolution three-dimensional structure prediction output by the spatiotemporal state analysis and uncertainty evaluation module after online correction, forming four-dimensional dynamic structure data that is continuous in space and time.

2. The chromosome 3D structure light field mapping and spatio-temporal alignment multi-modal fusion segmentation system according to claim 1, wherein, The hybrid asynchronous dual-modality imaging module comprises: a global dynamic light field acquisition unit is used to continuously acquire and reconstruct the low-resolution global dynamic data using a light field microscope; an adaptive structured light probe unit is used to generate a customized structured light illumination pattern according to the probe strategy and acquire the high-resolution anchor point images.

3. The chromosome 3D structure light field mapping and spatio-temporal alignment multi-modal fusion segmentation system according to claim 1, wherein, The spatiotemporal state analysis and uncertainty evaluation module comprises: a high-resolution state generation unit is used to generate the high-resolution three-dimensional structure prediction based on the low-resolution global dynamic data; a differentiable physical prior embedding unit is used to define a differentiable physical prior that describes the physical properties of chromosomes and output a physical constraint loss; a predicted uncertainty quantification unit is used to perform multiple random forward propagations on the high-resolution three-dimensional structure prediction, calculate and output the uncertainty atlas; a multi-source loss driven model training unit is used to correct the model parameters of the high-resolution state generation unit online based on the high-resolution anchor point images and the physical constraint loss.

4. The chromosome 3D structure light field mapping and spatio-temporal alignment multi-modal fusion segmentation system according to claim 3, wherein, The physical constraint loss output by the differentiable physical prior embedding unit is calculated based on a flexible polymer chain model that abstracts chromosomes as particles and springs, and the physical constraint loss comprises continuity constraints and volume repulsion constraints.

5. The chromosome 3D structure light field mapping and spatio-temporal alignment multi-modal fusion segmentation system according to claim 3, wherein, The uncertainty atlas output by the predicted uncertainty quantification unit is the pixel-wise variance of multiple high-resolution three-dimensional structure prediction results generated by the multiple random forward propagations.

6. The chromosome 3D structure light field mapping and spatio-temporal alignment multi-modal fusion segmentation system according to claim 2, wherein, The closed-loop feedback and probe strategy generation module comprises: a probe trigger decision unit is used to monitor the uncertainty atlas and trigger a probe and lock a target spatial region when the uncertainty value in the uncertainty atlas exceeds a preset threshold; an optimal probe pattern calculation unit is used to calculate an optimal illumination pattern for the target spatial region, and the optimal illumination pattern and the target spatial region together constitute the probe strategy.

7. The chromosome 3D structure light field mapping and spatio-temporal alignment multi-modal fusion segmentation system according to claim 6, wherein, The target of the optimal probe pattern calculation unit for calculating the optimal illumination pattern is to maximize the expected information gain, so as to reduce the uncertainty of the high-resolution three-dimensional structure prediction in the target spatial region to the greatest extent using a limited photon budget.

8. The chromosome 3D structure light field mapping and spatio-temporal alignment multi-modal fusion segmentation system according to claim 6, wherein, The probe trigger decision unit directly receives the uncertainty atlas output by the spatio-temporal state analysis and uncertainty evaluation module as the only basis for its decision.

9. The chromosome 3D structure light field mapping and spatio-temporal alignment multi-modal fusion segmentation system of claim 6, wherein, The adaptive structured light probe unit in the hybrid asynchronous dual-modal imaging module receives the optimal illumination pattern output by the optimal probe pattern calculation unit of the closed-loop feedback and probe strategy generation module to perform one-time directional high-resolution imaging.

10. The chromosome 3D structure light field mapping and spatio-temporal alignment multi-modal fusion segmentation system of claim 3, wherein, The multi-source loss-driven model training unit is configured to receive the anchor point images collected by the hybrid asynchronous dual-modal imaging module and calculate anchor point loss using the anchor point images, which is a key component for online correction of the model parameters of the high-resolution state generation unit.

Images