Model training method and three-dimensional reconstruction method of fluorescent light source

Abstract

The application provides a model training method and a three-dimensional reconstruction method of a fluorescent light source, and can be applied to the technical field of image reconstruction. The model training method comprises the following steps: based on Monte Carlo light forward propagation, sequentially forward transmitting a simulated internal fluorescent light source distribution according to time steps to obtain a simulated fluorescent intensity distribution of each time step; training an initial three-dimensional reconstruction model by using the simulated fluorescent intensity distribution of the plurality of time steps until a loss function reaches a preset condition, to obtain a trained three-dimensional reconstruction model; wherein the input of the three-dimensional reconstruction model is the simulated fluorescent intensity distribution of a current time step randomly obtained from the plurality of time steps, and the output of the three-dimensional reconstruction model is a predicted value of the fluorescent intensity distribution of a previous time step; and the loss function represents the difference between the predicted value of the fluorescent intensity distribution of the previous time step and the simulated fluorescent intensity distribution of the previous time step.

Description

Technical Field

[0001] At least one embodiment of this application relates to the field of image reconstruction technology, and more specifically to a model training method and a three-dimensional reconstruction method using a fluorescent light source. Background Technology

[0002] Fluorescence molecular tomography (FLCT) methods employ simplified approximate models of the radiative transfer equation (RTE) to simulate the forward photon propagation process of a light source. These simplified models are primarily based on diffusion equations with Robin boundary conditions. A critical and ill-conditioned inverse problem needs to be solved based on this simplified model: how to reconstruct the three-dimensional distribution of the light source from two-dimensional intensity information. Introducing regularization terms, higher-order models, or prior information can alleviate the ill-conditioning of the problem, establishing a more realistic photon propagation model and thus improving reconstruction accuracy. However, these methods inevitably rely on simplified photon propagation models, making the aforementioned inverse problem difficult to solve accurately, ultimately limiting reconstruction precision. Summary of the Invention

[0003] In view of the above problems, this application provides a model training method to improve reconstruction accuracy and a three-dimensional reconstruction method using a fluorescent light source.

[0004] According to a first aspect of this application, a model training method is provided, comprising: based on Monte Carlo forward propagation, transmitting a simulated internal fluorescence light source distribution sequentially forward according to time steps to obtain simulated fluorescence intensity distributions for multiple time steps; training an initial three-dimensional reconstruction model using the simulated fluorescence intensity distributions for the multiple time steps until a loss function reaches a preset condition to obtain a trained three-dimensional reconstruction model; wherein the input of the three-dimensional reconstruction model is the simulated fluorescence intensity distribution of the current time step randomly obtained from the multiple time steps, and the output of the three-dimensional reconstruction model is the predicted value of the fluorescence intensity distribution of the previous time step; the loss function characterizes the difference between the predicted value of the fluorescence intensity distribution of the previous time step and the simulated fluorescence intensity distribution of the previous time step.

[0005] The second aspect of this application provides a three-dimensional reconstruction method for a fluorescent light source. The three-dimensional reconstruction method includes: acquiring a white light image, a fluorescence image, and a computer-generated tomographic image of a target object; registering the fluorescence image to a grid formed by the computer-generated tomographic image using positioning points based on the white light image to obtain a surface fluorescence intensity distribution; determining N time steps based on the grid; determining the surface fluorescence intensity distribution as the input data for the Nth time step; inputting the input data for the i-th time step into the trained three-dimensional reconstruction model to obtain the fluorescence intensity distribution for the (i-1)-th time step; where i is an integer greater than 1 and less than or equal to N; and determining the fluorescence intensity distribution for the first time step as the internal fluorescence light source distribution of the target object.

[0006] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0007] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0008] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0009] According to embodiments of this application, based on Monte Carlo forward propagation, simulated fluorescence intensity distributions for multiple time steps are obtained. These simulated fluorescence intensity distributions are used as a training set. Supervised backpropagation training of the 3D reconstruction model is then performed using the simulated fluorescence intensity distributions of adjacent time steps, resulting in a trained 3D reconstruction model. Based on this trained 3D reconstruction model, the surface fluorescence distribution obtained from the detector can be backpropagated and reconstructed step-by-step to obtain the internal fluorescence light source. The model training method of this application, based on Monte Carlo forward propagation, provides a more realistic photon propagation process. Compared to relying on a simplified photon propagation model, this more realistic photon propagation process guides the 3D reconstruction module in reconstructing the light source, effectively solving the problem of the difficulty in accurately solving the inverse problem and improving the reconstruction accuracy of the 3D reconstruction model. Attached Figure Description

[0010] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments of this application with reference to the accompanying drawings.

[0011] Figure 1A flowchart illustrating the operation of a model training method according to an embodiment of this application is shown.

[0012] Figure 2 A schematic diagram of a simulated forward propagation process of light according to an embodiment of this application is shown.

[0013] Figure 3 A schematic diagram of the inverse prediction of a three-dimensional reconstruction model according to an embodiment of this application is shown.

[0014] Figure 4 A schematic diagram of a selective state-space model according to an embodiment of this application is shown.

[0015] Figure 5 A schematic diagram of the Monte Carlo-based forward propagation process according to an embodiment of this application is shown.

[0016] Figure 6A A flowchart illustrating the operation of a three-dimensional reconstruction method for a fluorescent light source according to an embodiment of this application is shown.

[0017] Figure 6B A flowchart illustrating the operation of cyclically inputting a trained 3D reconstruction model according to an embodiment of this application is shown.

[0018] Figure 7 A schematic diagram of a model training apparatus according to an embodiment of this application is shown.

[0019] Figure 8 A schematic diagram of a three-dimensional reconstruction apparatus for a fluorescent light source according to an embodiment of this application is shown.

[0020] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a three-dimensional reconstruction method for a fluorescent light source according to an embodiment of this application. Detailed Implementation

[0021] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0024] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0025] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0026] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0027] Fluorescent molecular imaging is an imaging method that uses fluorescent probes to detect tumors within biological tissues. Its development has made real-time imaging-guided neurosurgery possible, helping to improve the complete tumor resection rate. However, this technology can only acquire two-dimensional light intensity images of the tumor and cannot accurately characterize the three-dimensional morphology and location of the actual tumor. To address this, fluorescent molecular tomography has been gradually developed, which reconstructs the distribution of fluorescent probes to achieve three-dimensional visualization of the tumor.

[0028] The embodiments of this application provide a model training method.

[0029] Figure 1A flowchart illustrating the operation of a model training method according to an embodiment of this application is shown.

[0030] like Figure 1 As shown, the model training method includes operation S110 and operation S120.

[0031] In operation S110, based on Monte Carlo forward propagation, the simulated internal fluorescence light source distribution is transmitted forward sequentially according to time steps to obtain the simulated fluorescence intensity distribution for each of the multiple time steps.

[0032] In operation S120, the initial three-dimensional reconstruction model is trained using the simulated fluorescence intensity distribution at multiple time steps until the loss function reaches the preset condition, thus obtaining the trained three-dimensional reconstruction model.

[0033] According to an embodiment of this application, the input of the three-dimensional reconstruction model is the simulated fluorescence intensity distribution of the current time step, which is randomly obtained from multiple time steps, and the output of the three-dimensional reconstruction model is the predicted value of the fluorescence intensity distribution of the previous time step; the loss function characterizes the difference between the predicted value of the fluorescence intensity distribution of the previous time step and the simulated fluorescence intensity distribution of the previous time step.

[0034] In simulation experiments, fluorescent light sources of different shapes are designed to simulate the distribution of fluorescent light sources within the simulation. Based on the prior physical knowledge of Monte Carlo methods for forward light propagation, the forward light propagation process of the fluorescent light source distribution within the simulation is simulated. Forward light propagation represents the transmission process of light from the light source to the detector in a medium such as biological tissue. Monte Carlo is a random sampling method that can be used to simulate the random walk of photons step by step, such as the step size of each photon, whether it is absorbed or scattered, and in which direction it is scattered. Prior physical knowledge combines known and reliable physical laws and parameters with Monte Carlo simulation, making the simulation process more realistic.

[0035] Figure 2 A schematic diagram of a simulated forward propagation process of light according to an embodiment of this application is shown.

[0036] By setting a predetermined duration for each time step, we can obtain the simulated fluorescence intensity distribution for multiple time steps. This simulated fluorescence intensity distribution for each time step can be understood as the spatial distribution of fluorescence within each short predetermined time segment. By recording the distribution step by step, we can obtain the complete process of fluorescence distribution changing over time. For example... Figure 2As shown, the starting point of the forward propagation of light is the simulated internal fluorescence light source distribution. Propagating forward sequentially according to time steps, the simulated fluorescence intensity distribution at time step t is obtained. The simulated fluorescence intensity distribution at time step t is propagated forward one time step to obtain the simulated fluorescence intensity distribution at time step t+1. The simulated fluorescence intensity distribution at time step t+1 continues to propagate forward sequentially according to time steps to obtain the simulated fluorescence intensity distribution at time step T. t ≤ T.

[0037] The simulated internal fluorescence light source distribution and simulated fluorescence intensity distributions at multiple time steps are used as the training set. The simulated fluorescence intensity distribution at the current time step, randomly acquired, is input into the 3D reconstruction model. The 3D reconstruction model can output the predicted value of the fluorescence intensity distribution at the previous time step. Based on the difference between the predicted value of the fluorescence intensity distribution at the previous time step and the simulated fluorescence intensity distribution at the previous time step in the training set, the parameters of the 3D training model are adjusted. Then, a simulated fluorescence intensity distribution at a random time step is acquired from the training set and used as the simulated fluorescence intensity distribution at the current time step. The 3D reconstruction model is trained until the loss function reaches a preset condition, resulting in a well-trained 3D reconstruction model.

[0038] According to embodiments of this application, based on Monte Carlo forward propagation, simulated fluorescence intensity distributions for multiple time steps are obtained. These simulated fluorescence intensity distributions are used as a training set. Supervised backpropagation training of the 3D reconstruction model is then performed using the simulated fluorescence intensity distributions of adjacent time steps, resulting in a trained 3D reconstruction model. Based on this trained 3D reconstruction model, the surface fluorescence distribution obtained from the detector can be backpropagated and reconstructed step-by-step to obtain the internal fluorescence light source. The model training method of this application, based on Monte Carlo forward propagation, provides a more realistic photon propagation process. Compared to relying on a simplified photon propagation model, this more realistic photon propagation process guides the 3D reconstruction module in reconstructing the light source, effectively solving the problem of the difficulty in accurately solving the inverse problem and improving the reconstruction accuracy of the 3D reconstruction model.

[0039] The model training method of this application outputs a trained three-dimensional reconstruction model, which can be applied to tumor-related detection and reconstruction, and can improve the morphological restoration accuracy of tumor three-dimensional reconstruction.

[0040] In one example, the loss function could be based on the simulated fluorescence intensity distribution of the previous time step from the current time step in the training set. The predicted fluorescence intensity distribution of the previous time step at the current time step, as output by the 3D reconstruction model. Mean square error loss Mean squared error loss It can help 3D reconstruction models learn the mapping relationship between the fluorescence intensity distribution of the current time step and the previous time step, making the morphological recovery ability of 3D reconstruction models stronger.

[0041] Mean square error loss It can be represented as:

[0042] (1).

[0043] The 3D reconstruction model performs backpropagation of light based on the simulated fluorescence intensity distribution obtained at the current time step, and outputs the predicted value of the fluorescence intensity distribution at the previous time step. The network parameters of the 3D reconstruction model can be optimized using the stochastic gradient descent method to obtain a trained 3D reconstruction model. The stochastic gradient descent method calculates the gradient and updates the parameters using only one random sample at a time.

[0044] Deep learning-based tomographic imaging methods can directly solve the inverse problem without using a simplified model of RTE. However, because deep learning-based tomographic imaging methods do not have methods designed for the sequential structure characteristics of the data, the accuracy of tumor reconstruction is limited, and most of them do not provide physical prior knowledge of photon transport, resulting in weak interpretability.

[0045] According to an embodiment of this application, before inputting the three-dimensional reconstruction model, the simulated fluorescence intensity distribution at each time step is preprocessed. The preprocessing includes sorting the three-dimensional data of the simulated fluorescence intensity distribution at each time step according to a preset traversal rule to obtain the fluorescence sequence distribution for each time step. The three-dimensional data are read one by one according to the preset traversal rule and arranged into a sequence to obtain the fluorescence sequence distribution. The preset traversal rule can be the order of the three-dimensional coordinates of the three-dimensional data.

[0046] According to the embodiments of this application, the simulated fluorescence intensity distribution at each time step is preprocessed to obtain the fluorescence sequence distribution at each time step. The fluorescence sequence distribution at each time step is used to train the three-dimensional reconstruction model, which can reduce the complexity of data computation.

[0047] According to an embodiment of this application, the predicted value of the fluorescence intensity distribution at the previous time step is obtained by the following operation: using a neural network unit in a three-dimensional reconstruction model to predict the simulated fluorescence intensity distribution at the current time step obtained randomly, based on the reverse transmission of the time step, to obtain the predicted value of the fluorescence intensity distribution at the previous time step.

[0048] Figure 3 A schematic diagram of the inverse prediction of a three-dimensional reconstruction model according to an embodiment of this application is shown.

[0049] like Figure 3As shown, the 3D reconstruction model includes neural network units. A simulated fluorescence intensity distribution at a random time step, such as time step t+1, is obtained from the training set. This simulated fluorescence intensity distribution at time step t+1 is then input into the neural network unit, which can then predict the value of the simulated fluorescence intensity distribution at time step t.

[0050] According to embodiments of this application, such as Figure 3 As shown, the neural network unit in the 3D reconstruction model includes an extraction module and a prediction module.

[0051] The simulated fluorescence intensity distribution at the current time step is predicted by the neural network unit in the 3D reconstruction model based on the reverse propagation of the time step, and the predicted value of the fluorescence intensity distribution at the previous time step is obtained. This includes: using the extraction module to extract the fluorescence sequence distribution features of the simulated fluorescence intensity distribution at the current time step; and using the prediction module to reconstruct the fluorescence intensity distribution at the previous time step based on the fluorescence sequence distribution features at the current time step, and the predicted value of the fluorescence intensity distribution at the previous time step is obtained.

[0052] The distribution characteristics of fluorescence sequences refer to the statistical and structural regularities of fluorescence signals in terms of intensity, location, morphology, periodicity, and heterogeneity in a spatial sequence.

[0053] The prediction module can reconstruct the fluorescence sequence distribution characteristics at the current time step to rebuild the fluorescence intensity distribution at the previous time step.

[0054] According to embodiments of this application, such as Figure 3 As shown, the extraction module includes at least one selective state-space model. Using the extraction module, the fluorescence sequence distribution features of the simulated fluorescence intensity distribution at the current time step are extracted, including: using at least one selective state-space model to extract the fluorescence sequence distribution features of the simulated fluorescence intensity distribution at the current time step.

[0055] Selective state space modeling, also known as Mamba, is a linear-time-sequence modeling based on selective state spaces. It can model the distribution of fluorescence sequences at the current time step and adaptively extract the intensity distribution, morphological features, and heterogeneity patterns of fluorescence signals in the sequence dimension through an input-dependent dynamic parameter mechanism, thereby achieving efficient feature learning and key information screening of long fluorescence sequences.

[0056] like Figure 3As shown, multiple selective state-space models can be sequentially connected to extract the fluorescence sequence distribution features of the simulated fluorescence intensity distribution at the current time step. For example, four selective state-space models can fully explore the fluorescence distribution features in the data, thereby maximizing the accuracy and effect of the three-dimensional reconstruction model in reconstructing the light source.

[0057] According to embodiments of this application, such as Figure 3 As shown, the prediction module includes at least one linear layer. Using the prediction module, the fluorescence intensity distribution of the previous time step is reconstructed based on the fluorescence sequence distribution characteristics of the current time step to obtain a predicted value of the fluorescence intensity distribution of the previous time step. This includes: using at least one linear layer to reconstruct the fluorescence intensity distribution of the previous time step based on the fluorescence sequence distribution characteristics of the current time step to obtain a predicted value of the fluorescence intensity distribution of the previous time step.

[0058] Linear layers can be fully connected layers with activation functions. Multiple linear layers connected in series can form a deep fully connected network, which can fit complex patterns, automatically extract information from the fluorescence sequence distribution features at the current time step, and improve the expressive power of the 3D reconstruction model.

[0059] According to embodiments of this application, the selective state-space model includes a first path, a second path, a fusion unit, and a mapping unit.

[0060] Using at least one selective state-space model, the fluorescence sequence distribution features of the simulated fluorescence intensity distribution at the current time step are extracted, including: inputting the simulated fluorescence intensity distribution at the current time step into a first path to obtain dynamic gating weights; inputting the simulated fluorescence intensity distribution at the current time step into a second path to obtain the sequence features to be modulated; inputting the dynamic gating weights and the sequence features to be modulated into a fusion unit for multiplication to obtain the gated weighted modulated sequence features; and inputting the gated weighted modulated sequence features into a mapping unit for linear mapping to obtain the fluorescence sequence distribution features of the simulated fluorescence intensity distribution at the current time step.

[0061] Figure 4 A schematic diagram of a selective state-space model according to an embodiment of this application is shown.

[0062] like Figure 4As shown, before inputting the selective state-space model, the simulated fluorescence intensity distribution at time step t+1 can be layer-normalized. The layer-normalized simulated fluorescence intensity distribution at time step t+1 is then input into the first and second paths respectively. Layer normalization standardizes the simulated fluorescence intensity distribution at time step t+1, making the 3D reconstruction model more stable. In the first path, the simulated fluorescence intensity distribution at time step t+1 undergoes the following operations in sequence: linear mapping using a linear layer, one-dimensional convolution, activation using an activation function, modeling using the state-space model, and standardization using layer normalization. The first path outputs dynamic gating weights. The activation function can be the SiLU (Sigmoid Linear Unit) activation function. The state-space model (SSM) is suitable for processing very long sequences. In the second path, the simulated fluorescence intensity distribution at time step t+1 undergoes linear mapping using a linear layer and activation using an activation function in sequence to obtain the sequence features to be modulated. The dynamic gating weights and the sequence features to be modulated are multiplied element-wise to obtain the gated weighted modulated sequence features. The mapping unit can be a linear layer. The linear layer is used to perform a linear mapping on the gated weighted modulated sequence features to obtain the fluorescence sequence distribution features of the simulated fluorescence intensity distribution at time step t.

[0063] According to an embodiment of this application, the model training method further includes: performing data homogenization processing on the simulated internal fluorescence light source distribution and the simulated fluorescence intensity distribution at each time step.

[0064] The average value of all light intensities is based on the simulated internal fluorescence light source distribution and the simulated fluorescence intensity distribution at each time step. and standard deviation Perform homogenization:

[0065] (2);

[0066] in, The original light intensity value of a single point in the simulation of the internal fluorescence light source distribution and the simulation fluorescence intensity distribution at each time step; This provides a normalized light intensity for individual points in the simulated internal fluorescence light source distribution and the simulated fluorescence intensity distribution at each time step. It brings the brightness and intensity range of the simulated internal fluorescence light source distribution and the simulated fluorescence intensity distribution at each time step to the same standard, preventing interference from brightness magnitude during model training. This allows the 3D training model to stably, quickly, and accurately learn the shape of light intensity.

[0067] According to an embodiment of this application, based on Monte Carlo forward propagation, the simulated internal fluorescence light source distribution is sequentially propagated forward according to time steps to obtain the simulated fluorescence intensity distribution at each time step. This includes: for photons in the simulated fluorescence intensity distribution at the current time step, determining the step size of the photon based on the total attenuation coefficient of the photon in the medium and a first Monte Carlo random number; determining the position of the photon in the next time step based on the unit propagation direction of the photon in the next time step, the position of the photon in the current time step, and the step size of the photon; determining the energy of the photon in the next time step based on the energy, total attenuation coefficient, and absorption coefficient of the photon in the current time step; and obtaining the simulated fluorescence intensity distribution at the next time step based on the position of the photon in the next time step corresponding to each photon in the simulated fluorescence intensity distribution at the current time step and the energy of the photon in the next time step, thereby obtaining the simulated fluorescence intensity distribution at each time step.

[0068] Figure 5 A schematic diagram of the Monte Carlo-based forward propagation process according to an embodiment of this application is shown.

[0069] In one example, the medium could be biological tissue. The initial unit transmission direction of the photon. and the initial position of the photon It is obtained based on the simulated distribution of internal fluorescent light sources.

[0070] like Figure 5 As shown, based on the fact that the photon free propagation step size follows an exponential distribution, a random step size is obtained by sampling using the inverse transform method. This is based on the total attenuation coefficient of photons in the medium. And the first Monte Carlo random number Determine the step length of the photon Total attenuation coefficient It is the sum of the absorption coefficient and the scattering coefficient. First Monte Carlo random number. A random number uniformly distributed in the interval [0,1]. The step length of the photon. To simulate the step size of a photon's journey through biological tissue, we can simulate the distance a photon travels in a straight line through the tissue before absorption or scattering. The step size of a photon... It can be represented as:

[0071] (3).

[0072] Based on the unit propagation direction of the photon at the nth time step The position of the photon at time step n The progress of photons Determine the position of the photon at time step n+1. Along the unit propagation direction of the photon at time step n. The stride length of a linearly moving photon Update the position of the photon at time step n+1. The position of the photon at time step (n+1). It can be represented as:

[0073] (4).

[0074] Based on the energy of the photon at time step n Total attenuation coefficient and absorption coefficient Determine the energy of the photon at time step n+1. During propagation at each time step, energy is lost according to the absorption probability, and the remaining energy continues to propagate. The energy of the photon at time step n+1 is... It can be represented as:

[0075] (5).

[0076] Based on the position of the photon at time step (n+1) corresponding to each photon in the simulated fluorescence intensity distribution at time step n. The energy of the photon at time step n+1 The simulated fluorescence intensity distribution at time step n+1 is obtained, thus yielding the simulated fluorescence intensity distribution at each time step.

[0077] It determines whether the photon has reached the tissue surface. If it has, it stops; otherwise, it continues to propagate, which involves further updating the photon's energy, sampling the scattering deflection angle, and updating the unit transmission direction.

[0078] Based on the anisotropy factor g of biological tissues and the first Monte Carlo random number The scattering deflection angle θ of the photon is obtained.

[0079] (6);

[0080] The scattering deflection angle θ∈[0,π] can be obtained from cosθ.

[0081] Based on the photon scattering deflection angle θ and the rotation angle corresponding to the second Monte Carlo random number Determine the unit propagation direction of the photon at time step n+1. Photon scattering from biological tissue can be sampled using the Henyey-Greenstein phase function to obtain the scattering deflection angle θ, and the rotation angle corresponding to the second Monte Carlo random number. [0,2] A random number uniformly distributed within an interval. n is an integer.

[0082] The unit transmission direction of the photon at time step n Around the local coordinate system, according to the scattering deflection angle θ and the rotation angle Rotate to obtain the unit propagation direction of the photon at time step n+1. The unit propagation direction of the photon at time step n+1. It can be represented as:

[0083] (7);

[0084] in, This represents coordinate transformation.

[0085] The embodiments of this application also propose a three-dimensional reconstruction method for a fluorescent light source.

[0086] Figure 6A A flowchart illustrating the operation of a three-dimensional reconstruction method for a fluorescent light source according to an embodiment of this application is shown.

[0087] like Figure 6A As shown, the three-dimensional reconstruction method of the fluorescent light source includes operations S610 to S660.

[0088] In operation S610, white light images, fluorescence images, and computer-generated tomographic images of the target object are acquired.

[0089] In one example, the target object can be a body part of the detected object, such as the head, chest, or abdomen. A white light image refers to an image obtained by photographing the target object using visible light illumination, providing a reference point for the object's real-world outline. Tumors within the body part of the detected object are specifically labeled, causing them to emit specific light, visualizing the tumor's boundaries, and acquiring fluorescence images. Computed tomography (CT) systems are used to acquire computed tomographic images of the internal structures.

[0090] The three-dimensional reconstruction method for fluorescent light sources also includes noise reduction operations on white light images, fluorescent images, and computer-generated tomographic images.

[0091] In operation S620, the fluorescence image is registered to the grid formed by the internal tomographic image of the computer based on the white light image through positioning points to obtain the surface fluorescence intensity distribution.

[0092] Computed tomography (CT) systems can create a grid based on acquired computed tomographic images. This grid can be a three-dimensional voxel representation of a tumor within the body part being examined. Using the relatively clear outline of the real-world scene provided by the white light image as a reference, feature points at the same locations are selected as localization points. Through spatial point coordinate matching, the fluorescence image is registered onto the grid provided by the CT system, resulting in the surface fluorescence intensity distribution. This surface fluorescence intensity distribution represents the intensity data of the three-dimensional grid surface.

[0093] In operating the S630, N time steps are determined based on the grid.

[0094] In operation S640, the surface fluorescence intensity distribution is determined as the input data for the Nth time step.

[0095] In operation S650, for the input data at the i-th time step, the input data at the i-th time step is input into the trained 3D reconstruction model to obtain the fluorescence intensity distribution at the (i-1)-th time step. i is an integer greater than 1 and less than or equal to N.

[0096] In operation S660, the fluorescence intensity distribution of the first time step is determined as the internal fluorescence light source distribution of the target object.

[0097] Before inputting the trained 3D reconstruction model, the surface fluorescence intensity distribution can be sorted according to a preset traversal rule. This preset traversal rule can be based on positional sorting. Based on the mapping relationship, the intensity data of the 3D mesh surface is converted into a one-dimensional vector with positional information, resulting in the fluorescence sequence distribution of the surface fluorescence intensity distribution.

[0098] Figure 6B A flowchart illustrating the operation of cyclically inputting a trained 3D reconstruction model according to an embodiment of this application is shown.

[0099] like Figure 6B As shown,

[0100] In operation S651, let i = N.

[0101] In operation S650, for the input data of the i-th time step, the input data of the i-th time step is input into the trained three-dimensional reconstruction model to obtain the fluorescence intensity distribution of the (i-1)-th time step.

[0102] In operation S652, let i = i-1.

[0103] In operation S653, it is determined that i < 2. If the determination is yes, operation S660 is executed; if the determination is no, the process returns to executing operation S650.

[0104] In operation S660, the fluorescence intensity distribution of the first time step is determined as the internal fluorescence light source distribution of the target object.

[0105] According to an embodiment of this application, the fluorescence image is obtained using the near-infrared II region as the fluorescence light source.

[0106] The Second Near-Infrared Window (NIR-II) band spans 1000 nm to 1700 nm, while the First Near-Infrared Window (NIR-I) band spans 700 nm to 900 nm. Light scattering in NIR-II is weaker than in NIR-I, resulting in less interference from tissue scattering and refraction. NIR-II can penetrate deeper into biological tissues, enabling deep in vivo imaging and overcoming the limitations of insufficient imaging depth in NIR-I. The low scattering in NIR-II preserves the contours and details of the target object, resulting in higher imaging accuracy and further enhancing the morphological recovery of tumors.

[0107] According to the embodiments of this application, the distribution of internal fluorescent light sources of the target object is displayed by the image display unit, which can intuitively, clearly and from multiple angles display the three-dimensional morphological image of the tumor.

[0108] Based on the above model training method, this application also provides a model training device.

[0109] Figure 7 A schematic diagram of a model training apparatus according to an embodiment of this application is shown.

[0110] like Figure 7 As shown, the model training device 700 of this embodiment includes a first obtaining module 710 and a second obtaining module 720.

[0111] The first module 710 is used for Monte Carlo-based forward propagation of light, which transmits the simulated internal fluorescence light source distribution forward sequentially according to time steps to obtain the simulated fluorescence intensity distribution for each of the multiple time steps.

[0112] The second module 720 is used to train the initial three-dimensional reconstruction model using the simulated fluorescence intensity distribution of multiple time steps until the loss function reaches the preset condition, thereby obtaining the trained three-dimensional reconstruction model.

[0113] According to an embodiment of this application, the input of the three-dimensional reconstruction model is the simulated fluorescence intensity distribution of the current time step, which is randomly obtained from multiple time steps, and the output of the three-dimensional reconstruction model is the predicted value of the fluorescence intensity distribution of the previous time step; the loss function characterizes the difference between the predicted value of the fluorescence intensity distribution of the previous time step and the simulated fluorescence intensity distribution of the previous time step.

[0114] Based on the above-mentioned three-dimensional reconstruction method for fluorescent light sources, this application also provides a three-dimensional reconstruction device for fluorescent light sources.

[0115] Figure 8 A schematic diagram of a three-dimensional reconstruction apparatus for a fluorescent light source according to an embodiment of this application is shown.

[0116] like Figure 8 As shown, the three-dimensional reconstruction device 800 for the fluorescent light source in this embodiment includes an acquisition module 810, a third obtaining module 820, a first determining module 830, a second determining module 840, a fourth obtaining module 850, and a third determining module 860.

[0117] The acquisition module 810 is used to acquire white light images, fluorescence images, and computer-generated tomographic images of the target object.

[0118] The third module 820 is used to register the fluorescence image to the grid formed by the internal tomographic image of the computer based on the white light image through positioning points, so as to obtain the surface fluorescence intensity distribution.

[0119] The first determining module 830 is used to determine N time steps based on the grid.

[0120] The second determining module 840 is used to determine the surface fluorescence intensity distribution as the input data for the Nth time step.

[0121] The fourth module 850 is used to input the input data of the i-th time step into the trained 3D reconstruction model to obtain the fluorescence intensity distribution of the (i-1)-th time step. i is an integer greater than 1 and less than or equal to N.

[0122] The third determining module 860 is used to determine the fluorescence intensity distribution of the first time step as the internal fluorescence light source distribution of the target object.

[0123] According to embodiments of this application, any multiple modules among the acquisition module 810, the third obtaining module 820, the first determining module 830, the second determining module 840, the fourth obtaining module 850, and the third determining module 860 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the acquisition module 810, the third obtaining module 820, the first determining module 830, the second determining module 840, the fourth obtaining module 850, and the third determining module 860 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the acquisition module 810, the third obtaining module 820, the first determining module 830, the second determining module 840, the fourth obtaining module 850, and the third determining module 860 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0124] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a three-dimensional reconstruction method for a fluorescent light source according to an embodiment of this application.

[0125] like Figure 9 As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0126] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0127] According to embodiments of this application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0128] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0129] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0130] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the three-dimensional reconstruction method for a fluorescent light source provided in the embodiments of this application.

[0131] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0132] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0133] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0134] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0136] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0137] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A model training method, characterized in that, The model training method includes: Based on Monte Carlo light forward propagation, the simulated internal fluorescence light source distribution is transmitted forward sequentially according to time steps to obtain the simulated fluorescence intensity distribution for multiple time steps. The initial 3D reconstruction model is trained using the simulated fluorescence intensity distributions at multiple time steps until the loss function reaches a preset condition, resulting in a trained 3D reconstruction model. The input to the 3D reconstruction model is the simulated fluorescence intensity distribution at the current time step, randomly selected from the multiple time steps. The output of the 3D reconstruction model is the predicted value of the fluorescence intensity distribution at the previous time step. The loss function characterizes the difference between the predicted value of the fluorescence intensity distribution at the previous time step and the simulated fluorescence intensity distribution at the previous time step.

2. The model training method according to claim 1, characterized in that, The predicted value of the fluorescence intensity distribution at the previous time step is obtained based on the following operation: The simulated fluorescence intensity distribution of the current time step is predicted by using the neural network unit in the three-dimensional reconstruction model according to the reverse propagation of the time step, so as to obtain the predicted value of the fluorescence intensity distribution of the previous time step.

3. The model training method according to claim 2, characterized in that, The neural network unit in the 3D reconstruction model includes an extraction module and a prediction module; The step of using the neural network units in the three-dimensional reconstruction model to predict the fluorescence intensity distribution of the previous time step by reverse propagation of the randomly acquired simulated fluorescence intensity distribution at the current time step, and obtaining the predicted value of the fluorescence intensity distribution at the previous time step, includes: Using the extraction module, the fluorescence sequence distribution characteristics of the simulated fluorescence intensity distribution at the current time step are extracted; Using the prediction module, the fluorescence intensity distribution of the previous time step is reconstructed based on the fluorescence sequence distribution characteristics of the current time step, and the predicted value of the fluorescence intensity distribution of the previous time step is obtained.

4. The model training method according to claim 3, characterized in that, The extraction module includes at least one selective state-space model; The step of using the extraction module to extract the fluorescence sequence distribution features of the simulated fluorescence intensity distribution at the current time step includes: Using the at least one selective state-space model, the fluorescence sequence distribution features of the simulated fluorescence intensity distribution at the current time step are extracted.

5. The model training method according to claim 3, characterized in that, The prediction module includes at least one linear layer; The step of using the prediction module to reconstruct the fluorescence intensity distribution of the previous time step based on the fluorescence sequence distribution characteristics of the current time step, and obtaining the predicted value of the fluorescence intensity distribution of the previous time step, includes: Using the at least one linear layer, the fluorescence intensity distribution of the previous time step is reconstructed based on the fluorescence sequence distribution characteristics of the current time step, thereby obtaining the predicted value of the fluorescence intensity distribution of the previous time step.

6. The model training method according to claim 4, characterized in that, The selective state-space model includes a first path, a second path, a fusion unit, and a mapping unit; The step of extracting fluorescence sequence distribution features of the simulated fluorescence intensity distribution at the current time step using the at least one selective state-space model includes: Input the simulated fluorescence intensity distribution at the current time step into the first path to obtain the dynamic gating weight; The simulated fluorescence intensity distribution at the current time step is input into the second path to obtain the sequence features to be modulated; The dynamic gating weights and the sequence features to be modulated are input into the fusion unit and multiplied to obtain the gated weighted modulated sequence features; The gated weighted modulation sequence features are input into the mapping unit for linear mapping to obtain the fluorescence sequence distribution features of the simulated fluorescence intensity distribution at the current time step.

7. The model training method according to claim 1, characterized in that, The Monte Carlo-based forward propagation transmits the simulated internal fluorescence light source distribution forward sequentially according to time steps, obtaining the simulated fluorescence intensity distribution at each time step, including: For photons in the simulated fluorescence intensity distribution at the current time step, The step size of the photon is determined based on the total attenuation coefficient of the photon in the medium and the first Monte Carlo random number; The position of the photon in the next time step is determined based on the unit transmission direction of the photon in the current time step, the position of the photon in the current time step, and the step length of the photon. Based on the energy, total attenuation coefficient, and absorption coefficient of the photon at the current time step, determine the energy of the photon at the next time step; Based on the position of the photon in the next time step corresponding to each photon in the simulated fluorescence intensity distribution of the current time step, and the energy of the photon in the next time step, the simulated fluorescence intensity distribution of the next time step is obtained, thereby obtaining the simulated fluorescence intensity distribution of each time step.

8. The model training method according to claim 1, characterized in that, The model training method also includes: The simulated internal fluorescence light source distribution and the simulated fluorescence intensity distribution at each time step are processed for data homogenization.

9. A method for three-dimensional reconstruction of a fluorescent light source, characterized in that, The three-dimensional reconstruction method includes: Acquire white light images, fluorescence images, and internal computed tomographic images of the target object; The fluorescence image is registered to the grid formed by the computer's internal tomographic image using positioning points based on the white light image to obtain the surface fluorescence intensity distribution; N time steps are determined based on the grid; The surface fluorescence intensity distribution is determined as the input data for the Nth time step; For the input data at the i-th time step, the input data at the i-th time step is input into the trained three-dimensional reconstruction model as described in any one of claims 1-8 to obtain the fluorescence intensity distribution at the (i-1)-th time step; where i is an integer greater than 1 and less than or equal to N; The fluorescence intensity distribution at the first time step is determined as the internal fluorescence light source distribution of the target object.

10. The three-dimensional reconstruction method according to claim 9, characterized in that, The fluorescence image was obtained using the near-infrared II region as the fluorescence light source.

Images