A method and device for three-dimensional reconstruction of tumor fluorescence images

Abstract

This disclosure provides a method and apparatus for three-dimensional reconstruction of tumor fluorescence images, which can be applied to the field of biomedical molecular imaging technology. The method includes: constructing a finite element tetrahedral discrete mesh based on a medical tomographic image of the target organism; mapping the two-dimensional fluorescence image of the target organism onto the surface of the finite element tetrahedral discrete mesh through a registration operation to obtain a surface fluorescence intensity vector; using a trained multi-task reconstruction network to extract energy features and morphological features from the surface fluorescence intensity vector to obtain intermediate energy intensity features and intermediate morphological classification features; fusing the intermediate energy intensity features and intermediate morphological classification features to obtain a position information vector; and using the position information vector to perform three-dimensional rendering of the tumor of the target organism within the finite element tetrahedral discrete mesh to obtain the three-dimensional morphological information of the tumor within the target organism.

Description

Technical Field

[0001] This disclosure relates to the field of biomedical molecular imaging technology, and specifically to a method and apparatus for three-dimensional reconstruction of tumor fluorescence images. Background Technology

[0002] Malignant tumors often exhibit invasive growth, making it difficult to distinguish and define tumor boundaries, which is a significant factor affecting the success rate of tumor surgery. Near-infrared fluorescence imaging technology (including two windows: near-infrared I (700-900nm) and near-infrared II (1000-1700nm)) is a novel method for lesion localization with high potential for clinical translation.

[0003] Traditional tumor localization methods rely on simplified approximations of the radiative transfer equation to describe the propagation of photons within the body, thereby solving the inverse photon transport problem. However, these methods introduce systematic errors: the simplified approximation leads to systematic shifts in the reconstructed light source location at tissue boundaries, in high-scattering regions, or in low-absorption areas; simultaneously, estimation biases of tissue optical parameters are amplified and solidified into localization errors through the linearization of the inverse problem, and these errors do not diminish with increasing measurement counts; furthermore, the high condition number of the inverse problem under the diffusion approximation framework results in high ill-conditionedness, leading to a large search space during the solution selection process, meaning that even with optimization to minimize the error, an accurate solution may not be found; while regularization can reduce ill-conditionedness to some extent, it still cannot effectively address these issues. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a method and apparatus for three-dimensional reconstruction of tumor fluorescence images.

[0005] According to the first aspect of this disclosure, a method for three-dimensional reconstruction of tumor fluorescence images is provided, comprising: constructing a finite element tetrahedral discrete mesh based on a medical tomographic image of a target organism, and mapping a two-dimensional fluorescence image of the target organism onto the surface of the finite element tetrahedral discrete mesh through a registration operation to obtain a surface fluorescence intensity vector; using a trained multi-task reconstruction network to extract energy features and morphological features from the surface fluorescence intensity vector to obtain intermediate energy intensity features and intermediate morphological classification features, and fusing the intermediate energy intensity features and intermediate morphological classification features to obtain a position information vector; and using the position information vector to perform three-dimensional rendering of the tumor of the target organism in the finite element tetrahedral discrete mesh to obtain the three-dimensional morphological information of the tumor in the target organism.

[0006] According to embodiments of this disclosure, the above-described method of mapping a two-dimensional fluorescence image of a target organism onto the surface of a finite element tetrahedral discrete mesh through a registration operation to obtain a surface fluorescence intensity vector includes: iteratively optimizing the parameters of the optical system used to acquire the two-dimensional fluorescence image by minimizing the error between the coordinates of the marker points in the two-dimensional fluorescence image and the coordinates of the marker points in the medical tomographic image, to obtain an optimized optical system; using the optimized optical system to project and correct the position of the two-dimensional fluorescence image relative to the finite element tetrahedral discrete mesh, to obtain a corrected two-dimensional fluorescence image; and mapping the corrected two-dimensional fluorescence image onto the surface of the finite element tetrahedral discrete mesh by calculating the surface vertex distribution of the corrected two-dimensional fluorescence image in the finite element tetrahedral discrete mesh, to obtain a surface fluorescence intensity vector.

[0007] According to embodiments of this disclosure, the above-described method of iteratively optimizing the parameters of an optical system for acquiring a two-dimensional fluorescence image by minimizing the error between the coordinates of marker points in a two-dimensional fluorescence image and the coordinates of marker points in a medical tomographic image to obtain an optimized optical system includes: using the optical system to convert the coordinates of marker points in the two-dimensional fluorescence image and the coordinates of marker points in the medical tomographic image from pixel coordinates to actual physical dimensions, obtaining the physical coordinates of the fluorescent marker points and the physical coordinates of the marker points in the medical tomographic image; performing multiple rounds of three-dimensional coordinate adjustments on all possible lens centers in the optical system to minimize the error between the predicted coordinates of the fluorescent marker points predicted for each lens center and the physical coordinates of the marker points in the medical tomographic image, and summing the obtained errors to obtain an error function; using the error function to iteratively optimize the parameters of the optical system until a preset condition is met, thereby obtaining the optimized optical system.

[0008] According to embodiments of this disclosure, the above-described projection correction of the position of a two-dimensional fluorescence image relative to a finite element tetrahedral discrete mesh using an optimized optical system to obtain a corrected two-dimensional fluorescence image includes: determining the viewing angle direction of the two-dimensional fluorescence image and the spatial coordinates of the detector used to generate the two-dimensional fluorescence image using the optimized optical system; calculating the normal vector, area, and center coordinates of each surface triangular facet in the finite element tetrahedral discrete mesh based on the viewing angle direction and spatial coordinates, and determining the visible portion of the finite element tetrahedral discrete mesh in the two-dimensional fluorescence image based on the normal vector, area, and center coordinates; and correcting the two-dimensional fluorescence image by projecting the visible portion onto the plane where the detector is located to obtain the corrected two-dimensional fluorescence image.

[0009] According to embodiments of this disclosure, the above-mentioned method of mapping the corrected two-dimensional fluorescence image onto the surface of a finite element tetrahedral discrete mesh by calculating the distribution of surface vertices in the corrected two-dimensional fluorescence image includes: obtaining the power value of the surface triangular facets using the gray values ​​determined by the corrected two-dimensional fluorescence image and the optimized optical system; traversing all surface triangular facets based on the three-dimensional coordinates of the vertices in the finite element tetrahedral discrete mesh and the normal vector, area, center coordinates, and three-dimensional coordinates of the surface triangular facets; accumulating the power and area of ​​the target surface triangular facets when the traversal result shows that the target surface triangular facet has vertices, obtaining the accumulated power and total area; calculating the accumulated power and total area to obtain the average power of the vertices; and then mapping the corrected two-dimensional fluorescence image onto the surface of the finite element tetrahedral discrete mesh to obtain the surface fluorescence intensity vector.

[0010] According to embodiments of this disclosure, the above-mentioned extraction of energy features and morphological features from the surface fluorescence intensity vector using a trained multi-task reconstruction network to obtain intermediate energy intensity features and intermediate morphological classification features includes: extracting energy features from the surface fluorescence intensity vector using a first branch module of the trained multi-task reconstruction network to obtain intermediate energy intensity features, wherein the first branch module includes a first expert network; and extracting morphological binary classification features from the surface fluorescence intensity vector using a second branch module of the trained multi-task reconstruction network to obtain intermediate morphological classification features, wherein the second branch module includes a second expert network.

[0011] According to embodiments of this disclosure, the above-described fusion processing of energy intensity intermediate features and morphology classification intermediate features to obtain a location information vector includes: using the shared layer of a trained multi-task reconstruction network to perform feature sharing processing on the energy intensity intermediate features and morphology classification intermediate features to obtain a first shared feature and a second shared feature; using the gating unit of the trained multi-task reconstruction network to perform weighted processing on the first shared feature and the second shared feature respectively to obtain a first weighted feature and a second weighted feature; using the first tower structure network of the trained multi-task reconstruction network to perform hierarchical integration on the first weighted feature to obtain an energy intensity integrated feature, and using the second tower structure network of the trained multi-task reconstruction network to perform hierarchical integration on the second weighted feature to obtain a morphology binary classification integrated feature; and using the trained multi-task reconstruction network to perform a dot product operation on the energy intensity integrated feature and the morphology binary classification integrated feature to obtain a location information vector.

[0012] According to embodiments of this disclosure, the trained multi-task reconstruction network is obtained through the following operations: constructing a tumor fluorescence image simulation dataset and constructing a multi-task reconstruction network, wherein the multi-task reconstruction network includes multiple branch modules, multiple shared layers, gating units, and multiple tower structure networks; using the tumor fluorescence image simulation dataset and a preset loss function, iteratively optimizing the parameters of the multi-task reconstruction network based on gradient balance until the preset training conditions are met, thereby obtaining the trained multi-task reconstruction network, wherein the preset loss function includes a focus loss function, a binary classification smooth similarity coefficient loss function, and a mean squared error loss function.

[0013] According to embodiments of this disclosure, the above-mentioned construction of a tumor fluorescence image simulation dataset includes: constructing a standard anatomical structure grid using segmented medical tomographic image samples of the experimental organism; calculating the absorption coefficient and scattering coefficient of the experimental organism in the near-infrared II fluorescence spectrum, and generating a simulated single-source dataset of the experimental organism using the absorption coefficient and scattering coefficient; combining or aggregating the simulated single-source datasets to obtain simulated multi-source datasets and simulated large-source datasets, respectively, and constructing a simulated light source dataset based on the simulated single-source dataset, simulated multi-source dataset, and simulated large-source dataset; using the Monte Carlo algorithm to calculate the surface spot vectors of different light source vectors inside the grid in the simulated light source dataset on the standard anatomical structure grid, wherein the surface spot vectors are used to simulate the mapping result of the tumor fluorescence image of the experimental organism on the standard anatomical structure grid; and using the simulated light source dataset with grid-internal light source vector-surface spot vector pairs as the tumor fluorescence image simulation dataset of the experimental organism.

[0014] According to a second aspect of this disclosure, a three-dimensional reconstruction device for tumor fluorescence images is provided, comprising: a construction and registration module for constructing a finite element tetrahedral discrete mesh based on a medical tomographic image of a target organism, and mapping a two-dimensional fluorescence image of the target organism onto the surface of the finite element tetrahedral discrete mesh through a registration operation to obtain a surface fluorescence intensity vector; a feature extraction and fusion module for extracting energy features and morphological features from the surface fluorescence intensity vector using a trained multi-task reconstruction network to obtain intermediate energy intensity features and intermediate morphological classification features, and fusing the intermediate energy intensity features and intermediate morphological classification features to obtain a position information vector; and a three-dimensional reconstruction module for rendering the tumor of the target organism in three dimensions using the position information vector in the finite element tetrahedral discrete mesh to obtain the three-dimensional morphological information of the tumor in the target organism.

[0015] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0016] A fourth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0017] The fifth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0018] The tumor fluorescence image 3D reconstruction method disclosed herein effectively avoids the systematic errors caused by traditional tumor localization methods through finite element tetrahedral mesh modeling and multi-task deep learning fusion architecture. This improves the accuracy, stability, and operability of the 3D reconstruction process based on 2D tumor fluorescence images and enhances the performance of optical tomography. Furthermore, the method disclosed herein, through multi-modal fusion of fluorescence intensity vectors and morphological features, effectively avoids noise or artifacts that are actively filtered out due to mismatches in energy and morphological spatial features. This solves the problem of light source morphology deviation caused by existing deep learning-based tumor localization methods that output energy intensity vectors with fixed thresholds. In addition, the localized light source region in this disclosure is not limited to the tumor region, which is beneficial for studying the location of fluorescent probes in vivo and has significant implications for the study of tumor heterogeneity, promoting the application of deep learning in the field of biomedical imaging. Attached Figure Description

[0019] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 An application scenario diagram of the three-dimensional reconstruction method of tumor fluorescence images according to embodiments of the present disclosure is shown.

[0021] Figure 2 A flowchart of a method for three-dimensional reconstruction of tumor fluorescence images according to an embodiment of the present disclosure is shown.

[0022] Figure 3 A structural diagram of a multi-task reconstruction network according to an embodiment of the present disclosure is shown.

[0023] Figure 4 A structural block diagram of a three-dimensional reconstruction apparatus for tumor fluorescence images according to an embodiment of the present disclosure is shown.

[0024] Figure 5 A block diagram of an electronic device suitable for implementing a three-dimensional reconstruction method of tumor fluorescence images according to an embodiment of the present disclosure is shown. Detailed Implementation

[0025] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0029] Since traditional tumor localization methods can introduce systematic errors, this disclosure provides a three-dimensional reconstruction method for tumor fluorescence images. It utilizes a deep learning method based on a multi-task reconstruction network to achieve three-dimensional reconstruction of two-dimensional fluorescent molecules, thereby accurately locating the three-dimensional position and distribution of tumors within the body and expanding the application of fluorescence imaging in preclinical and clinical fields such as integrated tumor diagnosis and treatment and pharmacokinetics.

[0030] It should be noted that the fluorescence images, MRI (Magnetic Resonance Imaging) images, CT (Computed Tomography) images, other medical images or data, and training or testing datasets involved in this disclosure are all derived from experimental mice. Those skilled in the art should understand that the technical solutions of this disclosure can also be applied to the target users.

[0031] When the technical solution disclosed herein is applied to a target user, the acquisition of the target user's medical images, data, or information is authorized by the target user in advance, and the aforementioned medical images, data, or information are processed with the permission of the target user. The relevant process strictly complies with the provisions of laws and regulations, takes strict confidentiality measures, does not violate public order and good morals, and provides corresponding operation access points for the target user to authorize or refuse.

[0032] Figure 1 An application scenario diagram of the three-dimensional reconstruction method of tumor fluorescence images according to embodiments of the present disclosure is shown.

[0033] like Figure 1 As shown, application scenario 100 according to this embodiment may include application scenarios such as biomedical molecular imaging. Network 104 is used as a medium to provide a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0034] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0035] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0036] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0037] It should be noted that the tumor fluorescence image three-dimensional reconstruction method provided in this disclosure embodiment can generally be executed by server 105. Correspondingly, the tumor fluorescence image three-dimensional reconstruction device provided in this disclosure embodiment can generally be located in server 105. The tumor fluorescence image three-dimensional reconstruction method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the tumor fluorescence image three-dimensional reconstruction device provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0038] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system is only a limited number. Depending on implementation needs, there can be any number of terminal devices, networks, and servers.

[0039] The following will be based on Figure 1 The described scene, through Figures 2-3 The method for three-dimensional reconstruction of tumor fluorescence images according to the disclosed embodiments is described in detail.

[0040] Figure 2 A flowchart of a method for three-dimensional reconstruction of tumor fluorescence images according to an embodiment of the present disclosure is shown.

[0041] like Figure 2 As shown, the three-dimensional reconstruction of tumor fluorescence images in this embodiment includes operations S210 to S230.

[0042] In operation S210, a finite element tetrahedral discrete mesh is constructed based on the medical tomographic image of the target organism, and the two-dimensional fluorescence image of the target organism is mapped onto the surface of the finite element tetrahedral discrete mesh through a registration operation to obtain the surface fluorescence intensity vector.

[0043] The aforementioned medical computed tomographic images may include MRI images or CT images. In the embodiments of this disclosure, those skilled in the art can choose MRI images or CT images to implement the technical solutions of this disclosure.

[0044] The target organism mentioned above is generally a laboratory mouse; when the target organism is a target user, the user's consent or authorization is required before obtaining the target user's MRI, CT fluorescence images, or other information. For example, before operating S210, a request to obtain the user's MRI, CT fluorescence images, or other information can be sent to the user. If the user consents or authorizes the acquisition of the user's MRI, CT fluorescence images, or other information, operation S210 is performed.

[0045] Before constructing the finite element tetrahedral discrete mesh, it is necessary to obtain MRI, CT, and fluorescence images of the target organism. Taking a live mouse as an example: A live mouse orthotopic tumor transplantation animal model is constructed, human glioma cells are cultured, and the cultured cells are injected into the mouse brain; the mice can be 4-6 week old BALB / c female mice; the human glioma cells can be U87MG-GFTFLUC cells labeled with green fluorescent protein (GFP); a fluorescent probe is injected into the mouse via the tail vein. The fluorescent probe can be… Under continuous light excitation at a wavelength of 792 nm, fluorescence images of live mouse tumors were acquired using an EMCCD (Electron Multiplying Charge Coupled Device) fluorescence camera; MRI images, CT images, and frozen sections of mice were acquired and stained with hematoxylin-eosin (HE).

[0046] In operating S220, the trained multi-task reconstruction network is used to extract energy features and morphological features from the surface fluorescence intensity vector to obtain intermediate energy intensity features and intermediate morphological classification features. The intermediate energy intensity features and intermediate morphological classification features are then fused to obtain the position information vector.

[0047] The multi-task reconstruction network trained as described above has multiple branches, enabling it to perform different tasks. The aforementioned location information vector includes both location information and the energy intensity vector of the light source portion.

[0048] In operation S230, the tumor of the target organism is rendered in three dimensions in a finite element tetrahedral discrete mesh using the position information vector, and the three-dimensional morphological information of the tumor in the target organism is obtained.

[0049] When the technical solution of this disclosure is applied to a target user, executing the above-described operations S210 to S230 can provide the user with a corresponding operation entry point, allowing the user to choose to agree to or reject the automated decision-making result. That is, before processing the user information, the user can be provided with an instruction to agree to or reject the processing / decision through the corresponding operation entry point. If the user agrees to the processing / decision, the user information is processed, i.e., the above-described operations S210 to S230 are executed. If the user rejects the processing / decision, the expert decision-making process is initiated.

[0050] It should be noted that when the target organism is a human, the purpose of the above operations S210 to S230 is to provide more accurate intermediate information for the diagnosis of the lesion area, rather than to directly obtain the diagnostic information or health status of the target organism; at the same time, the above operations S210 to S230 and other operations in the embodiments of this disclosure are all information processing operations performed by a computer or other device.

[0051] The tumor fluorescence image 3D reconstruction method disclosed herein effectively avoids the systematic errors caused by traditional tumor localization methods through finite element tetrahedral mesh modeling and multi-task deep learning fusion architecture. This improves the accuracy, stability, and operability of the 3D reconstruction process based on 2D tumor fluorescence images and enhances the performance of optical tomography. Furthermore, the method disclosed herein, through multi-modal fusion of fluorescence intensity vectors and morphological features, effectively avoids noise or artifacts that are actively filtered out due to mismatches in energy and morphological spatial features. This solves the problem of light source morphology deviation caused by existing deep learning-based tumor localization methods that output energy intensity vectors with fixed thresholds. In addition, the localized light source region in this disclosure is not limited to the tumor region, which is beneficial for studying the location of fluorescent probes in vivo and has great significance for the study of tumor heterogeneity, promoting the application of deep learning in the field of biomedical imaging.

[0052] According to embodiments of this disclosure, the above-described method of mapping a two-dimensional fluorescence image of a target organism onto the surface of a finite element tetrahedral discrete mesh through a registration operation to obtain a surface fluorescence intensity vector includes: iteratively optimizing the parameters of the optical system used to acquire the two-dimensional fluorescence image by minimizing the error between the coordinates of the marker points in the two-dimensional fluorescence image and the coordinates of the marker points in the medical tomographic image, to obtain an optimized optical system; using the optimized optical system to project and correct the position of the two-dimensional fluorescence image relative to the finite element tetrahedral discrete mesh, to obtain a corrected two-dimensional fluorescence image; and mapping the corrected two-dimensional fluorescence image onto the surface of the finite element tetrahedral discrete mesh by calculating the surface vertex distribution of the corrected two-dimensional fluorescence image in the finite element tetrahedral discrete mesh, to obtain a surface fluorescence intensity vector.

[0053] The following uses CT images as an example to further illustrate the registration and mapping operation of two-dimensional fluorescence images involved in the above embodiments of this disclosure through specific implementation methods. Those skilled in the art should understand that the following specific implementation methods are also applicable to other medical tomographic images, such as MRI images.

[0054] First, the lens center position (horizontal axis, vertical axis, depth axis, etc.) of the optical system is adjusted through an iterative process to minimize the error between the marker points in the 2D fluorescence image and the marker points in the CT image. Second, the 2D fluorescence image and the finite element tetrahedral discrete mesh based on the CT image are analyzed and compared to determine the visible part of the finite element tetrahedral discrete mesh in the 2D fluorescence image. The information of the finite element tetrahedral discrete mesh is then projected onto the 2D fluorescence image for correction. Finally, the distribution of fluorescence intensity of the 2D fluorescence image acquired by the optical system on the surface vertices of the finite element tetrahedral discrete mesh is established and visualized accordingly.

[0055] The embodiments disclosed above achieve iterative optimization of optical imaging system parameters by minimizing marker point coordinate errors, effectively compensating for system distortion and geometric distortion, and improving image registration accuracy; they utilize the optimized optical system to perform projection correction on two-dimensional fluorescence images, eliminating imaging viewpoint deviations and ensuring spatial consistency between the image and the three-dimensional mesh; by calculating the vertex distribution of the corrected image on the finite element mesh surface, they achieve accurate mapping from pixel level to vertex level, avoiding blurring and distortion introduced by traditional interpolation methods; by combining CT or MRI anatomical information with fluorescence functional information, they achieve high-precision fusion of anatomical structure and functional localization, providing a more reliable basis for tumor localization; through parameter optimization and correction processes, they systematically eliminate imaging system errors, registration errors, and mapping errors, significantly improving the accuracy and reliability of three-dimensional reconstruction.

[0056] According to embodiments of this disclosure, the above-described method of iteratively optimizing the parameters of an optical system for acquiring a two-dimensional fluorescence image by minimizing the error between the coordinates of marker points in a two-dimensional fluorescence image and the coordinates of marker points in a medical tomographic image to obtain an optimized optical system includes: using the optical system to convert the coordinates of marker points in the two-dimensional fluorescence image and the coordinates of marker points in the medical tomographic image from pixel coordinates to actual physical dimensions, obtaining the physical coordinates of the fluorescent marker points and the physical coordinates of the marker points in the medical tomographic image; performing multiple rounds of three-dimensional coordinate adjustments on all possible lens centers in the optical system to minimize the error between the predicted coordinates of the fluorescent marker points predicted for each lens center and the physical coordinates of the marker points in the medical tomographic image, and summing the obtained errors to obtain an error function; using the error function to iteratively optimize the parameters of the optical system until a preset condition is met, thereby obtaining the optimized optical system.

[0057] The optical optimization operation of the above embodiments of this disclosure will be further described in detail below using CT images as an example through specific implementation methods. Those skilled in the art should understand that the following specific implementation methods are also applicable to other medical tomographic images, such as MRI images.

[0058] Initialization and Parameter Setting: Input basic camera parameters such as detector size and pixel size, as well as the coordinates of optical image markers and CT image markers. Data Mapping: Convert CT image markers from pixel coordinates to actual physical dimensions. Iterative Search: Through nested triple loops, make small adjustments to the 3D coordinates of the lens center position to minimize the error between the coordinates of the optical image markers predicted from this lens center and the coordinates of the markers in the actual CT image. Error Calculation: For each possible optical center position, calculate the position of each marker in the image and compare it with the actual marker in the optical image. Sum the errors of all points to obtain the error function. Search for Optical Center Position: Select the optical center position with the smallest error function and proceed to the next iteration. If the error is small enough, end the search and obtain the optimal solution for the current optical center position.

[0059] The above embodiments of this disclosure minimize errors based on actual physical coordinates, thereby improving registration accuracy; avoid local optima through multiple rounds of adjustment, thereby ensuring global convergence; utilize error functions to drive iterative optimization, thereby achieving automated parameter calibration; significantly reduce geometric distortion introduced by the optical system, thereby improving the spatial consistency between two-dimensional fluorescence images and medical tomographic images.

[0060] According to embodiments of this disclosure, the above-described projection correction of the position of a two-dimensional fluorescence image relative to a finite element tetrahedral discrete mesh using an optimized optical system to obtain a corrected two-dimensional fluorescence image includes: determining the viewing angle direction of the two-dimensional fluorescence image and the spatial coordinates of the detector used to generate the two-dimensional fluorescence image using the optimized optical system; calculating the normal vector, area, and center coordinates of each surface triangular facet in the finite element tetrahedral discrete mesh based on the viewing angle direction and spatial coordinates, and determining the visible portion of the finite element tetrahedral discrete mesh in the two-dimensional fluorescence image based on the normal vector, area, and center coordinates; and correcting the two-dimensional fluorescence image by projecting the visible portion onto the plane where the detector is located to obtain the corrected two-dimensional fluorescence image.

[0061] The following describes in more detail the process of obtaining the corrected two-dimensional fluorescence image according to the above embodiments of this disclosure, using CT images as an example. Those skilled in the art should understand that the following specific embodiments are also applicable to other medical tomographic images, such as MRI images.

[0062] Initialization and Parameter Setting: Input basic parameters such as image size and pixel size; input the center position and orientation of the optimized optical system lens to determine the input angle of the 2D fluorescence image and the detector position; input the data of the finite element tetrahedral discrete mesh. Surface Unit Calculation: Calculate the normal vector, area, and center coordinates of each triangular facet of the finite element tetrahedral discrete mesh, determine whether the facet is within the field of view of the optimized optical system, and calculate the pixel position of its projection onto the detector to acquire the optical image. Save Calculation Results and Visualize: Save the calculated facet data and display the comparison between the grayscale image acquired by the optimized optical system and the projection result of the finite element tetrahedral discrete mesh.

[0063] The embodiments disclosed above accurately determine the imaging angle and detector coordinates based on optimized parameters, thereby improving geometric accuracy; accurately identify the visible part of the mesh through triangular facet normal vectors and area calculations; use projection mapping technology to achieve image correction, effectively eliminating imaging distortion; and ensure that the fluorescence image is aligned with the finite element mesh space, thereby improving the accuracy of three-dimensional reconstruction.

[0064] According to embodiments of this disclosure, the above-mentioned method of mapping the corrected two-dimensional fluorescence image onto the surface of a finite element tetrahedral discrete mesh by calculating the distribution of surface vertices in the corrected two-dimensional fluorescence image includes: obtaining the power value of the surface triangular facets using the gray values ​​determined by the corrected two-dimensional fluorescence image and the optimized optical system; traversing all surface triangular facets based on the three-dimensional coordinates of the vertices in the finite element tetrahedral discrete mesh and the normal vector, area, center coordinates, and three-dimensional coordinates of the surface triangular facets; accumulating the power and area of ​​the target surface triangular facets when the traversal result shows that the target surface triangular facet has vertices, obtaining the accumulated power and total area; calculating the accumulated power and total area to obtain the average power of the vertices; and then mapping the corrected two-dimensional fluorescence image onto the surface of the finite element tetrahedral discrete mesh to obtain the surface fluorescence intensity vector.

[0065] The following detailed explanation, using CT images as an example, illustrates the process of obtaining surface fluorescence intensity vectors in the embodiments described above. Those skilled in the art should understand that the specific embodiments described below are also applicable to other medical tomographic images, such as MRI images.

[0066] Initialization and Parameter Setting: Input the vertex and surface patch coordinates of the finite element tetrahedral discrete mesh; input the normal vector, area, and center coordinates of the surface patches; input the grayscale and fluorescence images acquired by the optimized optical system. Vertex Intensity Assignment: For each vertex, traverse all surface patches. If a patch contains the vertex, accumulate the power and area of ​​that patch. Calculate the average power of each vertex by accumulating the power and dividing by the total area. Save Results and Visualize: Save the calculated vertex intensity vectors and generate a data file that can be used in visualization software. Input the mapped surface fluorescence intensity vectors into the trained multi-task reconstruction network to obtain the three-dimensional morphology and location information vectors of the tumor within the target organism (e.g., a mouse), and then visualize the results.

[0067] The embodiments of this disclosure calculate the power of triangular patches based on the grayscale values ​​of the corrected image, ensuring data accuracy; perform precise traversal statistics using the three-dimensional coordinates of the vertices and the geometric properties of the patches; and effectively eliminate local noise interference by calculating the average power of the vertices through power and face accumulation. This achieves high-fidelity mapping from a two-dimensional image to a three-dimensional mesh, significantly improving tumor localization accuracy. Furthermore, the embodiments of this disclosure calculate the power of triangular patches by correcting the grayscale values ​​of the image, perform precise statistics using the vertex coordinates and patch geometric properties, and calculate the average power of the vertices through power and face accumulation, achieving high-precision mapping from a two-dimensional fluorescence image to a three-dimensional finite element mesh, effectively improving the accuracy and reliability of three-dimensional tumor reconstruction.

[0068] According to embodiments of this disclosure, the above-mentioned extraction of energy features and morphological features from the surface fluorescence intensity vector using a trained multi-task reconstruction network to obtain intermediate energy intensity features and intermediate morphological classification features includes: extracting energy features from the surface fluorescence intensity vector using a first branch module of the trained multi-task reconstruction network to obtain intermediate energy intensity features, wherein the first branch module includes a first expert network; and extracting morphological binary classification features from the surface fluorescence intensity vector using a second branch module of the trained multi-task reconstruction network to obtain intermediate morphological classification features, wherein the second branch module includes a second expert network.

[0069] In the embodiments of this disclosure, the first expert network in the first branch module is dedicated to processing energy features to ensure the specificity of spectral information extraction; the second expert network in the second branch module focuses on morphological classification features to improve the accuracy of structural information recognition; the parallel processing of the two branches avoids feature interference and improves the independence and accuracy of feature extraction; the network structure supports end-to-end training to achieve synergistic optimization of energy and morphological features.

[0070] According to embodiments of this disclosure, the above-described fusion processing of energy intensity intermediate features and morphology classification intermediate features to obtain a location information vector includes: using the shared layer of a trained multi-task reconstruction network to perform feature sharing processing on the energy intensity intermediate features and morphology classification intermediate features to obtain a first shared feature and a second shared feature; using the gating unit of the trained multi-task reconstruction network to perform weighted processing on the first shared feature and the second shared feature respectively to obtain a first weighted feature and a second weighted feature; using the first tower structure network of the trained multi-task reconstruction network to perform hierarchical integration on the first weighted feature to obtain an energy intensity integrated feature, and using the second tower structure network of the trained multi-task reconstruction network to perform hierarchical integration on the second weighted feature to obtain a morphology binary classification integrated feature; and using the trained multi-task reconstruction network to perform a dot product operation on the energy intensity integrated feature and the morphology binary classification integrated feature to obtain a location information vector.

[0071] The shared layer in the above embodiments of this disclosure achieves deep fusion of energy and morphological features, enhancing feature representation capabilities; the gating unit adaptively weights the processing, dynamically adjusting feature importance; the dual-tower structure processes different feature streams separately, maintaining information integrity; dot product operations generate positional information vectors, enabling strong correlation modeling between features. The overall architecture of the above embodiments of this disclosure supports end-to-end training, improving tumor localization accuracy and robustness.

[0072] The following describes specific implementation methods in conjunction with appendices. Figure 3 The architecture and functions of the multi-task reconstruction network involved in the above embodiments of this disclosure will be described in further detail.

[0073] Figure 3 A structural diagram of a multi-task reconstruction network according to an embodiment of the present disclosure is shown.

[0074] like Figure 3 As shown, the multi-task reconstruction network involved in the embodiments of this disclosure includes an input model ( Figure 3 The input shown), expert unit ( Figure 3 The Expert A1, Expert A2, Expert B1, and Expert B2 shown, and the shared layer ( Figure 3 The diagram shows Share 1, Share 2, and the tower structure. Figure 3 Tower A and Tower B shown), gating mechanism ( Figure 3 As shown in G1~G5), output modules ( Figure 3 The output is shown.

[0075] The input module receives raw input data, typically the surface photon density vector in a two-dimensional projected view.

[0076] The expert unit comprises multiple expert sub-units: Expert A1 and Expert A2 are specifically responsible for feature extraction from the input data using energy intensity regression. They extract energy features from the input data and generate intermediate feature representations. Expert B1 and Expert B2 are specifically responsible for feature extraction from the input data using morphological binary classification. They extract morphological features from the input data and generate different intermediate feature representations.

[0077] The network includes two shared layers: shared layer 1 and shared layer 2, which are used to fuse and share intermediate feature representations from different expert units and features from the input module. The design of the shared layers enables the outputs of multiple expert units to interact effectively, thereby enhancing the network's representational capabilities.

[0078] The gating mechanism involves passing the feature information output from the shared layer through multiple gating units. Each gating unit performs a weighted operation on the feature stream to determine the relevance and importance of different features and dynamically adjusts the influence of each expert unit. The use of gating mechanisms effectively reduces feature redundancy and improves network performance and robustness.

[0079] The tower structure, after feature information is adjusted by the gating mechanism unit, is input to towers A and B for further processing. The tower structure is responsible for integrating features from multiple levels, ultimately generating outputs for two sub-tasks: energy intensity regression and morphological binary classification.

[0080] In the final output layer, the outputs of tower A and tower B are integrated into the final network output through dot product operation.

[0081] According to embodiments of this disclosure, the trained multi-task reconstruction network is obtained through the following operations: constructing a tumor fluorescence image simulation dataset and constructing a multi-task reconstruction network, wherein the multi-task reconstruction network includes multiple branch modules, multiple shared layers, gating units, and multiple tower structure networks; using the tumor fluorescence image simulation dataset and a preset loss function, iteratively optimizing the parameters of the multi-task reconstruction network based on gradient balance until the preset training conditions are met, thereby obtaining the trained multi-task reconstruction network, wherein the preset loss function includes a focus loss function, a binary classification smooth similarity coefficient loss function, and a mean squared error loss function.

[0082] The GradNorm (gradient normalization) method is used to train the multi-task reconstruction network. GradNorm adjusts the training speed of each task by calculating and balancing the gradient norm of each task. The purpose of GradNorm is to learn different weights for each task, making the training progress of each task relatively balanced, thereby improving the training imbalance problem in multi-task learning. The embodiments of this disclosure train the multi-task reconstruction network using the following operations:

[0083] (1) Calculate the gradient of the loss function with respect to the shared parameters for each task, denoted as . .

[0084] (2) Calculate the gradient norm for each task. .

[0085] (3) Adjust the training rate using the ratio of task loss to initial loss, as shown in formula (1):

[0086] (1).

[0087] in This is the initial loss value for the task.

[0088] (4) Calculate the average value of the task gradient The ratio of the training rate to the task's training rate is used to generate the GradNorm loss function, as shown in formula (2):

[0089] (2).

[0090] in It is a hyperparameter that controls the strength of gradient balancing.

[0091] (5) The final GradNorm loss is minimized by calculating the difference between the task gradient and the target gradient, and the weight update is shown in Equation (3):

[0092] (3).

[0093] Then the gradient of the weights is backpropagated.

[0094] Loss functions include:

[0095] (1) Focal Loss (an improved cross-entropy loss function):

[0096] Primarily used to address class imbalance, it emphasizes the learning of difficult-to-classify samples by adjusting the loss weights of easily classified samples. The loss function is defined as shown in formula (4):

[0097] (4).

[0098] in, For predicted values, This is a factor that modulates the loss. (Through...) By adjusting the loss, a higher loss weight can be assigned to samples that are difficult to classify.

[0099] (2) Soft Dice Loss (a loss function based on Dice coefficients for image semantic segmentation tasks):

[0100] Soft Dice Loss is a loss function used for binary classification problems. It measures the performance of the model by calculating the overlap between the predicted result and the true label, as shown in Equation (5):

[0101] (5).

[0102] The numerator represents the intersection of the prediction and the true label, the denominator represents the intersection of the prediction and the true label, and "smooth" indicates a smoothing term to prevent the denominator from being 0.

[0103] (3) MSE loss (mean squared error loss function):

[0104] MSE loss is a loss function used for regression tasks. It measures the prediction accuracy of the model by calculating the mean squared error between the predicted and the true values, as shown in formula (6):

[0105] (6).

[0106] The simulation datasets in the embodiments of this disclosure ensure the diversity and availability of training samples; multi-branch modules enable parallel extraction of energy and morphological features; shared layers and gating units improve feature fusion efficiency; multi-tower network structures enhance feature representation capabilities; gradient balancing optimization ensures the stability of multi-task learning; and combined loss functions (focal loss, smoothed similarity coefficient, mean squared error) improve the network's adaptability to different features, achieving high-precision tumor localization. The embodiments of this disclosure, by constructing simulation datasets and multi-branch network architectures, combined with gradient balancing optimization and combined loss functions, achieve efficient fusion of energy and morphological features, improving the accuracy and robustness of three-dimensional tumor reconstruction.

[0107] According to embodiments of this disclosure, the above-mentioned construction of a tumor fluorescence image simulation dataset includes: constructing a standard anatomical structure grid using segmented medical tomographic image samples of the experimental organism; calculating the absorption coefficient and scattering coefficient of the experimental organism in the near-infrared II fluorescence spectrum, and generating a simulated single-source dataset of the experimental organism using the absorption coefficient and scattering coefficient; combining or aggregating the simulated single-source datasets to obtain simulated multi-source datasets and simulated large-source datasets, respectively, and constructing a simulated light source dataset based on the simulated single-source dataset, simulated multi-source dataset, and simulated large-source dataset; using the Monte Carlo algorithm to calculate the surface spot vectors of different light source vectors inside the grid in the simulated light source dataset on the standard anatomical structure grid, wherein the surface spot vectors are used to simulate the mapping result of the tumor fluorescence image of the experimental organism on the standard anatomical structure grid; and using the simulated light source dataset with grid-internal light source vector-surface spot vector pairs as the tumor fluorescence image simulation dataset of the experimental organism.

[0108] The different internal light source vectors under the aforementioned unified standard mesh (e.g., a standard anatomical structure mesh) refer to the 0 / 1 label indicating whether each point belongs to a light source. In the simulation dataset, different light source vectors are different vector labels belonging to the same mesh.

[0109] The above simulation dataset is randomly divided into a simulation training dataset and a simulation test dataset.

[0110] The process of obtaining the simulation dataset in the above embodiments of this disclosure will be further described in detail below through specific implementation methods.

[0111] A standard anatomical structure mesh is constructed based on medical tomographic images (e.g., MRI or CT images) of the target organism. In one embodiment, the medical tomographic images (or medical sequence images) are segmented into different organ regions according to grayscale ranges using 3D visualization and analysis software.

[0112] The absorption and scattering coefficients of photons with excitation light at 800 nm and emission light at 1300 nm in the near-infrared II region are calculated using the formula. The absorption coefficient is calculated as shown in formula (7):

[0113] (7).

[0114] in, The absorption coefficient is... The oxygen-protein uptake coefficient. This refers to the deoxyhemoglobin absorption coefficient. The water absorption coefficient, Hemoglobin content, Water content, This represents the deoxyhemoglobin ratio. The absorption coefficients, calculated based on empirical parameters and experimentally measured values ​​from various organs of the experimental animals, are shown in Table 1.

[0115] Table 1. Empirical parameters and experimentally measured absorption coefficients of various organs in experimental animals.

[0116]

[0117] The scattering coefficient is calculated as shown in formula (8):

[0118] (8).

[0119] in, The scattering coefficient is... The anisotropy coefficient (which can be approximated as a constant) is the anisotropy coefficient. These are empirical parameters. These are empirical parameters. The scattering coefficients calculated based on empirical parameters of various organs of the experimental animals and experimentally measured values ​​are shown in Table 2.

[0120] Table 2. Empirical parameters and experimentally measured scattering coefficients of various organs of experimental animals.

[0121]

[0122] The surface contour and tetrahedral mesh are generated directly from the segmented CT or MRI data. After the initial mesh generation, it needs to be refined to obtain a standard anatomical structure mesh composed of tetrahedral units.

[0123] Single-source mesh data of various types, including spheres, cylinders, and ellipsoids, with different radii and positions, are generated. Forward simulation is performed using the Monte Carlo method to obtain the surface spot vectors corresponding to different single sources. The photon number of the source can be set to 1,000,000, and the spectral energy can be set to 1.0. 80% of the dataset can be selected as the training set, and 20% as the test set.

[0124] Simulation datasets for dual-light sources, three-light sources, and four-light sources are generated by combining simulation datasets from single-light sources. The combination method involves arbitrarily selecting data from different locations within the single-light source dataset. Simulation datasets for large light sources of varying sizes are then aggregated using single-light source datasets. Surface spot vectors corresponding to different light sources in the datasets are calculated using the Monte Carlo method to obtain data pairs between internal light source vectors and surface spot vectors, which are then used to train a neural network model. The training process can be performed using a desktop computer equipped with an Intel Core i7 CPU (3.70 GHz), 16 GB DDR3 RAM, and an Nvidia GTX 1080 Ti GPU.

[0125] The above embodiments of this disclosure construct a standard anatomical structure mesh based on real medical tomographic images to ensure the biological realism of the simulation environment; generate simulated single-source data using near-infrared II spectral parameters to ensure the accuracy of the physical process; construct multi-source and large-source datasets through combined aggregation processing to improve the diversity of training samples; use the Monte Carlo algorithm to accurately calculate the photon propagation path to achieve high-fidelity surface spot simulation; 5) construct a complete light source vector-spot vector pair dataset to provide high-quality training samples for deep learning networks and significantly improve tumor localization accuracy.

[0126] The tumor fluorescence image 3D reconstruction method disclosed herein involves constructing a finite element discrete standard mesh (i.e., a standard anatomical structure mesh) of the target organism (e.g., a mouse), calculating the optical parameters of the biological tissue, constructing multiple light source datasets based on the finite element discrete standard mesh, performing forward process simulation mapping of the light source using the Monte Carlo method, obtaining the photon vectors of the corresponding surface of the light source, forming data pairs, and inputting them into a multi-task reconstruction network for training; constructing a deep reconstruction network based on a multi-task framework (i.e., a multi-task reconstruction network), and using simulation data to iteratively optimize and train the reconstruction network (i.e., the multi-task reconstruction network) to enable the reconstruction network to acquire accurate solution capabilities; constructing a live mouse orthotopic transplantation animal model of tumor cells, collecting fluorescence data of the injected probe over a period of time, mapping and registering it with the standard mesh to obtain the network input vector, and inputting it into the reconstruction network to obtain the morphological and 3D position information of the tumor in the mouse.

[0127] Based on the above-mentioned method for three-dimensional reconstruction of tumor fluorescence images, this disclosure also provides a device for three-dimensional reconstruction of tumor fluorescence images. The following will be combined with... Figure 4 The device is described in detail.

[0128] Figure 4 A structural block diagram of a three-dimensional reconstruction apparatus for tumor fluorescence images according to an embodiment of the present disclosure is shown.

[0129] like Figure 4 As shown, the tumor fluorescence image three-dimensional reconstruction device 400 of this embodiment includes a construction and registration module 410, a feature extraction and fusion module 420, and a three-dimensional reconstruction module 430.

[0130] The construction and registration module 410 is used to construct a finite element tetrahedral discrete mesh based on the medical tomographic image of the target organism, and to map the two-dimensional fluorescence image of the target organism onto the surface of the finite element tetrahedral discrete mesh through a registration operation to obtain a surface fluorescence intensity vector. In one embodiment, the construction and registration module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0131] The feature extraction and fusion module 420 is used to extract energy features and morphological features from the surface fluorescence intensity vector using the trained multi-task reconstruction network, respectively, to obtain intermediate energy intensity features and intermediate morphological classification features. The intermediate energy intensity features and intermediate morphological classification features are then fused to obtain a position information vector. In one embodiment, the feature extraction and fusion module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0132] The 3D reconstruction module 430 is used to perform 3D rendering of the tumor of the target organism in a finite element tetrahedral discrete mesh using position information vectors, thereby obtaining the 3D morphological information of the tumor in the target organism. In one embodiment, the 3D reconstruction module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0133] According to embodiments of this disclosure, any plurality of modules among the construction and registration module 410, the feature extraction and fusion module 420, and the 3D reconstruction module 430 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the construction and registration module 410, the feature extraction and fusion module 420, and the 3D reconstruction module 430 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any one of the three implementation methods or a suitable combination of any of them. Alternatively, at least one of the construction and registration module 410, the feature extraction and fusion module 420, and the 3D reconstruction module 430 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0134] Figure 5 A block diagram of an electronic device suitable for implementing a three-dimensional reconstruction method of tumor fluorescence images according to an embodiment of the present disclosure is shown.

[0135] like Figure 5As shown, an electronic device 500 according to an embodiment of this disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0136] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0137] According to embodiments of this disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0138] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0139] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0140] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the tumor fluorescence image three-dimensional reconstruction method provided in the embodiments of this disclosure.

[0141] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0142] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0143] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0144] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0146] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0147] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method of three-dimensional reconstruction of a tumor fluorescence image, characterized by, The method includes: A finite element tetrahedral discrete mesh is constructed based on the medical tomographic images of the target organism, and the two-dimensional fluorescence image of the target organism is mapped onto the surface of the finite element tetrahedral discrete mesh through a registration operation to obtain the surface fluorescence intensity vector. The trained multi-task reconstruction network is used to extract energy features and morphological features from the surface fluorescence intensity vector to obtain intermediate energy intensity features and intermediate morphological classification features. The intermediate energy intensity features and intermediate morphological classification features are then fused to obtain the position information vector. The tumor of the target organism is rendered in three dimensions using the position information vector in the finite element tetrahedral discrete mesh, thereby obtaining the three-dimensional morphological information of the tumor in the target organism.

2. The method of claim 1, wherein, The two-dimensional fluorescence image of the target organism is mapped onto the surface of the finite element tetrahedral discrete mesh through a registration operation to obtain the surface fluorescence intensity vector, which includes: The parameters of the optical system used to acquire the two-dimensional fluorescence image are iteratively optimized by minimizing the error between the coordinates of the marker points in the two-dimensional fluorescence image and the coordinates of the marker points in the medical tomographic image, resulting in an optimized optical system. The position of the two-dimensional fluorescence image relative to the finite element tetrahedral discrete mesh is projected and corrected using the optimized optical system to obtain the corrected two-dimensional fluorescence image. The corrected two-dimensional fluorescence image is mapped onto the surface of the finite element tetrahedral discrete mesh by calculating the surface vertex distribution of the corrected two-dimensional fluorescence image in the finite element tetrahedral discrete mesh, thereby obtaining the surface fluorescence intensity vector.

3. The method of claim 2, wherein, The parameters of the optical system used to acquire the two-dimensional fluorescence image are iteratively optimized by minimizing the error between the coordinates of the marker points in the two-dimensional fluorescence image and the coordinates of the marker points in the medical tomographic image, resulting in an optimized optical system comprising: The optical system is used to convert the coordinates of the marker points in the two-dimensional fluorescence image and the coordinates of the marker points in the medical tomographic image from pixel coordinates to actual physical dimensions, thereby obtaining the physical coordinates of the fluorescent marker points and the physical coordinates of the marker points in the medical tomographic image. The three-dimensional coordinates of all possible lens centers in the optical system are adjusted in multiple rounds to minimize the error between the predicted coordinates of the fluorescent marker points of each lens center and the physical coordinates of the marker points in the medical tomographic image. The obtained errors are then summed to obtain the error function. The parameters of the optical system are iteratively optimized using the error function until a preset condition is met, resulting in an optimized optical system.

4. The method of claim 2, wherein, The optimized optical system is used to project and correct the position of the two-dimensional fluorescence image relative to the finite element tetrahedral discrete mesh, resulting in a corrected two-dimensional fluorescence image including: The optimized optical system is used to determine the viewing angle of the two-dimensional fluorescence image and the spatial coordinates of the detector used to generate the two-dimensional fluorescence image; Based on the viewing direction and the spatial coordinates, the normal vector, area and center coordinates of each surface triangular facet in the finite element tetrahedral discrete mesh are calculated, and the visible part of the finite element tetrahedral discrete mesh in the two-dimensional fluorescence image is determined based on the normal vector, area and center coordinates. The corrected two-dimensional fluorescence image is obtained by projecting the visible portion onto the plane of the detector and then correcting the two-dimensional fluorescence image.

5. The method of claim 4, wherein, The corrected two-dimensional fluorescence image is mapped onto the surface of the finite element tetrahedral discrete mesh by calculating the surface vertex distribution of the two-dimensional fluorescence image in the mesh, resulting in the surface fluorescence intensity vector, which includes: The power value of the surface triangular facet is obtained using the grayscale value determined by the corrected two-dimensional fluorescence image and the optimized optical system. Based on the three-dimensional coordinates of the vertices in the finite element tetrahedral discrete mesh and the normal vector, area, center coordinates and three-dimensional coordinates of the surface triangular facets, all the surface triangular facets are traversed. If the traversal result is that the target surface triangular facet has vertices, the power and area of ​​the target surface triangular facets are accumulated to obtain the accumulated power and total area. The accumulated power and the total area are calculated to obtain the average power of the vertex. Then, the corrected two-dimensional fluorescence image is mapped onto the surface of the finite element tetrahedral discrete mesh to obtain the surface fluorescence intensity vector.

6. The method of claim 1, wherein, The trained multi-task reconstruction network is used to extract energy features and morphological features from the surface fluorescence intensity vector, respectively, to obtain intermediate energy intensity features and intermediate morphological classification features, including: The first branch module of the trained multi-task reconstruction network is used to extract energy features from the surface fluorescence intensity vector to obtain intermediate energy intensity features. The first branch module includes a first expert network. The second branch module of the trained multi-task reconstruction network is used to extract morphological binary classification features from the surface fluorescence intensity vector to obtain intermediate morphological classification features. The second branch module includes a second expert network.

7. The method of claim 1, wherein, The energy intensity intermediate feature and the morphology classification intermediate feature are fused to obtain the location information vector, which includes: The energy intensity intermediate feature and the morphological classification intermediate feature are processed by sharing through the shared layer of the trained multi-task reconstruction network to obtain the first shared feature and the second shared feature. The gating unit of the trained multi-task reconstruction network is used to perform weighted processing on the first shared feature and the second shared feature to obtain the first weighted feature and the second weighted feature. The first weighted feature is hierarchically integrated using the first tower structure network of the trained multi-task reconstruction network to obtain energy intensity integrated features, and the second weighted feature is hierarchically integrated using the second tower structure network of the trained multi-task reconstruction network to obtain morphological binary classification integrated features. The location information vector is obtained by performing a dot product operation on the energy intensity integrated feature and the morphological binary classification integrated feature using the trained multi-task reconstruction network.

8. The method according to any one of claims 1 to 7, characterized in that, The trained multi-task reconstruction network is obtained through the following operations: A tumor fluorescence image simulation dataset is constructed, and a multi-task reconstruction network is constructed, wherein the multi-task reconstruction network includes multiple branch modules, multiple shared layers, gating units, and multiple tower structure networks; The parameters of the multi-task reconstruction network are iteratively optimized based on gradient balance using the tumor fluorescence image simulation dataset and a preset loss function until the preset training conditions are met, thus obtaining the trained multi-task reconstruction network. The preset loss function includes a focus loss function, a binary classification smooth similarity coefficient loss function, and a mean squared error loss function.

9. The method according to claim 8, characterized in that, The construction of the tumor fluorescence image simulation dataset includes: A standard anatomical structure mesh was constructed using medical tomographic image samples of segmented experimental organisms; The absorption coefficient and scattering coefficient of the experimental organism in the near-infrared II fluorescence spectrum are calculated, and the absorption coefficient and scattering coefficient are used to generate a simulated single-light source dataset of the experimental organism. The simulated single-light source datasets are combined or aggregated to obtain simulated multi-light source datasets and simulated large-light source datasets, respectively. A simulated light source dataset is then constructed based on the simulated single-light source datasets, the simulated multi-light source datasets, and the simulated large-light source datasets. The Monte Carlo algorithm is used to calculate the surface spot vectors of different light source vectors inside the grid in the simulated light source dataset on the standard anatomical structure grid, wherein the surface spot vectors are used to simulate the mapping result of the tumor fluorescence image of the experimental organism on the standard anatomical structure grid; The simulated light source dataset with grid-internal light source vector-surface light spot vector pairs is used as the tumor fluorescence image simulation dataset of the experimental organism.

10. A three-dimensional reconstruction device for tumor fluorescence images, characterized in that, The device includes: The construction and registration module is used to construct a finite element tetrahedral discrete mesh based on the medical tomographic image of the target organism, and to map the two-dimensional fluorescence image of the target organism onto the surface of the finite element tetrahedral discrete mesh through a registration operation to obtain the surface fluorescence intensity vector. The feature extraction and fusion module is used to extract energy features and morphological features from the surface fluorescence intensity vector using the trained multi-task reconstruction network, respectively, to obtain intermediate energy intensity features and intermediate morphological classification features, and then fuses the intermediate energy intensity features and the intermediate morphological classification features to obtain a position information vector. The three-dimensional rendering module is used to perform three-dimensional rendering of the tumor of the target organism in the finite element tetrahedral discrete mesh using the position information vector, so as to obtain the three-dimensional morphological information of the tumor in the target organism.

Images