Three-dimensional reconstruction method and device for biomedical body surface image

Abstract

The invention provides a three-dimensional reconstruction method of a biomedical body surface image, which can be applied to the technical field of biomedical image processing. The method comprises the following steps: preprocessing a three-dimensional reconstruction data sample set to obtain a standardized data sample set, and training a body surface three-dimensional reconstruction neural network by using the standardized data sample set to obtain a trained body surface three-dimensional reconstruction neural network; preprocessing the obtained 2D body surface image of the target organism to be imaged and the camera parameters corresponding to the 2D body surface image to obtain a preprocessed 2D body surface image and preprocessed camera parameters; and performing feature extraction, feature fusion and 3D reconstruction on the preprocessed 2D body surface image and the preprocessed camera parameters by using the trained body surface three-dimensional reconstruction neural network to obtain 3D body surface structure data after biological reconstruction of the to-be-imaged target. The invention further provides a three-dimensional reconstruction device for the biomedical body surface image, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present invention relates to the technical field of biomedical image processing, and more specifically to a three-dimensional reconstruction method and device, electronic equipment and storage medium for biomedical body surface images. Background Art

[0002] Biomedical surface imaging is a technology used to capture and analyze images of the surface of organisms. Various imaging techniques are usually used to visualize the morphology, structure and function of the surface of organisms. The core task of biomedical surface imaging is to infer the three-dimensional (3D) surface structure of the sample from a two-dimensional image (2D). At present, the commonly used biomedical surface imaging technologies mainly include laser scanning imaging, time-of-flight imaging, stereoscopic vision imaging and structured light imaging. However, the existing biomedical surface imaging technology is highly dependent on a large number of multi-view and low-noise 2D image samples, which leads to limited application scenarios and high cost of use of the existing biomedical surface imaging technology. Other biomedical surface imaging technologies that do not rely on a large number of multi-view and low-noise 2D image samples have technical problems such as low imaging accuracy and poor reconstruction effect. Summary of the invention

[0003] In view of the above problems, the present invention provides a three-dimensional reconstruction method and device for biomedical body surface images, an electronic device and a storage medium, which are used to solve at least one of the above technical problems.

[0004] According to a first aspect of the present invention, a method for three-dimensional reconstruction of a biomedical body surface image is provided, the method comprising:

[0005] Preprocessing the constructed three-dimensional reconstruction data sample set to obtain a standardized data sample set, and using the standardized data sample set to train the constructed body surface three-dimensional reconstruction neural network to obtain a trained body surface three-dimensional reconstruction neural network;

[0006] Preprocessing the acquired 2D body surface image of the target organism to be imaged and the camera parameters corresponding to the 2D body surface image to obtain a preprocessed 2D body surface image and the preprocessed camera parameters;

[0007] The trained body surface three-dimensional reconstruction neural network is used to perform feature extraction, feature fusion and 3D reconstruction on the preprocessed 2D body surface image and the preprocessed camera parameters to obtain the reconstructed 3D body surface structure data of the simulated imaging target organism.

[0008] According to an embodiment of the present invention, the above-mentioned body surface three-dimensional reconstruction neural network includes an image encoder module, a parameter encoder module, a feature fusion module, a 3D body surface feature generation module, a 2D rendering module and a 3D rendering module.

[0009] According to an embodiment of the present invention, the above-mentioned image encoder module is constructed based on a residual network;

[0010] Wherein, the parameter encoder module includes a plurality of first fully connected layers and a first SiLu activation function layer;

[0011] Among them, the feature fusion module includes a Transformer layer and multiple Swin-Transformer layers with different window sizes;

[0012] Among them, the 3D body surface feature generation module includes multiple second fully connected layers and multiple second SiLU activation function layers;

[0013] Among them, the 2D rendering module is constructed based on the predefined rendering formula;

[0014] Among them, the 3D rendering module is built based on the Marching Cube algorithm.

[0015] According to an embodiment of the present invention, the above-mentioned use of a standardized data sample set to train the constructed body surface three-dimensional reconstruction neural network to obtain a trained body surface three-dimensional reconstruction neural network includes:

[0016] A three-dimensional reconstruction data sample set is constructed using the collected body surface image samples and the camera intrinsic reference samples and camera extrinsic reference samples corresponding to the body surface image samples, and the three-dimensional reconstruction data sample set is preprocessed to obtain a standardized data sample set;

[0017] Using the body surface three-dimensional reconstruction neural network to process multiple sets of training data samples randomly selected from the standardized data sample set, the sample processing results are obtained;

[0018] The mean square error loss function is used to calculate the difference between the sample processing results and the body surface image samples in multiple groups of training data samples to obtain the loss value;

[0019] The loss value is used to optimize the parameters of the body surface three-dimensional reconstruction neural network, and the preset training mode is used to control the training process until the preset training conditions are met to obtain a trained body surface three-dimensional reconstruction neural network.

[0020] According to an embodiment of the present invention, the above-mentioned construction of a three-dimensional reconstruction data sample set using the collected body surface image samples and the camera intrinsic reference samples and camera extrinsic reference samples corresponding to the body surface image samples includes:

[0021] Based on the authorization of the data owner, biomedical body surface images and camera parameters corresponding to the biomedical body surface images are collected to obtain body surface image samples and camera parameter samples, wherein the camera parameter samples include camera intrinsic parameter samples and camera extrinsic parameter samples;

[0022] Performing background removal on the body surface image samples to obtain body surface image samples with white backgrounds, thereby completing preprocessing of the body surface image samples;

[0023] The camera intrinsic parameters are normalized based on the size of the body surface image samples to generate a normalized camera intrinsic parameter matrix, and the camera extrinsic parameters are converted into a predefined global coordinate system to complete the preprocessing of the camera parameter samples;

[0024] The preprocessed body surface image samples and the preprocessed camera extrinsic reference samples and the preprocessed camera intrinsic reference samples corresponding to the preprocessed body surface image samples are combined to obtain three-dimensional reconstruction data samples.

[0025] According to an embodiment of the present invention, the above-mentioned body surface three-dimensional reconstruction neural network is used to process multiple groups of training data samples randomly selected from a standardized data sample set, and the sample processing results obtained include:

[0026] Randomly select a group of training data samples from the standardized data sample set as the main training data samples of the current training round, and randomly select multiple other groups of training data samples from the standardized data sample set as the slave training data samples of the current training round;

[0027] The main training data samples are input into the body surface three-dimensional reconstruction neural network for processing, and the camera intrinsic parameters and camera extrinsic parameters in the training samples are simultaneously input into the body surface three-dimensional reconstruction neural network for processing to obtain multiple sample processing results.

[0028] According to an embodiment of the present invention, the main training data samples are input into the body surface 3D reconstruction neural network for processing, and the camera intrinsic parameters and camera extrinsic parameters in the training samples are simultaneously input into the body surface 3D reconstruction neural network for processing, and the multiple sample processing results obtained include:

[0029] Extracting sample image features from body surface image samples of main training data samples using an image encoder module;

[0030] The camera intrinsic parameter samples and the camera extrinsic parameter samples are spliced ​​together, and the parameter encoder module is used to extract the features of the spliced ​​camera parameters to obtain the sample parameter features;

[0031] The sample parameter features and the sample image features are fused using the feature fusion module to obtain the sample fusion features;

[0032] The sample fusion features are processed by the 3D body surface feature generation module to obtain sample 3D feature data with body surface structure information;

[0033] The 2D rendering module is used to process the sample 3D feature data with body surface structure information and the camera intrinsic reference samples and camera extrinsic reference samples in the training data samples to obtain 2D body surface image samples with a new perspective;

[0034] The 3D rendering module is used to process the sample 3D feature data with body surface structure information to obtain a sample reconstruction result representing the 3D body surface structure.

[0035] According to an embodiment of the present invention, the sample parameter feature and the sample image feature are fused by the feature fusion module to obtain the sample fusion feature, which includes:

[0036] Processing the sample parameter features and the sample image features through the Transformer layer of the feature fusion module to obtain a first processing result;

[0037] The first processing result is processed by the first type of Swin-Transformer of the feature fusion module to obtain a second processing result;

[0038] Processing the first processing result through the second type of Swin-Transformer of the feature fusion module to obtain a third processing result, wherein the first type of Swin-Transformer and the second type of Swin-Transformer have different window values;

[0039] The feature fusion module is used to perform feature splicing on the second processing result and the third processing result to obtain a sample fusion feature.

[0040] According to an embodiment of the present invention, the preset training mode is set by adopting the Adam optimization algorithm, setting the initial learning rate, and using the cosine annealing algorithm as a learning rate adjustment strategy.

[0041] According to an embodiment of the present invention, the above-mentioned trained body surface three-dimensional reconstruction neural network performs feature extraction, feature fusion and 3D reconstruction on the preprocessed 2D body surface image and the preprocessed camera parameters to obtain the 3D body surface structure data after reconstruction of the simulated imaging target organism, including:

[0042] The trained body surface three-dimensional reconstruction neural network is used to extract features from the preprocessed 2D body surface image to obtain image features, and the preprocessed camera parameters are extracted to obtain parameter features;

[0043] The trained body surface three-dimensional reconstruction neural network is used to fuse the image features and parameter features to obtain fused features, and the fused features are used to generate 3D body surface features to obtain 3D feature data with body surface structure information;

[0044] The trained body surface three-dimensional reconstruction neural network is used to perform 3D reconstruction on the 3D feature data with body surface structure information based on voxel-level isosurface extraction to obtain the three-dimensional reconstruction result of the simulated imaging target organism.

[0045] According to a second aspect of the present invention, there is provided a three-dimensional reconstruction device for biomedical body surface images, the device comprising:

[0046] A model training module is used to preprocess the constructed three-dimensional reconstruction data sample set to obtain a standardized data sample set, and use the standardized data sample set to train the constructed body surface three-dimensional reconstruction neural network to obtain a trained body surface three-dimensional reconstruction neural network;

[0047] An image preprocessing module is used to preprocess the acquired 2D body surface image of the target organism to be imaged and the camera parameters corresponding to the 2D body surface image to obtain the preprocessed 2D body surface image and the preprocessed camera parameters;

[0048] The model processing module is used to use the trained body surface three-dimensional reconstruction neural network to perform feature extraction, feature fusion and 3D reconstruction on the preprocessed 2D body surface image and the preprocessed camera parameters to obtain the reconstructed 3D body surface structure data of the simulated imaging target organism.

[0049] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0050] The fourth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0051] The three-dimensional reconstruction method of the biomedical body surface image provided by the present invention solves the defect that the prior art needs to rely on a large number of accurately labeled 2D body surface images for three-dimensional reconstruction, by training a body surface three-dimensional reconstruction neural network, and using the trained body surface three-dimensional reconstruction neural network to perform high-precision three-dimensional reconstruction of a single body surface image of the target organism to be imaged. At the same time, the reconstruction process of the above-mentioned method provided by the present invention is efficient, avoids a large number of tedious optimization operations, saves the time of technical personnel in this field, and reduces the cost of using three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0053] Figure 1 is a flow chart of a method for three-dimensional reconstruction of a biomedical body surface image according to an embodiment of the present invention;

[0054] Figure 2 is a schematic diagram of the structure of a neural network for three-dimensional reconstruction of body surface according to an embodiment of the present invention;

[0055] Figure 3 is a schematic diagram of the structure of a parameter encoder module in a body surface three-dimensional reconstruction neural network according to an embodiment of the present invention;

[0056] Figure 4 is a schematic diagram of the structure of a feature fusion module in a body surface three-dimensional reconstruction neural network according to an embodiment of the present invention;

[0057] Figure 5 is a schematic structural diagram of a 3D body surface feature generation module in a body surface three-dimensional reconstruction neural network according to an embodiment of the present invention;

[0058] Figure 6 A schematic structural block diagram of a three-dimensional reconstruction device for biomedical body surface images according to an embodiment of the present invention is shown;

[0059] Figure 7 A block diagram of an electronic device suitable for implementing a three-dimensional reconstruction method of a biomedical body surface image according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0060] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0061] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0062] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0063] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0064] Existing biomedical body surface imaging technology solutions can provide relatively accurate 3D body surface reconstruction when acquiring a sufficient number of 2D images with wide angle coverage and low noise. However, if the above conditions are not met, the reconstruction results are often unsatisfactory, such as poor reconstruction accuracy.

[0065] In recent years, deep learning technology has shown significant breakthroughs in many fields. Therefore, there is an urgent need for a deep learning-based body surface structure reconstruction method. Compared with existing methods, this new method can accurately reconstruct three-dimensional body surface structures through a single 2D image, thus overcoming the limitations of traditional technology.

[0066] Figure 1 is a flow chart of a method for three-dimensional reconstruction of a biomedical body surface image according to an embodiment of the present invention.

[0067] like Figure 1 As shown, the above-mentioned three-dimensional reconstruction method of biomedical body surface images includes operations S110 to S130.

[0068] In operation S110, the constructed 3D reconstruction data sample set is preprocessed to obtain a standardized data sample set, and the constructed body surface 3D reconstruction neural network is trained using the standardized data sample set to obtain a trained body surface 3D reconstruction neural network.

[0069] Before performing three-dimensional reconstruction of the body surface image of the target organism to be imaged, the body surface three-dimensional reconstruction neural network used needs to be trained.

[0070] In the process of obtaining the training data set, taking mice as the target organism for simulation imaging, multiple 2D images of mice and corresponding camera parameters (including camera extrinsic parameters and camera intrinsic parameters) can be obtained from any appropriate source, such as research institutions, public data sets, etc. The collected 3D reconstruction data set can be expressed as: ={(( ), ..., ( ), ..., ( )),...,(( ), ..., ( ), ..., ( )),...,(( ), ..., ( ), ..., ( ))}, where the data set contains The data of mice, each mouse corresponds to n sets of 2D images and camera parameters. Specifically, Indicates The first mouse Set of 2D images, Indicates The first mouse Group camera external parameters, Indicates The first mouse Set camera internal parameters.

[0071] It should be noted that the data set used in the present invention is based on an authorized public data set. The present invention strictly abides by the provisions of relevant laws and regulations in the use of the above data set and adopts strict confidentiality measures.

[0072] In operation S120, the acquired 2D body surface image of the target organism to be imaged and the camera parameters corresponding to the 2D body surface image are preprocessed to obtain the preprocessed 2D body surface image and the preprocessed camera parameters.

[0073] In operation S130, the trained body surface three-dimensional reconstruction neural network is used to perform feature extraction, feature fusion and 3D reconstruction on the preprocessed 2D body surface image and the preprocessed camera parameters to obtain the reconstructed 3D body surface structure data of the simulated imaging target organism.

[0074] The three-dimensional reconstruction method of the biomedical body surface image provided by the present invention solves the defect that the prior art needs to rely on a large number of accurately labeled 2D body surface images for three-dimensional reconstruction, by training a body surface three-dimensional reconstruction neural network, and using the trained body surface three-dimensional reconstruction neural network to perform high-precision three-dimensional reconstruction of a single body surface image of the target organism to be imaged. At the same time, the reconstruction process of the above-mentioned method provided by the present invention is efficient, avoids a large number of tedious optimization operations, saves the time of technical personnel in this field, and reduces the cost of using three-dimensional reconstruction.

[0075] According to an embodiment of the present invention, the above-mentioned body surface three-dimensional reconstruction neural network includes an image encoder module, a parameter encoder module, a feature fusion module, a 3D body surface feature generation module, a 2D rendering module and a 3D rendering module.

[0076] According to an embodiment of the present invention, the above-mentioned image encoder module is constructed based on a residual network; wherein the parameter encoder module includes multiple first fully connected layers and first SiLu activation function layers; wherein the feature fusion module includes a Transformer layer and multiple Swin-Transformer layers with different window sizes; wherein the 3D body surface feature generation module includes multiple second fully connected layers and multiple second SiLU activation function layers; wherein the 2D rendering module is constructed based on a predefined rendering formula; wherein the 3D rendering module is constructed based on the Marching Cube algorithm.

[0077] The following is a specific embodiment and combined with the attached Figure 2 The structure and function of the body surface three-dimensional reconstruction neural network involved in the present invention are further described in detail.

[0078] Figure 2 4 is a schematic diagram of the structure of a neural network for three-dimensional reconstruction of body surface according to an embodiment of the present invention.

[0079] like Figure 2 As shown in FIG, the structure of the body surface 3D reconstruction neural network includes an image encoder module, a parameter encoder module, a feature fusion module, a 3D body surface feature generation module, a 2D rendering module, and a 3D rendering module. In the process of training the body surface 3D reconstruction neural network, it is necessary to add new perspective camera parameters and use the 2D rendering module to process the new perspective camera parameters. Figure 2 As can be seen in the figure, the body surface 3D reconstruction neural network needs to simultaneously input the 2D body surface image of the target organism to be imaged and the corresponding camera parameters, and finally output the 3D body surface structure, that is, the 3D reconstruction result; during the training process, new perspective 2D images are also output.

[0080] According to an embodiment of the present invention, the above-mentioned use of a standardized data sample set to train the constructed body surface three-dimensional reconstruction neural network to obtain a trained body surface three-dimensional reconstruction neural network includes: using the collected body surface image samples and the camera intrinsic parameter samples and the camera extrinsic parameter samples corresponding to the body surface image samples to construct a three-dimensional reconstruction data sample set, and preprocessing the three-dimensional reconstruction data sample set to obtain a standardized data sample set; using the body surface three-dimensional reconstruction neural network to process multiple groups of training data samples randomly selected from the standardized data sample set to obtain sample processing results; using a mean square error loss function to calculate the difference between the sample processing results and the body surface image samples in the multiple groups of training data samples to obtain a loss value; using the loss value to optimize the parameters of the body surface three-dimensional reconstruction neural network, and using a preset training mode to control the training process until the preset training conditions are met to obtain a trained body surface three-dimensional reconstruction neural network.

[0081] According to an embodiment of the present invention, the above-mentioned construction of a three-dimensional reconstruction data sample set using the collected body surface image samples and the camera intrinsic parameter samples and camera extrinsic parameter samples corresponding to the body surface image samples includes: based on the authorization of the data owner, biomedical body surface images and camera parameters corresponding to the biomedical body surface images are collected to obtain body surface image samples and camera parameter samples, wherein the camera parameter samples include camera intrinsic parameter samples and camera extrinsic parameter samples; background removal is performed on the body surface image samples to obtain body surface image samples with a white background to thereby complete the preprocessing of the body surface image samples; camera intrinsic parameters are normalized based on the size of the body surface image samples to generate a normalized camera intrinsic parameter matrix, and the camera extrinsic parameters are converted into a predefined global coordinate system to thereby complete the preprocessing of the camera parameter samples; the preprocessed body surface image samples and the preprocessed camera extrinsic parameter samples and the preprocessed camera intrinsic parameter samples corresponding to the preprocessed body surface image samples are combined to obtain three-dimensional reconstruction data samples.

[0082] The following describes the data preprocessing process in detail by taking mice as an example of the simulated imaging target organism.

[0083] First, all 2D images are background-removed to obtain 2D images with white backgrounds. Then, the camera intrinsic parameters are normalized based on the image size to generate a normalized camera intrinsic parameter matrix. Finally, all camera extrinsic parameters are converted to a predefined global coordinate system. After the above steps, a standardized 3D reconstruction dataset is finally formed, which is expressed as: ={(( ), ..., ( ), ..., ( )),...,(( ), ..., ( ), ..., ( )),...,(( ), ..., ( ), ..., ( ))},in, After preprocessing, The first mouse Set of 2D images, After preprocessing, The first mouse Group camera external parameters, After preprocessing, The first mouse To ensure the robustness and generalization ability of the model, the entire 3D reconstruction dataset is divided into training set, validation set and test set in a ratio of 5:2:3.

[0084] According to an embodiment of the present invention, the above-mentioned use of the body surface three-dimensional reconstruction neural network to process multiple groups of training data samples randomly selected from a standardized data sample set to obtain sample processing results includes: randomly selecting a group of training data samples from the standardized data sample set as the main training data samples of the current training round, and randomly extracting another multiple groups of training data samples from the standardized data sample set as the slave training data samples of the current training round; inputting the main training data samples into the body surface three-dimensional reconstruction neural network for processing, and at the same time inputting the camera intrinsic parameters and camera extrinsic parameters in the training samples into the body surface three-dimensional reconstruction neural network for processing, to obtain multiple sample processing results.

[0085] The following uses mice as the target organism for simulated imaging to explain in detail the training process of the neural network for three-dimensional reconstruction of the body surface through specific examples.

[0086] First, randomly extract a group of mouse data from the training set: (( ), ..., ( ), ..., ( )), for each set of 2D images and camera parameter data, two more sets of 2D images and camera parameter data are randomly extracted from the mouse data, for example: ( ),( ),( ).

[0087] Will( ) This set of data is input into the body surface 3D reconstruction neural network, and then ( )and( ) These two sets of camera parameters are also input into the body surface 3D reconstruction neural network to obtain the 2D image rendered by the body surface 3D reconstruction neural network and , and then use the MSE loss function to evaluate and , and The difference between them is used to train the neural network for body surface 3D reconstruction.

[0088] According to an embodiment of the present invention, the Adam optimization algorithm is used, the initial learning rate is set to 0.001, and the learning rate adjustment strategy is cosine annealing. The above steps are used to train the body surface three-dimensional reconstruction neural network for each set of 2D images and camera parameters of each mouse data.

[0089] According to an embodiment of the present invention, the main training data samples are input into the body surface three-dimensional reconstruction neural network for processing, and the camera intrinsic parameters and camera extrinsic parameters in the training samples are simultaneously input into the body surface three-dimensional reconstruction neural network for processing, and multiple sample processing results are obtained, including: using an image encoder module to extract sample image features from the body surface image samples of the main training data samples; splicing the camera intrinsic parameter samples and the camera extrinsic parameter samples, and using a parameter encoder module to extract features of the spliced ​​camera parameters to obtain sample parameter features; using a feature fusion module to fuse the sample parameter features and the sample image features to obtain sample fusion features; using a 3D body surface feature generation module to process the sample fusion features to obtain sample 3D feature data with body surface structure information; using a 2D rendering module to process the sample 3D feature data with body surface structure information and the camera intrinsic parameter samples and camera extrinsic parameter samples in the training data samples to obtain 2D body surface image samples with a new perspective; using a 3D rendering module to process the sample 3D feature data with body surface structure information to obtain a sample reconstruction result representing the 3D body surface structure.

[0090] The following is a further detailed description of the various functional modules of the body surface three-dimensional reconstruction neural network in the present invention through a specific embodiment (taking mice as the simulated imaging target organism) and in combination with the accompanying drawings.

[0091] Image encoder module: The body surface 3D reconstruction neural network contains an image encoder module, which uses ResNet-50 pre-trained on ImageNet to extract abstract high-level image features from 2D images. The input of the image encoder module is a 2D image of a mouse, such as , the image features of the 2D image are output, and the output data is input into the feature fusion module.

[0092] Figure 3 It is a structural schematic diagram of a parameter encoder module in a body surface three-dimensional reconstruction neural network according to an embodiment of the present invention.

[0093] like Figure 3 As shown in Figure 1, the parameter encoder module contains two fully connected layers and one SiLU activation function, which is used to extract abstract high-level parameter features from camera parameters.

[0094] The input of the parameter encoder module is the camera parameters, for example , After being spliced ​​together, they are input into the parameter encoder to output parameter features, and the output data is input into the feature fusion module.

[0095] According to an embodiment of the present invention, the above-mentioned use of the feature fusion module to fuse the sample parameter features and the sample image features to obtain the sample fusion features includes: processing the sample parameter features and the sample image features through the Transformer layer of the feature fusion module to obtain a first processing result; processing the first processing result through the first type of Swin-Transformer of the feature fusion module to obtain a second processing result; processing the first processing result through the second type of Swin-Transformer of the feature fusion module to obtain a third processing result, wherein the first type of Swin-Transformer and the second type of Swin-Transformer have different window values; and using the feature fusion module to perform feature splicing on the second processing result and the third processing result to obtain the sample fusion feature.

[0096] Figure 4 4 is a schematic diagram of the structure of a feature fusion module in a body surface three-dimensional reconstruction neural network according to an embodiment of the present invention.

[0097] like Figure 4 As shown, the feature fusion module includes a Transformer layer, 5 Swin-Transformer layers with a window size of 2, and 5 Swin-Transformer layers with a window size of 4, which are used to guide the image features to generate abstract features of the three-dimensional body surface using parameter features.

[0098] The input of the feature fusion module is image features and parameter features. After passing through a Transformer layer, these two features are respectively input into the Swin-Transformer layer with a window size of 2 and the Swin-Transformer layer with a window size of 4. Finally, the outputs of these two layers are simply added together as the output of the feature fusion module. The feature data output by the feature fusion module will be input into the 3D body surface feature generation module.

[0099] Figure 5 It is a structural schematic diagram of a 3D body surface feature generation module in a body surface three-dimensional reconstruction neural network according to an embodiment of the present invention.

[0100] like Figure 5 As shown, the 3D body surface feature generation module includes four fully connected layers and three SiLU activation function layers, which are used to generate feature data containing body surface structure information from the abstract features of the three-dimensional body surface.

[0101] The input of the 3D body surface feature generation module is the feature data output by the feature fusion module. After passing through multiple layers of fully connected layers and SiLU activation function layers, the 3D feature data is output. The 3D feature data will be input into the 2D rendering module and the 3D rendering module.

[0102] 2D rendering module: The body surface 3D reconstruction neural network contains a 2D rendering module, which is used to generate 2D images under given camera parameters to realize the training of body surface 3D reconstruction neural network. The 2D rendering module needs to pass in new camera parameters, such as and ,The whole module generates a new 2D image based on the volume rendering formula (existing ,technology) and utilizes the 3D features output by the 3D body surface feature generation ,module.

[0103] 3D rendering module: The body surface three-dimensional reconstruction neural network includes a 3D rendering module, which uses the Marching Cube algorithm to generate 3D body surface structure data from the 3D features output by the 3D body surface feature generation module.

[0104] After taking a 2D image of the mouse to be 3D reconstructed using a camera, the 2D image and the corresponding camera parameters are input into the trained body surface 3D reconstruction neural network, and the 3D rendering module of the 3D reconstruction neural network can output 3D body surface structure data.

[0105] According to an embodiment of the present invention, the above-mentioned use of the trained body surface three-dimensional reconstruction neural network to perform feature extraction, feature fusion and 3D reconstruction on the preprocessed 2D body surface image and the preprocessed camera parameters to obtain the 3D body surface structure data reconstructed for the simulated imaging target organism includes: using the trained body surface three-dimensional reconstruction neural network to perform feature extraction on the preprocessed 2D body surface image to obtain image features, and performing feature extraction on the preprocessed camera parameters to obtain parameter features; using the trained body surface three-dimensional reconstruction neural network to fuse the image features and the parameter features to obtain fused features, and performing 3D body surface feature generation on the fused features to obtain 3D feature data with body surface structure information; using the trained body surface three-dimensional reconstruction neural network to perform 3D reconstruction based on voxel-level isosurface extraction on the 3D feature data with body surface structure information to obtain the three-dimensional reconstruction result of the simulated imaging target organism.

[0106] The 3D reconstruction based on voxel-level isosurface extraction used in the above operation may optionally utilize a Marching Cube algorithm.

[0107] Based on the above-mentioned three-dimensional reconstruction method of biomedical body surface images, the present invention also provides a three-dimensional reconstruction device for biomedical body surface images. Figure 6 The device is described in detail.

[0108] Figure 6 The structure block diagram of the apparatus for three-dimensional reconstruction of biomedical body surface images according to an embodiment of the present invention is schematically shown.

[0109] like Figure 6As shown, the three-dimensional reconstruction device 600 of the biomedical body surface image of this embodiment includes a model training module 610 , an image preprocessing module 620 and a model processing module 630 .

[0110] The model training module 610 is used to preprocess the constructed 3D reconstruction data sample set to obtain a standardized data sample set, and use the standardized data sample set to train the constructed body surface 3D reconstruction neural network to obtain a trained body surface 3D reconstruction neural network. In one embodiment, the training module 610 can be used to perform the operation S110 described above, which will not be repeated here.

[0111] The image preprocessing module 620 is used to preprocess the acquired 2D body surface image of the target organism to be imaged and the camera parameters corresponding to the 2D body surface image to obtain the preprocessed 2D body surface image and the preprocessed camera parameters. In one embodiment, the preprocessing module 620 can be used to perform the operation S120 described above, which will not be repeated here.

[0112] The model processing module 630 is used to perform feature extraction, feature fusion and 3D reconstruction on the preprocessed 2D body surface image and the preprocessed camera parameters using the trained body surface 3D reconstruction neural network to obtain the 3D body surface structure data reconstructed by the simulated imaging target organism. In one embodiment, the feature extraction module 630 can be used to perform the operation S130 described above, which will not be repeated here.

[0113] Any multiple modules of the model training module 610, the image preprocessing module 620, and the model processing module 630 can be combined into one module for implementation, or any one of them can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the model training module 610, the image preprocessing module 620, and the model processing module 630 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the model training module 610, the image preprocessing module 620, and the model processing module 630 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0114] Figure 7A block diagram of an electronic device suitable for implementing a three-dimensional reconstruction method of a biomedical body surface image according to an embodiment of the present invention is schematically shown.

[0115] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0116] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0117] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output (I / O) interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0118] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0119] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0120] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0122] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A three-dimensional reconstruction method for biomedical body surface images, characterized in that: The method comprises: Preprocessing the constructed three-dimensional reconstruction data sample set to obtain a standardized data sample set, and using the standardized data sample set to train the constructed body surface three-dimensional reconstruction neural network to obtain a trained body surface three-dimensional reconstruction neural network; Preprocessing the acquired 2D body surface image of the target organism to be imaged and the camera parameters corresponding to the 2D body surface image to obtain a preprocessed 2D body surface image and the preprocessed camera parameters; The trained body surface three-dimensional reconstruction neural network is used to perform feature extraction, feature fusion and 3D reconstruction on the preprocessed 2D body surface image and the preprocessed camera parameters to obtain the reconstructed 3D body surface structure data of the simulated imaging target organism.

2. The method according to claim 1, characterized in that The body surface three-dimensional reconstruction neural network includes an image encoder module, a parameter encoder module, a feature fusion module, a 3D body surface feature generation module, a 2D rendering module and a 3D rendering module.

3. The method according to claim 2, characterized in that The image encoder module is constructed based on a residual network; Wherein, the parameter encoder module includes a plurality of first fully connected layers and a first SiLu activation function layer; The feature fusion module includes a Transformer layer and a plurality of Swin-Transformer layers with different window sizes; Wherein, the 3D body surface feature generation module includes a plurality of second fully connected layers and a plurality of second SiLU activation function layers; Wherein, the 2D rendering module is constructed based on a predefined rendering formula; Wherein, the 3D rendering module is constructed based on the Marching Cube algorithm.

4. The method according to claim 2, characterized in that: The constructed body surface three-dimensional reconstruction neural network is trained using the standardized data sample set to obtain a trained body surface three-dimensional reconstruction neural network including: Constructing a three-dimensional reconstruction data sample set by using the collected body surface image samples and the camera intrinsic reference samples and camera extrinsic reference samples corresponding to the body surface image samples, and preprocessing the three-dimensional reconstruction data sample set to obtain the standardized data sample set; Using the body surface three-dimensional reconstruction neural network to process a plurality of groups of training data samples randomly selected from the standardized data sample set to obtain sample processing results; Calculating the difference between the sample processing result and the body surface image samples in the plurality of groups of training data samples using a mean square error loss function to obtain a loss value; The loss value is used to optimize the parameters of the body surface three-dimensional reconstruction neural network, and the training process is controlled using a preset training mode until the preset training conditions are met, thereby obtaining the trained body surface three-dimensional reconstruction neural network.

5. The method according to claim 4, characterized in that Constructing the three-dimensional reconstruction data sample set using the collected body surface image samples and the camera intrinsic reference samples and camera extrinsic reference samples corresponding to the body surface image samples includes: Based on the authorization of the data owner, biomedical body surface images and camera parameters corresponding to the biomedical body surface images are collected to obtain body surface image samples and camera parameter samples, wherein the camera parameter samples include camera intrinsic parameter samples and camera extrinsic parameter samples; Performing background removal on the body surface image sample to obtain a body surface image sample with a white background, thereby completing preprocessing of the body surface image sample; Normalizing the camera intrinsic parameters based on the size of the body surface image samples to generate a normalized camera intrinsic parameter matrix, and converting the camera extrinsic parameters into a predefined global coordinate system to complete preprocessing of the camera parameter samples; The preprocessed body surface image samples and the preprocessed camera extrinsic reference samples and the preprocessed camera intrinsic reference samples corresponding to the preprocessed body surface image samples are combined to obtain the three-dimensional reconstruction data samples.

6. The method according to claim 4, characterized in that The body surface three-dimensional reconstruction neural network is used to process a plurality of groups of training data samples randomly selected from the standardized data sample set, and the sample processing results obtained include: Randomly select a group of training data samples from the standardized data sample set as the main training data samples of the current training round, and randomly select another plurality of groups of training data samples from the standardized data sample set as the slave training data samples of the current training round; The main training data samples are input into the body surface three-dimensional reconstruction neural network for processing, and the camera intrinsic parameters and camera extrinsic parameters in the slave training samples are simultaneously input into the body surface three-dimensional reconstruction neural network for processing to obtain multiple sample processing results.

7. The method according to claim 6, characterized in that The main training data samples are input into the body surface three-dimensional reconstruction neural network for processing, and the camera intrinsic parameters and camera extrinsic parameters in the slave training samples are simultaneously input into the body surface three-dimensional reconstruction neural network for processing, and a plurality of sample processing results are obtained, including: Extracting sample image features from body surface image samples of the main training data samples using the image encoder module; Splicing the camera intrinsic parameter sample and the camera extrinsic parameter sample, and extracting features of the spliced ​​camera parameters using the parameter encoder module to obtain sample parameter features; Using the feature fusion module to fuse the sample parameter feature and the sample image feature to obtain a sample fusion feature; Processing the sample fusion features using the 3D body surface feature generation module to obtain sample 3D feature data with body surface structure information; Processing the sample 3D feature data having body surface structure information and the camera intrinsic reference samples and camera extrinsic reference samples from the training data samples by using the 2D rendering module to obtain a 2D body surface image sample with a new viewing angle; The 3D rendering module is used to process the sample 3D feature data having body surface structure information to obtain a sample reconstruction result representing the 3D body surface structure.

8. The method according to claim 7, characterized in that The sample parameter feature and the sample image feature are fused by using the feature fusion module to obtain the sample fusion feature, including: Processing the sample parameter feature and the sample image feature through the Transformer layer of the feature fusion module to obtain a first processing result; Processing the first processing result through the first type of Swin-Transformer of the feature fusion module to obtain a second processing result; Processing the first processing result through the second type of Swin-Transformer of the feature fusion module to obtain a third processing result, wherein the first type of Swin-Transformer and the second type of Swin-Transformer have different window values; The feature fusion module is used to perform feature splicing on the second processing result and the third processing result to obtain the sample fusion feature.

9. The method according to claim 4, characterized in that The preset training mode is set by adopting the Adam optimization algorithm, setting the initial learning rate, and using the cosine annealing algorithm as a learning rate adjustment strategy.

10. The method according to claim 1, characterized in that Using the trained body surface three-dimensional reconstruction neural network to perform feature extraction, feature fusion and 3D reconstruction on the preprocessed 2D body surface image and the preprocessed camera parameters, the 3D body surface structure data reconstructed by the simulated imaging target organism is obtained, including: Using the trained body surface three-dimensional reconstruction neural network, extracting features from the preprocessed 2D body surface image to obtain image features, and extracting features from the preprocessed camera parameters to obtain parameter features; The image feature and the parameter feature are fused using the trained body surface three-dimensional reconstruction neural network to obtain a fused feature, and 3D body surface feature generation is performed on the fused feature to obtain 3D feature data with body surface structure information; The trained body surface three-dimensional reconstruction neural network is used to perform 3D reconstruction based on voxel-level isosurface extraction on the 3D feature data with body surface structure information to obtain a three-dimensional reconstruction result of the simulated imaging target organism.

11. A three-dimensional reconstruction device for biomedical body surface images, characterized in that: The device comprises: A model training module is used to preprocess the constructed three-dimensional reconstruction data sample set to obtain a standardized data sample set, and use the standardized data sample set to train the constructed body surface three-dimensional reconstruction neural network to obtain a trained body surface three-dimensional reconstruction neural network; An image preprocessing module, used for preprocessing the acquired 2D body surface image of the target organism to be imaged and the camera parameters corresponding to the 2D body surface image to obtain the preprocessed 2D body surface image and the preprocessed camera parameters; The model processing module is used to use the trained body surface three-dimensional reconstruction neural network to perform feature extraction, feature fusion and 3D reconstruction on the preprocessed 2D body surface image and the preprocessed camera parameters to obtain the reconstructed 3D body surface structure data of the simulated imaging target organism.

Images