A three-dimensional reconstruction method and device for biomedical body surface images

Abstract

The application provides a three-dimensional reconstruction method of biomedical body surface images, which can be applied to the technical field of biomedical image processing. The method comprises the following steps: preprocessing a three-dimensional reconstruction data sample set to obtain a standardized data sample set, and training a body surface three-dimensional reconstruction neural network by using the standardized data sample set to obtain a trained body surface three-dimensional reconstruction neural network; preprocessing a 2D body surface image of an imaging target organism and camera parameters corresponding to the 2D body surface image to obtain a preprocessed 2D body surface image and preprocessed camera parameters; and performing feature extraction, feature fusion and 3D reconstruction on the preprocessed 2D body surface image and the preprocessed camera parameters by using the trained body surface three-dimensional reconstruction neural network to obtain 3D body surface structure data of the imaging target organism after reconstruction. The application further provides a three-dimensional reconstruction device of biomedical body surface images, an electronic device and a storage medium.

Description

Technical Field

[0001] This invention relates to the field of biomedical image processing technology, and more specifically to a method and apparatus for three-dimensional reconstruction of biomedical body surface images, electronic devices, and storage media. Background Technology

[0002] Biomedical surface imaging is a technique used to capture and analyze images of biological surfaces. It typically utilizes various imaging techniques to visualize the morphology, structure, and function of biological surfaces. The core task of biomedical surface imaging is to infer the three-dimensional (3D) surface structure of a sample from a two-dimensional (2D) image. Currently, commonly used biomedical surface imaging techniques mainly include laser scanning imaging, time-of-flight imaging, stereo vision imaging, and structured light imaging. However, existing biomedical surface imaging techniques heavily rely on a large number of multi-view, low-noise 2D image samples, resulting in limited application scenarios and high costs. Other biomedical surface imaging techniques that do not rely on a large number of multi-view, low-noise 2D image samples suffer from low imaging accuracy and poor reconstruction results. Summary of the Invention

[0003] In view of the above problems, the present invention provides a method and apparatus for three-dimensional reconstruction of biomedical body surface images, an electronic device and a storage medium, for at least solving one of the above technical problems.

[0004] According to a first aspect of the present invention, a method for three-dimensional reconstruction of biomedical body surface images is provided, the method comprising:

[0005] The constructed 3D reconstruction data sample set is preprocessed to obtain a standardized data sample set, and the constructed 3D reconstruction neural network of the body surface is trained using the standardized data sample set to obtain the trained 3D reconstruction neural network of the body surface.

[0006] The acquired 2D body surface image of the target organism to be imaged and the corresponding camera parameters are preprocessed to obtain the preprocessed 2D body surface image and the preprocessed camera parameters.

[0007] The trained 3D reconstruction neural network is used to extract features, fuse features, and reconstruct 3D structures of the preprocessed 2D body surface image and preprocessed camera parameters to obtain the 3D body surface structure data of the target biological organism after reconstruction.

[0008] According to an embodiment of the present invention, the above-mentioned three-dimensional reconstruction neural network for body surface includes an image encoder module, a parameter encoder module, a feature fusion module, a 3D body surface feature generation module, a 2D rendering module, and a 3D rendering module.

[0009] According to an embodiment of the present invention, the above-described image encoder module is constructed based on a residual network;

[0010] The parameter encoder module includes multiple first fully connected layers and a first SiLu activation function layer.

[0011] The feature fusion module includes a Transformer layer and multiple Swing-Transformer layers with different window sizes.

[0012] The 3D body surface feature generation module includes multiple second fully connected layers and multiple second SiLU activation function layers;

[0013] The 2D rendering module is built based on predefined rendering formulas;

[0014] The 3D rendering module is built based on the Marching Cube algorithm.

[0015] According to an embodiment of the present invention, the above-mentioned training of the constructed three-dimensional reconstruction neural network of the body surface using a standardized data sample set to obtain the trained three-dimensional reconstruction neural network of the body surface includes:

[0016] A 3D reconstruction data sample set is constructed using collected body surface image samples and corresponding camera intrinsic and extrinsic parameter samples. The 3D reconstruction data sample set is then preprocessed to obtain a standardized data sample set.

[0017] The three-dimensional reconstruction neural network of the body surface is used to process multiple sets of training data samples randomly selected from a standardized data sample set to obtain the sample processing results;

[0018] The difference between the sample processing results and the body surface image samples in multiple training data samples is calculated using the mean squared error loss function to obtain the loss value;

[0019] The parameters of the three-dimensional reconstruction neural network of the body surface are optimized using the loss value, and the training process is controlled using a preset training mode until the preset training conditions are met, thus obtaining the trained three-dimensional reconstruction neural network of the body surface.

[0020] According to an embodiment of the present invention, the above-described method of constructing a three-dimensional reconstruction data sample set using collected body surface image samples and corresponding camera intrinsic and extrinsic parameter samples includes:

[0021] Based on the authorization of the data owner, biomedical body surface images and corresponding camera parameters are collected to obtain body surface image samples and camera parameter samples. Among them, the camera parameter samples include camera intrinsic parameter samples and camera extrinsic parameter samples.

[0022] Background removal is performed on the body surface image samples to obtain body surface image samples with a white background, thus completing the preprocessing of the body surface image samples.

[0023] The camera intrinsic parameters are normalized based on the size of the body surface image samples to generate a normalized camera intrinsic parameter matrix, and the camera extrinsic parameters are transformed into a predefined global coordinate system to complete the preprocessing of the camera parameter samples.

[0024] The preprocessed body surface image samples, along with the corresponding preprocessed camera extrinsic and intrinsic parameters, are combined to obtain the 3D reconstruction data samples.

[0025] According to an embodiment of the present invention, the above-described processing of multiple sets of training data samples randomly selected from a standardized data sample set using a three-dimensional reconstruction neural network of the body surface to obtain sample processing results includes:

[0026] A set of training data samples is randomly selected from the standardized data sample set as the master training data sample for the current training round, and multiple other sets of training data samples are randomly extracted from the standardized data sample set as slave training data samples for the current training round.

[0027] The main training data samples are input into the body surface 3D reconstruction neural network for processing, and the camera intrinsic and extrinsic parameters from the training data samples are simultaneously input into the body surface 3D reconstruction neural network for processing, resulting in multiple sample processing results.

[0028] According to an embodiment of the present invention, the above-described input of the main training data samples into the three-dimensional reconstruction neural network of the body surface for processing, and the simultaneous input of the camera intrinsic and extrinsic parameters from the training data samples into the three-dimensional reconstruction neural network of the body surface for processing, to obtain multiple sample processing results, including:

[0029] The image encoder module is used to extract sample image features from the body surface image samples of the main training data samples;

[0030] The camera intrinsic parameter samples and camera extrinsic parameter samples are stitched together, and the parameter encoder module is used to extract features from the stitched camera parameters to obtain the sample parameter features.

[0031] The feature fusion module is used to fuse sample parameter features and sample image features to obtain sample fused features;

[0032] The 3D body surface feature generation module is used to process the sample fusion features to obtain 3D feature data of the sample with body surface structure information.

[0033] By using the 2D rendering module to process 3D feature data of samples with body surface structure information, as well as camera intrinsic and extrinsic parameter samples from training data samples, 2D body surface image samples with a new perspective are obtained.

[0034] The 3D feature data of samples with body surface structure information are processed using the 3D rendering module to obtain the sample reconstruction results that characterize the 3D body surface structure.

[0035] According to an embodiment of the present invention, the above-described method of fusing sample parameter features and sample image features using a feature fusion module to obtain sample fusion features includes:

[0036] The sample parameter features and sample image features are processed through the Transformer layer of the feature fusion module to obtain the first processing result;

[0037] The first processing result is processed by the first type of Swin-Transformer in the feature fusion module to obtain the second processing result;

[0038] The first processing result is processed by the second type of Swin-Transformer in the feature fusion module to obtain the third processing result. The first type of Swin-Transformer and the second type of Swin-Transformer have different window values.

[0039] The feature fusion module is used to concatenate the features of the second and third processing results to obtain the sample fusion features.

[0040] According to an embodiment of the present invention, the above-mentioned preset training mode is set by employing the Adam optimization algorithm, setting an initial learning rate, and using the cosine annealing algorithm as a learning rate adjustment strategy.

[0041] According to an embodiment of the present invention, the above-mentioned use of a trained three-dimensional reconstruction neural network to perform feature extraction, feature fusion, and 3D reconstruction on a preprocessed 2D body surface image and preprocessed camera parameters to obtain 3D body surface structure data of the reconstructed target organism includes:

[0042] The trained 3D reconstruction neural network for body surface is used to extract features from the preprocessed 2D body surface image to obtain image features, and the preprocessed camera parameters are also used to extract features to obtain parameter features.

[0043] The trained 3D reconstruction neural network for body surface is used to fuse image features and parametric features to obtain fused features. Then, 3D body surface features are generated from the fused features to obtain 3D feature data with body surface structure information.

[0044] The trained three-dimensional reconstruction neural network is used to perform 3D reconstruction based on voxel-level isosurface extraction on 3D feature data with body surface structural information, and the three-dimensional reconstruction results of the target organism to be imaged are obtained.

[0045] According to a second aspect of the present invention, a three-dimensional reconstruction apparatus for biomedical body surface images is provided, the apparatus comprising:

[0046] The model training module is used to preprocess the constructed 3D reconstruction data sample set to obtain a standardized data sample set, and to train the constructed 3D reconstruction neural network of the body surface using the standardized data sample set to obtain the trained 3D reconstruction neural network of the body surface.

[0047] The image preprocessing module is used to preprocess the acquired 2D body surface image of the target organism to be imaged and the corresponding camera parameters to obtain the preprocessed 2D body surface image and the preprocessed camera parameters.

[0048] The model processing module is used to extract features, fuse features, and reconstruct 3D structures from preprocessed 2D body surface images and preprocessed camera parameters using a trained 3D body surface reconstruction neural network, thereby obtaining 3D body surface structure data of the target biological organism after reconstruction.

[0049] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0050] A fourth aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0051] The three-dimensional reconstruction method for biomedical body surface images provided by this invention trains a three-dimensional reconstruction neural network for body surface and uses the trained neural network to perform high-precision three-dimensional reconstruction of a single body surface image of the target organism. This solves the problem that existing technologies require a large number of precisely labeled 2D body surface images for three-dimensional reconstruction. At the same time, the reconstruction process provided by this invention is efficient, avoids a lot of tedious optimization operations, saves time for those skilled in the art, and reduces the cost of using three-dimensional reconstruction. Attached Figure Description

[0052] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0053] Figure 1 This is a flowchart of a three-dimensional reconstruction method for biomedical body surface images according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the structure of a three-dimensional reconstruction neural network for the body surface according to an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the structure of the parameter encoder module in the three-dimensional reconstruction neural network of the body surface according to an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of the feature fusion module in a three-dimensional reconstruction neural network for body surface according to an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the structure of the 3D body surface feature generation module in the three-dimensional reconstruction neural network of the body surface according to an embodiment of the present invention;

[0058] Figure 6 A schematic diagram illustrating the structure of a three-dimensional reconstruction apparatus for biomedical body surface images according to an embodiment of the present invention is shown.

[0059] Figure 7 A block diagram of an electronic device suitable for implementing a three-dimensional reconstruction method for biomedical body surface images according to an embodiment of the present invention is shown schematically. Detailed Implementation

[0060] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0061] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0062] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0063] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0064] Existing biomedical surface imaging techniques can provide relatively accurate three-dimensional surface reconstructions when acquiring a sufficient number of 2D images with wide angular coverage and low noise. However, if these conditions are not met, the reconstruction results are often unsatisfactory, such as poor reconstruction accuracy.

[0065] In recent years, deep learning technology has demonstrated significant breakthroughs in multiple fields, thus creating an urgent need for a deep learning-based method for reconstructing body surface structures. Compared to existing methods, this new method can achieve accurate reconstruction of three-dimensional body surface structures from a single 2D image, thereby overcoming the limitations of traditional techniques.

[0066] Figure 1 This is a flowchart of a three-dimensional reconstruction method for biomedical body surface images according to an embodiment of the present invention.

[0067] like Figure 1 As shown, the three-dimensional reconstruction method of the above-mentioned biomedical body surface image includes operations S110 to S130.

[0068] In operation S110, the constructed 3D reconstruction data sample set is preprocessed to obtain a standardized data sample set, and the constructed 3D reconstruction neural network of the body surface is trained using the standardized data sample set to obtain the trained 3D reconstruction neural network of the body surface.

[0069] Before performing 3D reconstruction of the surface image of the target organism, the neural network for 3D reconstruction of the surface needs to be trained.

[0070] In acquiring the training dataset, taking mice as the target organism for simulation, 2D images of multiple mice and their corresponding camera parameters (including camera extrinsic and intrinsic parameters) can be obtained from any suitable source, such as research institutions or public datasets. The collected 3D reconstruction dataset can be represented as: ={(( ), ..., ( ), ..., ( )), ..., (( ), ..., ( ), ..., ( )), ..., (( ), ..., ( ), ..., ( ))}, where the dataset contains Data for only one mouse, with each mouse corresponding to n sets of 2D images and camera parameters. Specifically, Indicates the first The first mouse Group of 2D images, Indicates the first The first mouse Camera external parameters, Indicates the first The first mouse Camera internal parameters.

[0071] It should be noted that the dataset used in this invention is based on authorized public datasets. The use of the above datasets in this invention strictly complies with the provisions of relevant laws and regulations and has taken strict confidentiality measures.

[0072] In operation S120, the acquired 2D body surface image of the target organism to be imaged and the corresponding camera parameters are preprocessed to obtain the preprocessed 2D body surface image and the preprocessed camera parameters.

[0073] In operation S130, the trained 3D reconstruction neural network is used to extract features, fuse features, and reconstruct 3D structures of the preprocessed 2D body surface image and preprocessed camera parameters to obtain the 3D body surface structure data of the target biological reconstruction.

[0074] The three-dimensional reconstruction method for biomedical body surface images provided by this invention trains a three-dimensional reconstruction neural network for body surface and uses the trained neural network to perform high-precision three-dimensional reconstruction of a single body surface image of the target organism. This solves the problem that existing technologies require a large number of precisely labeled 2D body surface images for three-dimensional reconstruction. At the same time, the reconstruction process provided by this invention is efficient, avoids a lot of tedious optimization operations, saves time for those skilled in the art, and reduces the cost of using three-dimensional reconstruction.

[0075] According to an embodiment of the present invention, the above-mentioned three-dimensional reconstruction neural network for body surface includes an image encoder module, a parameter encoder module, a feature fusion module, a 3D body surface feature generation module, a 2D rendering module, and a 3D rendering module.

[0076] According to embodiments of the present invention, the image encoder module is constructed based on a residual network; wherein, the parameter encoder module includes multiple first fully connected layers and a first SiLu activation function layer; wherein, the feature fusion module includes a Transformer layer and multiple Swin-Transformer layers with different window sizes; wherein, the 3D body surface feature generation module includes multiple second fully connected layers and multiple second SiLu activation function layers; wherein, the 2D rendering module is constructed based on a predefined rendering formula; wherein, the 3D rendering module is constructed based on the Marching Cube algorithm.

[0077] The following specific embodiments, in conjunction with the appendix, demonstrate this process. Figure 2 The structure and function of the three-dimensional reconstruction neural network of the body surface involved in this invention will be described in further detail.

[0078] Figure 2 This is a schematic diagram of the structure of a three-dimensional reconstruction neural network for the body surface according to an embodiment of the present invention.

[0079] like Figure 2 As shown, the neural network structure for 3D body surface reconstruction includes an image encoder module, a parameter encoder module, a feature fusion module, a 3D body surface feature generation module, a 2D rendering module, and a 3D rendering module. During the training of the 3D body surface reconstruction neural network, new perspective camera parameters need to be incorporated, and the 2D rendering module processes these parameters. Figure 2 As can be seen, the 3D reconstruction neural network for body surface requires the simultaneous input of a 2D body surface image of the target organism to be imaged and the corresponding camera parameters, and finally outputs a 3D body surface structure, i.e., the 3D reconstruction result; during the training process, it will also output a new perspective 2D image.

[0080] According to an embodiment of the present invention, the above-mentioned training of the constructed three-dimensional reconstruction neural network of the body surface using a standardized data sample set to obtain the trained three-dimensional reconstruction neural network of the body surface includes: constructing a three-dimensional reconstruction data sample set using collected body surface image samples and corresponding camera intrinsic and extrinsic parameter samples, and preprocessing the three-dimensional reconstruction data sample set to obtain a standardized data sample set; processing multiple sets of training data samples randomly selected from the standardized data sample set using the three-dimensional reconstruction neural network of the body surface to obtain sample processing results; calculating the difference between the sample processing results and the body surface image samples in the multiple sets of training data samples using the mean squared error loss function to obtain a loss value; optimizing the parameters of the three-dimensional reconstruction neural network of the body surface using the loss value, and controlling the training process using a preset training mode until the preset training conditions are met to obtain the trained three-dimensional reconstruction neural network of the body surface.

[0081] According to an embodiment of the present invention, the above-mentioned construction of a three-dimensional reconstruction data sample set using collected body surface image samples and corresponding camera intrinsic and extrinsic parameter samples includes: collecting biomedical body surface images and corresponding camera parameters based on authorization from the data owner to obtain body surface image samples and camera parameter samples, wherein the camera parameter samples include camera intrinsic and extrinsic parameter samples; removing the background from the body surface image samples to obtain body surface image samples with a white background, thereby completing the preprocessing of the body surface image samples; normalizing the camera intrinsic parameters based on the size of the body surface image samples to generate a normalized camera intrinsic parameter matrix, and converting the camera extrinsic parameters to a predefined global coordinate system to complete the preprocessing of the camera parameter samples; and combining the preprocessed body surface image samples and corresponding preprocessed camera extrinsic and intrinsic parameter samples to obtain three-dimensional reconstruction data samples.

[0082] The following section uses a mouse as an example to illustrate the above data preprocessing process in detail.

[0083] First, background removal is performed on all 2D images to obtain 2D images with a white background. Then, camera intrinsic parameters are normalized based on image dimensions to generate a normalized camera intrinsic parameter matrix. Finally, all camera extrinsic parameters are transformed to a predefined global coordinate system. After these steps, a standardized 3D reconstruction dataset is formed, represented as follows: ={(( ), ..., ( ), ..., ( )), ..., (( ), ..., ( ), ..., ( )), ..., (( ), ..., ( ), ..., ( ))},in, Indicates the number after preprocessing. The first mouse Group of 2D images, Indicates the number after preprocessing. The first mouse Camera external parameters, Indicates the number after preprocessing. The first mouse Camera intrinsics. To ensure the robustness and generalization ability of the model, the entire 3D reconstruction dataset is divided into training, validation, and test sets in a 5:2:3 ratio.

[0084] According to an embodiment of the present invention, the above-described method of using a three-dimensional reconstruction neural network of the body surface to process multiple sets of training data samples randomly selected from a standardized data sample set to obtain sample processing results includes: randomly selecting a set of training data samples from the standardized data sample set as the main training data samples for the current training round, and randomly extracting multiple other sets of training data samples from the standardized data sample set as secondary training data samples for the current training round; inputting the main training data samples into the three-dimensional reconstruction neural network of the body surface for processing, and simultaneously inputting the camera intrinsic parameters and camera extrinsic parameters from the secondary training data samples into the three-dimensional reconstruction neural network of the body surface for processing, thereby obtaining multiple sample processing results.

[0085] The following uses a mouse as the target organism for simulation imaging to illustrate the training process of a three-dimensional reconstruction neural network for the body surface in detail through specific examples.

[0086] First, a set of mouse data is randomly selected from the training set: (( ), ..., ( ), ..., ( For each set of 2D images and camera parameter data, two more sets of 2D images and camera parameter data are randomly extracted from the mouse data, for example: ), ( ), ( ).

[0087] Will( This set of data is input into a three-dimensional reconstruction neural network of the body surface, and then ( ) )and( These two sets of camera parameters are also input into the body surface 3D reconstruction neural network to obtain the 2D image rendered by the body surface 3D reconstruction neural network. and Then, the MSE loss function is used to evaluate... and , and The differences between them are used to train a neural network for 3D reconstruction of the body surface.

[0088] According to an embodiment of the present invention, the Adam optimization algorithm is employed, with an initial learning rate set to 0.001 and a cosine annealing strategy used for learning rate adjustment. The above steps are applied to train the 3D reconstruction neural network for the body surface for each set of 2D images and camera parameters for each mouse.

[0089] According to an embodiment of the present invention, the above-mentioned input of the main training data samples into the body surface 3D reconstruction neural network for processing, and simultaneously input of the camera intrinsic and extrinsic parameters from the training data samples into the body surface 3D reconstruction neural network for processing, to obtain multiple sample processing results include: extracting sample image features from the body surface image samples of the main training data samples using an image encoder module; stitching the camera intrinsic and extrinsic parameter samples together, and extracting features from the stitched camera parameters using a parameter encoder module to obtain sample parameter features; fusing the sample parameter features and sample image features using a feature fusion module to obtain sample fusion features; processing the sample fusion features using a 3D body surface feature generation module to obtain sample 3D feature data with body surface structure information; processing the sample 3D feature data with body surface structure information and the camera intrinsic and extrinsic parameter samples from the training data samples using a 2D rendering module to obtain 2D body surface image samples with a new perspective; and processing the sample 3D feature data with body surface structure information using a 3D rendering module to obtain sample reconstruction results characterizing the 3D body surface structure.

[0090] The following detailed description of the various functional modules of the three-dimensional reconstruction neural network for the body surface in this invention, using specific embodiments (with a mouse as the target organism for imaging) and in conjunction with the accompanying drawings, will further illustrate these concepts.

[0091] Image Encoder Module: The 3D reconstruction neural network for the body surface includes an image encoder module. This module uses a ResNet-50 pre-trained on ImageNet to extract abstract high-level image features from 2D images. The input to the image encoder module is a 2D image of a mouse, for example... The output image features of the 2D image are then input into the feature fusion module.

[0092] Figure 3 This is a schematic diagram of the parameter encoder module in a three-dimensional reconstruction neural network for body surface according to an embodiment of the present invention.

[0093] like Figure 3 As shown, the parameter encoder module contains two fully connected layers and one SiLU activation function, which are used to extract abstract high-level parameter features from camera parameters.

[0094] The input to the parameter encoder module is camera parameters, such as... , After being concatenated together, the data is input into the parameter encoder, which outputs parameter features. The output data is then input into the feature fusion module.

[0095] According to an embodiment of the present invention, the above-mentioned method of fusing sample parameter features and sample image features using a feature fusion module to obtain sample fusion features includes: processing the sample parameter features and sample image features through the Transformer layer of the feature fusion module to obtain a first processing result; processing the first processing result through a first type of Swin-Transformer of the feature fusion module to obtain a second processing result; processing the first processing result through a second type of Swin-Transformer of the feature fusion module to obtain a third processing result, wherein the first type of Swin-Transformer and the second type of Swin-Transformer have different window values; and using the feature fusion module to concatenate the second processing result and the third processing result to obtain the sample fusion features.

[0096] Figure 4 This is a schematic diagram of the feature fusion module in a three-dimensional reconstruction neural network for body surface according to an embodiment of the present invention.

[0097] like Figure 4 As shown, the feature fusion module includes one Transformer layer, five Swin-Transformer layers with a window size of 2, and five Swin-Transformer layers with a window size of 4, which are used to guide the parametric features to generate abstract features of the three-dimensional volume surface from the image features.

[0098] The input to the feature fusion module is image features and parametric features. These two types of features are passed through a Transformer layer and then fed into a Swin-Transformer layer with a window size of 2 and a Swin-Transformer layer with a window size of 4, respectively. Finally, the outputs of the two layers are simply added together as the output of the feature fusion module. The feature data output by the feature fusion module will be input into the 3D body surface feature generation module.

[0099] Figure 5 This is a schematic diagram of the structure of the 3D body surface feature generation module in the three-dimensional reconstruction neural network of the body surface according to an embodiment of the present invention.

[0100] like Figure 5 As shown, the 3D body surface feature generation module contains four fully connected layers and three SiLU activation function layers, which are used to generate feature data containing body surface structural information from the abstract features of the three-dimensional body surface.

[0101] The input to the 3D body surface feature generation module is the feature data output by the feature fusion module. After passing through multiple fully connected layers and SiLU activation function layers, the output is 3D feature data, which will be input into the 2D rendering module and the 3D rendering module.

[0102] 2D Rendering Module: The body surface 3D reconstruction neural network includes a 2D rendering module. This module generates 2D images under given camera parameters to train the neural network. The 2D rendering module requires new camera parameters as input, for example... and The entire module generates new 2D images based on the volume rendering formula (existing technology) and the 3D features output by the 3D volume surface feature generation module.

[0103] 3D Rendering Module: The 3D reconstruction neural network for body surface includes a 3D rendering module. The 3D rendering module uses the Marching Cube algorithm to generate 3D body surface structure data from the 3D features output by the 3D body surface feature generation module.

[0104] After taking 2D images of the mouse to be reconstructed using a camera, the 2D images and corresponding camera parameters are input into a trained 3D reconstruction neural network for the body surface. The 3D rendering module of the 3D reconstruction neural network can then output 3D body surface structure data.

[0105] According to an embodiment of the present invention, the above-mentioned method of using a trained three-dimensional reconstruction neural network to perform feature extraction, feature fusion, and 3D reconstruction on a preprocessed 2D body surface image and preprocessed camera parameters to obtain 3D body surface structure data of the target organism to be imaged includes: using the trained three-dimensional reconstruction neural network to extract features from the preprocessed 2D body surface image to obtain image features, and using the preprocessed camera parameters to extract features to obtain parameter features; using the trained three-dimensional reconstruction neural network to fuse the image features and parameter features to obtain fused features, and using the fused features to generate 3D body surface features to obtain 3D feature data with body surface structure information; and using the trained three-dimensional reconstruction neural network to perform 3D reconstruction based on voxel-level isosurface extraction on the 3D feature data with body surface structure information to obtain the 3D reconstruction result of the target organism to be imaged.

[0106] The 3D reconstruction based on voxel-level isosurface extraction used in the above operations may optionally utilize the Marching Cube algorithm.

[0107] Based on the aforementioned three-dimensional reconstruction method for biomedical body surface images, this invention also provides a three-dimensional reconstruction device for biomedical body surface images. The following will be combined with... Figure 6 The device is described in detail.

[0108] Figure 6 A schematic diagram of a three-dimensional reconstruction apparatus for biomedical body surface images according to an embodiment of the present invention is shown.

[0109] like Figure 6As shown, the three-dimensional reconstruction device 600 for biomedical body surface images in this embodiment includes a model training module 610, an image preprocessing module 620, and a model processing module 630.

[0110] The model training module 610 is used to preprocess the constructed 3D reconstruction data sample set to obtain a standardized data sample set, and then use the standardized data sample set to train the constructed 3D reconstruction neural network of the body surface to obtain a trained 3D reconstruction neural network of the body surface. In one embodiment, the training module 610 can be used to perform the operation S110 described above, which will not be repeated here.

[0111] The image preprocessing module 620 is used to preprocess the acquired 2D body surface image of the target organism and the corresponding camera parameters to obtain a preprocessed 2D body surface image and preprocessed camera parameters. In one embodiment, the preprocessing module 620 can be used to perform the operation S120 described above, which will not be repeated here.

[0112] The model processing module 630 is used to perform feature extraction, feature fusion, and 3D reconstruction on the preprocessed 2D body surface image and preprocessed camera parameters using a trained 3D body surface reconstruction neural network, to obtain the reconstructed 3D body surface structure data of the target biological image. In one embodiment, the feature extraction module 630 can be used to perform the operation S130 described above, which will not be repeated here.

[0113] Any plurality of modules in the model training module 610, image preprocessing module 620, and model processing module 630 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the model training module 610, image preprocessing module 620, and model processing module 630 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the model training module 610, image preprocessing module 620, and model processing module 630 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0114] Figure 7A block diagram of an electronic device suitable for implementing a three-dimensional reconstruction method for biomedical body surface images according to an embodiment of the present invention is shown schematically.

[0115] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0116] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0117] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0118] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0119] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0121] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0122] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for three-dimensional reconstruction of biomedical body surface images, characterized in that, The method includes: The constructed 3D reconstruction data sample set is preprocessed to obtain a standardized data sample set, and the constructed 3D reconstruction neural network of the body surface is trained using the standardized data sample set to obtain a trained 3D reconstruction neural network of the body surface. The 3D reconstruction neural network of the body surface includes an image encoder module, a parameter encoder module, a feature fusion module, a 3D body surface feature generation module, a 2D rendering module, and a 3D rendering module. The acquired 2D body surface image of the target organism to be imaged and the corresponding camera parameters are preprocessed to obtain the preprocessed 2D body surface image and the preprocessed camera parameters. The trained 3D reconstruction neural network is used to perform feature extraction, feature fusion, and 3D reconstruction on the preprocessed 2D body surface image and the preprocessed camera parameters to obtain the 3D body surface structure data of the target biological organism after reconstruction. The standardized data sample set is used to train the constructed three-dimensional reconstruction neural network of the body surface, resulting in a trained three-dimensional reconstruction neural network of the body surface, including: A three-dimensional reconstruction data sample set is constructed using collected body surface image samples and corresponding camera intrinsic and extrinsic parameter samples. The three-dimensional reconstruction data sample set is then preprocessed to obtain the standardized data sample set. A set of training data samples is randomly selected from the standardized data sample set as the main training data sample for the current training round, and multiple other sets of training data samples are randomly extracted from the standardized data sample set as the secondary training data samples for the current training round. The image encoder module is used to extract sample image features from the body surface image samples of the main training data samples; The camera intrinsic parameter samples and the camera extrinsic parameter samples are concatenated, and the parameter encoder module is used to extract features from the concatenated camera parameters to obtain sample parameter features. The feature fusion module is used to fuse the sample parameter features and the sample image features to obtain sample fusion features; The 3D body surface feature generation module is used to process the sample fusion features to obtain sample 3D feature data with body surface structure information. The 2D rendering module is used to process the 3D feature data of the sample with body surface structure information, as well as the camera intrinsic and extrinsic parameters from the training data sample, to obtain a 2D body surface image sample with a new perspective. The 3D rendering module is used to process the 3D feature data of the sample with body surface structure information to obtain the sample reconstruction result characterizing the 3D body surface structure. The difference between the sample reconstruction result and the body surface image samples in the multiple sets of training data samples is calculated using the mean squared error loss function to obtain the loss value; The parameters of the three-dimensional reconstruction neural network of the body surface are optimized using the loss value, and the training process is controlled using a preset training mode until the preset training conditions are met, thus obtaining the trained three-dimensional reconstruction neural network of the body surface.

2. The method according to claim 1, characterized in that, The image encoder module is constructed based on a residual network; The parameter encoder module includes multiple first fully connected layers and a first SiLu activation function layer; The feature fusion module includes a Transformer layer and multiple Swing-Transformer layers with different window sizes. The 3D body surface feature generation module includes multiple second fully connected layers and multiple second SiLU activation function layers. The 2D rendering module is constructed based on a predefined rendering formula. The 3D rendering module is built based on the Marching Cube algorithm.

3. The method according to claim 1, characterized in that, The construction of the 3D reconstruction data sample set using collected body surface image samples and corresponding camera intrinsic and extrinsic parameter samples includes: Based on the authorization of the data owner, biomedical body surface images and corresponding camera parameters are collected to obtain body surface image samples and camera parameter samples, wherein the camera parameter samples include camera intrinsic parameter samples and camera extrinsic parameter samples. Background removal is performed on the body surface image sample to obtain a body surface image sample with a white background, thus completing the preprocessing of the body surface image sample; The camera intrinsic parameters are normalized based on the size of the body surface image sample to generate a normalized camera intrinsic parameter matrix, and the camera extrinsic parameters are transformed into a predefined global coordinate system to complete the preprocessing of the camera parameter sample. The preprocessed body surface image sample, along with the corresponding preprocessed camera extrinsic and intrinsic parameters, are combined to obtain the three-dimensional reconstruction data sample.

4. The method according to claim 1, characterized in that, The feature fusion module fuses the sample parameter features and the sample image features to obtain sample fusion features, including: The sample parameter features and the sample image features are processed through the Transformer layer of the feature fusion module to obtain the first processing result; The first processing result is processed by the first type of Swin-Transformer of the feature fusion module to obtain the second processing result; The first processing result is processed by the second type of Swin-Transformer of the feature fusion module to obtain the third processing result, wherein the first type of Swin-Transformer and the second type of Swin-Transformer have different window values; The feature fusion module is used to concatenate the second processing result and the third processing result to obtain the sample fusion feature.

5. The method according to claim 1, characterized in that, The preset training mode is set by using the Adam optimization algorithm, setting an initial learning rate, and using the cosine annealing algorithm as the learning rate adjustment strategy.

6. The method according to claim 1, characterized in that, Using the trained 3D reconstruction neural network for body surface, feature extraction, feature fusion, and 3D reconstruction are performed on the preprocessed 2D body surface image and the preprocessed camera parameters to obtain the reconstructed 3D body surface structure data of the target biological organism, including: The trained 3D reconstruction neural network for body surface is used to extract features from the preprocessed 2D body surface image to obtain image features, and the preprocessed camera parameters are also used to extract features to obtain parameter features. The trained 3D reconstruction neural network is used to fuse the image features and the parameter features to obtain fused features, and 3D surface features are generated from the fused features to obtain 3D feature data with surface structure information. The trained three-dimensional reconstruction neural network is used to perform 3D reconstruction based on voxel-level isosurface extraction on the 3D feature data containing body surface structural information, thereby obtaining the three-dimensional reconstruction result of the target organism to be imaged.

7. A three-dimensional reconstruction device for biomedical body surface images, characterized in that, The device includes: The model training module is used to preprocess the constructed 3D reconstruction data sample set to obtain a standardized data sample set, and to train the constructed 3D reconstruction neural network of the body surface using the standardized data sample set to obtain the trained 3D reconstruction neural network of the body surface. The 3D reconstruction neural network of the body surface includes an image encoder module, a parameter encoder module, a feature fusion module, a 3D body surface feature generation module, a 2D rendering module, and a 3D rendering module. The image preprocessing module is used to preprocess the acquired 2D body surface image of the target organism to be imaged and the camera parameters corresponding to the 2D body surface image to obtain the preprocessed 2D body surface image and the preprocessed camera parameters. The model processing module is used to perform feature extraction, feature fusion and 3D reconstruction on the preprocessed 2D body surface image and the preprocessed camera parameters using the trained body surface 3D reconstruction neural network to obtain the 3D body surface structure data of the target biological to be imaged. The standardized data sample set is used to train the constructed three-dimensional reconstruction neural network of the body surface, resulting in a trained three-dimensional reconstruction neural network of the body surface, including: A three-dimensional reconstruction data sample set is constructed using collected body surface image samples and corresponding camera intrinsic and extrinsic parameter samples. The three-dimensional reconstruction data sample set is then preprocessed to obtain the standardized data sample set. A set of training data samples is randomly selected from the standardized data sample set as the main training data sample for the current training round, and multiple other sets of training data samples are randomly extracted from the standardized data sample set as the secondary training data samples for the current training round. The image encoder module is used to extract sample image features from the body surface image samples of the main training data samples; The camera intrinsic parameter samples and the camera extrinsic parameter samples are concatenated, and the parameter encoder module is used to extract features from the concatenated camera parameters to obtain sample parameter features. The feature fusion module is used to fuse the sample parameter features and the sample image features to obtain sample fusion features; The 3D body surface feature generation module is used to process the sample fusion features to obtain sample 3D feature data with body surface structure information. The 2D rendering module is used to process the 3D feature data of the sample with body surface structure information, as well as the camera intrinsic and extrinsic parameters from the training data sample, to obtain a 2D body surface image sample with a new perspective. The 3D rendering module is used to process the 3D feature data of the sample with body surface structure information to obtain the sample reconstruction result characterizing the 3D body surface structure. The difference between the sample reconstruction result and the body surface image samples in the multiple sets of training data samples is calculated using the mean squared error loss function to obtain the loss value; The parameters of the three-dimensional reconstruction neural network of the body surface are optimized using the loss value, and the training process is controlled using a preset training mode until the preset training conditions are met, thus obtaining the trained three-dimensional reconstruction neural network of the body surface.

Images