A three-dimensional modeling method and device based on multi-source heterogeneous data

Abstract

The present invention provides a three-dimensional modeling method and device based on multi-source heterogeneous data, which are as follows: obtaining a combination of images corresponding to multiple anatomical planes of a target part; for any anatomical plane, fusing the combination of images corresponding to the anatomical plane based on a multi-source heterogeneous fusion model to obtain a fused image; performing three-dimensional reconstruction on the fused images of the multiple anatomical planes to obtain a three-dimensional model of the target part; wherein a discriminator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for a sample image combination corresponding to a sample anatomical plane and the output probability of a fused image output by a generator; a generator is trained based on the output probability of the discriminator for a fused image output by the generator and the difference between the fused image output by the generator and a two-dimensional CT image and a two-dimensional MRI image in the sample image combination, so that the generator can be used to generate a fused image with more complete details, thereby generating a better three-dimensional model.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional modeling, and in particular to a three-dimensional modeling method and device based on multi-source heterogeneous data. Background Art

[0002] At present, medical imaging technology has been widely used in medical medicine and other fields. Among them, imaging technologies represented by CT (Computed Tomography) and MRI (Magnetic Resonance Imaging) can provide non-invasive or minimally invasive ways to observe the patient's body conditions, and are widely used in disease screening, surgical navigation, treatment planning, etc. Among them, different imaging technologies have their own advantages and disadvantages. For example, CT images have high spatial resolution and are suitable for viewing bones and hard tissue structures, but the contrast of soft tissues is poor. MRI images are suitable for showing high-contrast soft tissue details, but the spatial resolution is lower than CT. Therefore, integrating medical imaging data from different imaging devices and different imaging methods into the same three-dimensional model can help medical personnel obtain more comprehensive anatomical and pathological information.

[0003] At present, when integrating medical images from different imaging devices, registration techniques such as rigid registration are usually used to register multiple medical images, and then the registered medical images are superimposed and 3D reconstructed to form a 3D model. However, due to the different imaging principles of different medical images, there may be differences in resolution, noise, distortion, etc., and registration technology is usually difficult to ensure the consistency of all anatomical details. Therefore, the superimposed image will have the problem of missing or blurred feature information, resulting in poor 3D modeling effect. Summary of the invention

[0004] The present invention provides a three-dimensional modeling method and device based on multi-source heterogeneous data, which is used to solve the problem in the prior art that the three-dimensional modeling effect is poor due to the different imaging principles of different medical images, resulting in missing or blurred image feature information after fusion.

[0005] In one aspect, the present invention provides a three-dimensional modeling method based on multi-source heterogeneous data, comprising:

[0006] Acquire a combination of images corresponding to multiple anatomical planes of the target part; wherein the combination of images corresponding to any anatomical plane at least includes a registered two-dimensional CT image and a two-dimensional MRI image corresponding to any anatomical plane;

[0007] For any anatomical plane, the image combination corresponding to the any anatomical plane is fused based on the multi-source heterogeneous fusion model to obtain a fused image of the any anatomical plane;

[0008] Performing three-dimensional reconstruction on the fused images of multiple anatomical planes to obtain a three-dimensional model of the target part;

[0009] Among them, the multi-source heterogeneous fusion model is constructed based on a generative adversarial network; the discriminator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the sample image combination corresponding to the sample anatomical plane and the output probability of the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane; the generator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, and the difference between the two-dimensional CT image and the two-dimensional MRI image in the sample image combination.

[0010] In an optional embodiment of the present application, the generator of the multi-source heterogeneous fusion model is trained based on the following method:

[0011] Inputting the sample image combination corresponding to the sample anatomical plane into the generator to obtain a fused image of the sample anatomical plane output by the generator;

[0012] Inputting the fused image of the sample anatomical plane into the discriminator, and obtaining an output probability corresponding to the fused image of the sample anatomical plane output by the discriminator;

[0013] Inputting the fused image of the sample anatomical plane and the sample image combination corresponding to the sample anatomical plane into the image segmentation model respectively, to obtain the contour of the corresponding image output by the image segmentation model;

[0014] Determining a generation loss based on a difference between a value of 1 and an output probability corresponding to a fused image of the sample anatomical plane;

[0015] Determine a contour loss based on a difference between a contour of the fused image of the sample anatomical plane and a contour of a two-dimensional CT image and a contour of a two-dimensional MRI image in a combination of sample images corresponding to the sample anatomical plane;

[0016] Determining a structural loss based on a structural similarity between a fused image of the sample anatomical plane and a two-dimensional MRI image in a combination of sample images corresponding to the sample anatomical plane;

[0017] determining a perceptual loss based on a difference in image features between a fused image of the sample anatomical plane and a two-dimensional CT image in a combination of sample images corresponding to the sample anatomical plane;

[0018] Based on the generation loss, the contour loss, the structure loss and the perceptual loss, the parameters of the generator are adjusted.

[0019] In an optional embodiment of the present application, the training process of the generator further includes:

[0020] Determining MRI structural differences based on structural similarities between two-dimensional MRI images in a sample image combination corresponding to adjacent sample anatomical planes and two-dimensional MRI images in a sample image combination corresponding to the sample anatomical planes;

[0021] Determining CT structural differences based on structural similarities between a two-dimensional CT image in a sample image combination corresponding to adjacent sample anatomical planes and a two-dimensional CT image in a sample image combination corresponding to the sample anatomical planes;

[0022] determining a fused structural difference based on a structural similarity between a fused image of an adjacent sample anatomical plane and a fused image of the sample anatomical plane;

[0023] determining a reconstruction loss based on a difference between an average of the MRI structural difference and the CT structural difference and the fused structural difference;

[0024] Parameters of the generator are adjusted based on the generation loss, the contour loss, the structure loss, the perceptual loss and the reconstruction loss.

[0025] In an optional embodiment of the present application, determining the contour loss based on the difference between the contour of the fused image of the sample anatomical plane and the contour of the two-dimensional CT image and the contour of the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane includes:

[0026] determining contours of the fused images of adjacent sample anatomical planes;

[0027] determining a first difference between a contour of the fused image of the adjacent sample anatomical plane and a contour of the fused image of the sample anatomical plane;

[0028] Determine a second difference and a third difference between the contour of the fused image of the sample anatomical plane and the contour of the two-dimensional CT image and the contour of the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane, respectively;

[0029] Based on the first difference, the second difference and the third difference, the contour loss is determined.

[0030] In an optional embodiment of the present application, determining the structural loss based on the structural similarity between the fused image of the sample anatomical plane and the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane includes:

[0031] Calculate the structural similarity between the fused image of the sample anatomical plane and the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane based on the SSIM index;

[0032] The difference between the value 1 and the structural similarity is determined as the structural loss.

[0033] In an optional embodiment of the present application, determining the perceptual loss based on the image feature difference between the fused image of the sample anatomical plane and the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane includes:

[0034] Based on the image feature extraction model, image features of the fused image of the sample anatomical plane and image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane are respectively extracted;

[0035] The perceptual loss is determined based on the difference in image features between the image features of the fused image of the sample anatomical plane and the image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane.

[0036] In an optional embodiment of the present application, the image feature difference between the image features of the fused image of the sample anatomical plane and the image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane is determined based on the Euclidean distance between the image features of the fused image of the sample anatomical plane and the image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane.

[0037] In an optional embodiment of the present application, the discriminator of the multi-source heterogeneous fusion model is trained based on the following method:

[0038] Inputting the two-dimensional CT image and the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane into the discriminator respectively, and obtaining the output probabilities corresponding to the two-dimensional CT image and the two-dimensional MRI image in the sample image combination output by the discriminator;

[0039] Inputting the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane into the discriminator, and obtaining the output probability corresponding to the fused image of the sample anatomical plane output by the discriminator;

[0040] Determine the discrimination loss based on a first difference between the value 1 and the output probability corresponding to the two-dimensional CT image in the sample image combination, a second difference between the value 1 and the output probability corresponding to the two-dimensional MRI image in the sample image combination, and the output probability corresponding to the fused image of the sample anatomical plane;

[0041] Parameters of the discriminator are adjusted based on the discriminative loss.

[0042] In an optional embodiment of the present application, the image combination corresponding to any anatomical plane is fused based on the multi-source heterogeneous fusion model to obtain a fused image of any anatomical plane, including:

[0043] splicing the two-dimensional CT image and the two-dimensional MRI image in the image combination corresponding to any anatomical plane to obtain a spliced ​​image;

[0044] The stitched image is input into the generator of the multi-source heterogeneous fusion model to obtain a fused image of any anatomical plane.

[0045] In a second aspect, the present invention further provides a three-dimensional modeling device based on multi-source heterogeneous data, comprising:

[0046] An image acquisition unit, used to obtain image combinations corresponding to multiple anatomical planes of the target part; wherein the image combination corresponding to any anatomical plane at least includes a registered two-dimensional CT image and a two-dimensional MRI image corresponding to any anatomical plane;

[0047] A two-dimensional image fusion unit is used for fusing the image combination corresponding to any anatomical plane based on a multi-source heterogeneous fusion model to obtain a fused image of any anatomical plane;

[0048] A three-dimensional modeling unit, used for performing three-dimensional reconstruction on the fused images of multiple anatomical planes to obtain a three-dimensional model of the target part;

[0049] Among them, the multi-source heterogeneous fusion model is constructed based on a generative adversarial network; the discriminator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the sample image combination corresponding to the sample anatomical plane and the output probability of the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane; the generator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, and the difference between the two-dimensional CT image and the two-dimensional MRI image in the sample image combination.

[0050] The present invention provides a 3D modeling method and device based on multi-source heterogeneous data, which obtains a combination of images corresponding to multiple anatomical planes of a target part; wherein the image combination corresponding to any anatomical plane at least includes a registered two-dimensional CT image and a two-dimensional MRI image corresponding to the anatomical plane; for any anatomical plane, the image combination corresponding to the anatomical plane is fused based on a multi-source heterogeneous fusion model to obtain a fused image of the anatomical plane; the fused images of the multiple anatomical planes are 3D reconstructed to obtain a 3D model of the target part; wherein the multi-source heterogeneous fusion model is constructed based on a generative adversarial network; a discriminator of the multi-source heterogeneous fusion model is based on the input of the discriminator to a sample image combination corresponding to a sample anatomical plane The generator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, the output probability of the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, and the difference between the two-dimensional CT image and the two-dimensional MRI image in the sample image combination, so that the generator can be used to generate a fused image that integrates and retains the image information in the two-dimensional CT image and the two-dimensional MRI image, and then generate a three-dimensional model with more complete feature information and better visual effect according to the fused images of each anatomical plane. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 It is a flow chart of a three-dimensional modeling method based on multi-source heterogeneous data provided by the present invention;

[0053] Figure 2 It is a schematic diagram of the structure of the multi-source heterogeneous fusion model provided by the present invention;

[0054] Figure 3 It is one of the flow charts of the generator training method provided by the present invention;

[0055] Figure 4 This is the second flow chart of the generator training method provided by the present invention;

[0056] Figure 5 It is a structural block diagram of a three-dimensional modeling device based on multi-source heterogeneous data provided by the present invention;

[0057] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The specific implementation process of the three-dimensional modeling method and device based on multi-source heterogeneous data in an embodiment of the present invention is described below.

[0060] Figure 1 A flow chart of a 3D modeling method based on multi-source heterogeneous data provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method may include:

[0061] Step 110, obtaining a combination of images corresponding to multiple anatomical planes of the target part; wherein the combination of images corresponding to any anatomical plane at least includes a registered two-dimensional CT image and a two-dimensional MRI image corresponding to any anatomical plane;

[0062] Step 120, for any anatomical plane, fuse the image combination corresponding to the any anatomical plane based on the multi-source heterogeneous fusion model to obtain a fused image of the any anatomical plane;

[0063] Step 130, performing three-dimensional reconstruction on the fused images of multiple anatomical planes to obtain a three-dimensional model of the target part;

[0064] Among them, the multi-source heterogeneous fusion model is constructed based on a generative adversarial network; the discriminator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the sample image combination corresponding to the sample anatomical plane and the output probability of the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane; the generator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, and the difference between the two-dimensional CT image and the two-dimensional MRI image in the sample image combination.

[0065] Specifically, the target part is a certain part of the human body of the object to be detected, such as the brain, chest or abdomen. Using medical imaging equipment, such as CT imaging equipment and MRI imaging equipment, the CT slice sequence and MRI slice sequence of the target part are obtained. It is worth noting that when acquiring the CT slice sequence and MRI slice sequence of the target part, the posture and scanning range of the object to be detected need to be kept unchanged, and the scanning parameters (such as slice spacing and scanning direction, etc.) of the CT imaging equipment and the MRI imaging equipment are set to the same to obtain the CT slices and MRI slices corresponding to the same anatomical plane. Subsequently, based on a registration algorithm, such as a rigid registration algorithm, the CT slices and MRI slices corresponding to the same anatomical plane can be registered to obtain the registered two-dimensional CT image and two-dimensional MRI image corresponding to the same anatomical plane, thereby combining them into an image combination corresponding to the same anatomical plane. After obtaining the image combinations corresponding to multiple anatomical planes of the target part, the image combinations corresponding to each anatomical plane can be sorted according to the scanning order of the anatomical plane.

[0066] For any anatomical plane, the image combination corresponding to the anatomical plane can be fused based on the multi-source heterogeneous fusion model to obtain a fused image of the anatomical plane. The multi-source heterogeneous fusion model can be constructed based on a generative adversarial network, so the multi-source heterogeneous fusion model contains a generator and a discriminator, and its structure is as follows: Figure 2 As shown. It should be noted that in the model training stage of the multi-source heterogeneous fusion model, the generator and the discriminator will be trained alternately in a loop, but in the model inference stage, the discriminator of the multi-source heterogeneous fusion model can be removed. Here, the two-dimensional CT image and the two-dimensional MRI image in the image combination corresponding to the anatomical plane can be spliced, and the spliced ​​image can be input into the generator in the multi-source heterogeneous fusion model so that the generator can generate a fused image that retains the image information of the active image (i.e., the two-dimensional CT image and the two-dimensional MRI image). After obtaining the fused images of each anatomical plane, the fused images of each anatomical plane can be three-dimensionally reconstructed according to the scanning order of the anatomical plane based on a three-dimensional reconstruction tool (such as 3D Slicer, VisualSFM, etc.) to obtain a three-dimensional model of the target part.

[0067] In order to ensure that the multi-source heterogeneous fusion model can generate a fused image that integrates the image information of two-dimensional CT images and two-dimensional MRI images, it is necessary to design the training process of the multi-source heterogeneous fusion model in a targeted manner to improve the model performance in the multi-source heterogeneous image fusion scenario.

[0068] Specifically, when training a multi-source heterogeneous fusion model, the generator and the discriminator can be trained alternately in a loop. The training data used in the model training process includes a sample image combination corresponding to multiple sample anatomical planes of a sample part (the same part of a different object than the target part). It is worth noting that the sample image combination corresponding to any sample anatomical plane includes a registered two-dimensional CT image and a two-dimensional MRI image corresponding to the sample anatomical plane. In an optional embodiment of the present application, a CT slice sequence and an MRI slice sequence of a sample part can be obtained from a public database, and a sample image combination corresponding to each sample anatomical plane of the sample part can be obtained in a manner similar to the image combination corresponding to the multiple anatomical planes of the target part described above.

[0069] In the training process of the discriminator of the multi-source heterogeneous fusion model, the model loss of the discriminator can be calculated based on the output probability of the discriminator for the sample image combination corresponding to the sample anatomical plane and the output probability of the fusion image output by the generator based on the sample image combination corresponding to the sample anatomical plane, and the parameters of the discriminator can be adjusted based on the model loss of the discriminator using a model training method such as gradient descent. Among them, the output probability output by the discriminator represents the probability that the discriminator considers the input image to be a real image.

[0070] In an optional embodiment of the present application, the two-dimensional CT image and the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane can be respectively input into the discriminator to obtain the output probability corresponding to the two-dimensional CT image and the two-dimensional MRI image in the sample image combination output by the discriminator. In addition, the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane is also input into the discriminator to obtain the output probability corresponding to the fused image of the sample anatomical plane output by the discriminator. Among them, the greater the output probability output by the discriminator, the greater the probability that the discriminator believes that the corresponding input image is a real image, so for the two-dimensional CT image and the two-dimensional MRI image in the sample image combination, the higher the output probability, the more accurate the discriminator is, and for the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, the lower the output probability, the more accurate the discriminator is. Therefore, in order to train the discriminator, the discriminant loss can be determined based on the first difference between the value 1 and the output probability of the two-dimensional CT image in the sample image combination, the second difference between the value 1 and the output probability of the two-dimensional MRI image in the sample image combination, and the output probability of the fused image of the sample anatomical plane, so as to adjust the parameters of the discriminator based on the discriminant loss. Here, the discrimination loss can be obtained by weighted summation based on the first difference, the second difference and the weight corresponding to the output probability of the fused image. It should be noted that in the early stage of training, due to the poor performance of the generator, the fused image output by it is easier to be recognized, so the weight corresponding to the output probability of the fused image can be lower, and the weight corresponding to the output probability of the fused image is gradually increased as the training progresses.

[0071] During the training process of the generator of the multi-source heterogeneous fusion model, in order to make the fused image output by the generator retain the image information in the source image and be sufficiently close to the real medical image, the model loss of the generator can be calculated based on the output probability of the fused image output by the discriminator based on the sample image combination corresponding to the sample anatomical plane, and the difference between the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane and the two-dimensional CT image and the two-dimensional MRI image in the sample image combination, and the parameters of the generator can be adjusted based on the model loss of the generator using model training methods such as gradient descent.

[0072] In an optional embodiment of the present application, Figure 3 As shown, the generator of the multi-source heterogeneous fusion model can be trained in the following way:

[0073] Step 310, inputting the sample image combination corresponding to the sample anatomical plane into the generator to obtain a fused image of the sample anatomical plane output by the generator;

[0074] Step 320, inputting the fused image of the sample anatomical plane into the discriminator, and obtaining an output probability corresponding to the fused image of the sample anatomical plane output by the discriminator;

[0075] Step 330, inputting the fused image of the sample anatomical plane and the sample image combination corresponding to the sample anatomical plane into the image segmentation model respectively, to obtain the contour of the corresponding image output by the image segmentation model;

[0076] Step 340, determining a generation loss based on a difference between the value 1 and the output probability corresponding to the fused image of the sample anatomical plane;

[0077] Step 350, determining a contour loss based on a difference between a contour of the fused image of the sample anatomical plane and a contour of a two-dimensional CT image and a contour of a two-dimensional MRI image in a sample image combination corresponding to the sample anatomical plane;

[0078] Step 360, determining a structural loss based on a structural similarity between the fused image of the sample anatomical plane and a two-dimensional MRI image in a combination of sample images corresponding to the sample anatomical plane;

[0079] Step 370, determining the perceptual loss based on the image feature difference between the fused image of the sample anatomical plane and the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane;

[0080] Step 380: Adjust the parameters of the generator based on the generation loss, the contour loss, the structure loss and the perceptual loss.

[0081] Specifically, the sample image combination corresponding to the sample anatomical plane can be input into the generator to obtain a fused image of the sample anatomical plane. Considering that the two-dimensional MRI image contains more structural details of soft tissue, the structural loss can be determined based on the structural similarity between the fused image of the sample anatomical plane and the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane to characterize the degree of retention of the structural features of the source image by the fused image output by the generator, and guide the generator to generate a fused image with more complete structural details. Among them, in an optional embodiment of the present application, the structural similarity between the fused image of the sample anatomical plane and the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane can be calculated based on the SSIM index, and the difference between the value 1 and the structural similarity can be determined as the above-mentioned structural loss. It is worth noting that the SSIM index can be used to calculate the structural similarity in the embodiments of the present application. Considering that the spatial resolution of the two-dimensional CT image is higher, the perceptual loss can be determined based on the image feature difference between the fused image of the sample anatomical plane and the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane to characterize the degree of retention of the image semantics of the fused image output by the generator for the source image, and guide the generator to retain more image semantics including texture information when generating a fused image. In an optional embodiment of the present application, the image features of the fused image of the sample anatomical plane and the image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane can be extracted respectively based on a trained image feature extraction model (e.g., VGG16), so as to determine the above-mentioned perceptual loss based on the image feature difference between the image features of the fused image of the sample anatomical plane and the image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane. Here, the image feature difference between the image features of the fused image of the sample anatomical plane and the image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane can be determined based on the Euclidean distance between the image features of the fused image of the sample anatomical plane and the image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane.

[0082] Subsequently, the fused image of the sample anatomical plane and the sample image combination corresponding to the sample anatomical plane can be respectively input into the image segmentation model to obtain the contour of the corresponding image, and the contour difference between the fused image of the sample anatomical plane and the two-dimensional CT image and the two-dimensional MRI image corresponding to the sample anatomical plane is used to determine the contour loss to guide the generator to generate a fused image that is more similar to the contour of the source image. In an optional embodiment of the present application, considering that the organ contours of adjacent anatomical planes are similar, in order to further improve the visual effect of the subsequently generated three-dimensional model, when determining the contour loss, the contour of the fused image of the adjacent sample anatomical plane can be determined, and the first difference between the contour of the fused image of the adjacent sample anatomical plane and the contour of the fused image of the sample anatomical plane is determined, and the second difference and the third difference between the contour of the fused image of the sample anatomical plane and the contour of the two-dimensional CT image and the contour of the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane are determined, respectively, so as to determine the contour loss based on the above-mentioned first difference, second difference and third difference, so as to guide the generator to generate a fused image with a similar contour for two adjacent anatomical planes, so as to facilitate the smoothness of the model boundary when the fused images of adjacent anatomical planes are superimposed to generate a three-dimensional model.

[0083] In addition, the fused image of the sample anatomical plane can also be input into the discriminator to obtain the output probability corresponding to the fused image of the sample anatomical plane. Since the output probability output by the discriminator represents the probability that the discriminator considers the input image to be a real image, if the input image is sufficiently real, the output probability output by the discriminator will be close to the value 1. Therefore, based on the difference between the value 1 and the output probability corresponding to the fused image of the sample anatomical plane, the generation loss can be determined to guide the generator to generate sufficiently real images.

[0084] In summary, the parameters of the generator can be adjusted based on the above generation loss, contour loss, structure loss, and perceptual loss.

[0085] It is worth noting that the embodiment of the present application does not impose any limitation on the execution order between step 320 and step 330, and does not impose any limitation on the execution order of steps 340-370.

[0086] In an optional embodiment of the present application, considering that the fused image output by the generator needs to be used in the subsequent 3D reconstruction process, and the 3D reconstruction uses a superposition or volume rendering algorithm to stack the 2D images into a 3D model, the 3D reconstruction performance can be improved by improving the similarity between the fused images of adjacent anatomical planes to obtain a 3D model with better visual effects. Specifically, in order to achieve this purpose, Figure 4 As shown, the generator can also be trained as follows:

[0087] Step 410, determining MRI structural differences based on structural similarities between the two-dimensional MRI images in the sample image combination corresponding to the adjacent sample anatomical planes and the two-dimensional MRI images in the sample image combination corresponding to the sample anatomical planes;

[0088] Step 420, determining CT structural differences based on structural similarities between the two-dimensional CT images in the sample image combination corresponding to the adjacent sample anatomical planes and the two-dimensional CT images in the sample image combination corresponding to the sample anatomical planes;

[0089] Step 430, determining a fused structural difference based on a structural similarity between a fused image of an adjacent sample anatomical plane and the fused image of the sample anatomical plane;

[0090] Step 440, determining a reconstruction loss based on a difference between an average value of the MRI structure difference and the CT structure difference and the fusion structure difference;

[0091] Step 450: Adjust the parameters of the generator based on the generation loss, the contour loss, the structure loss, the perceptual loss and the reconstruction loss.

[0092] Here, the MRI structural difference can be determined based on the structural similarity between the two-dimensional MRI image in the sample image combination corresponding to the adjacent sample anatomical plane (i.e., the previous sample anatomical plane or the next sample anatomical plane of the sample anatomical plane according to the scanning direction) and the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane. For example, the structural similarity between the two two-dimensional MRI images can be subtracted from the value 1 as the MRI structural difference. Similarly, the CT structural difference can be determined based on the structural similarity between the two-dimensional CT image in the sample image combination corresponding to the adjacent sample anatomical plane and the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane, and the fused structural difference can be determined based on the structural similarity between the fused image of the adjacent sample anatomical plane and the fused image of the sample anatomical plane. In order to improve the performance of subsequent three-dimensional reconstruction, if the two-dimensional MRI images corresponding to the two adjacent anatomical planes are similar and the two-dimensional CT images corresponding to the two adjacent anatomical planes are similar, then the fused images of the two adjacent anatomical planes should also be as similar as possible. Therefore, the reconstruction loss can be determined based on the average value of the MRI structural difference and the CT structural difference and the difference between the above-mentioned fused structural difference to guide the generator to generate similar fused images for two adjacent and similar anatomical planes. Subsequently, the above generation loss, contour loss, structure loss, perception loss and reconstruction loss may be combined to adjust the parameters of the generator. It should be noted that the embodiment of the present application does not impose any limitation on the execution order of steps 310-330.

[0093] In summary, the solution provided by the present invention is to obtain a combination of images corresponding to multiple anatomical planes of the target part; wherein the image combination corresponding to any anatomical plane at least includes a registered two-dimensional CT image and a two-dimensional MRI image corresponding to the anatomical plane; for any anatomical plane, the image combination corresponding to the anatomical plane is fused based on a multi-source heterogeneous fusion model to obtain a fused image of the anatomical plane; the fused images of the multiple anatomical planes are three-dimensionally reconstructed to obtain a three-dimensional model of the target part; wherein the multi-source heterogeneous fusion model is constructed based on a generative adversarial network; the discriminator of the multi-source heterogeneous fusion model is based on the output probability of the discriminator for the sample image combination corresponding to the sample anatomical plane and the output probability of the discriminator for the sample image combination corresponding to the sample anatomical plane The generator is trained based on the output probability of the fused image output by the combination of sample images corresponding to the sample anatomical planes; the generator of the multi-source heterogeneous fusion model is trained based on the output probability of the fused image output by the discriminator based on the sample image combination corresponding to the sample anatomical planes, and the difference between the fused image output by the generator based on the sample image combination corresponding to the sample anatomical planes and the two-dimensional CT image and the two-dimensional MRI image in the sample image combination, so that the generator can be used to generate a fused image that integrates and retains the image information in the two-dimensional CT image and the two-dimensional MRI image, and then generate a three-dimensional model with more complete feature information and better visual effect according to the fused images of each anatomical plane.

[0094] Figure 5 A structural block diagram of a 3D modeling device based on multi-source heterogeneous data provided by an embodiment of the present invention, such as Figure 5 As shown, the device comprises:

[0095] An image acquisition unit 201 is used to obtain an image combination corresponding to multiple anatomical planes of a target part; wherein the image combination corresponding to any anatomical plane at least includes a registered two-dimensional CT image and a two-dimensional MRI image corresponding to any anatomical plane;

[0096] A two-dimensional image fusion unit 202 is used for fusing the image combination corresponding to any anatomical plane based on a multi-source heterogeneous fusion model to obtain a fused image of any anatomical plane;

[0097] A three-dimensional modeling unit 203 is used to perform three-dimensional reconstruction on the fused images of multiple anatomical planes to obtain a three-dimensional model of the target part;

[0098] Among them, the multi-source heterogeneous fusion model is constructed based on a generative adversarial network; the discriminator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the sample image combination corresponding to the sample anatomical plane and the output probability of the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane; the generator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, and the difference between the two-dimensional CT image and the two-dimensional MRI image in the sample image combination.

[0099] The solution provided by the present invention is to obtain a combination of images corresponding to multiple anatomical planes of a target part; wherein the image combination corresponding to any anatomical plane includes at least a registered two-dimensional CT image and a two-dimensional MRI image corresponding to the anatomical plane; for any anatomical plane, the image combination corresponding to the anatomical plane is fused based on a multi-source heterogeneous fusion model to obtain a fused image of the anatomical plane; the fused images of the multiple anatomical planes are three-dimensionally reconstructed to obtain a three-dimensional model of the target part; wherein the multi-source heterogeneous fusion model is constructed based on a generative adversarial network; the discriminator of the multi-source heterogeneous fusion model is based on the output probability of the discriminator for the sample image combination corresponding to the sample anatomical plane and the output probability of the discriminator for the sample image combination corresponding to the generative adversarial network. The generator of the multi-source heterogeneous fusion model is trained based on the output probability of the fused image output by the discriminator based on the sample image combination corresponding to the sample anatomical plane, and the difference between the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane and the two-dimensional CT image and the two-dimensional MRI image in the sample image combination. Therefore, the generator can be used to generate a fused image that integrates and retains the image information in the two-dimensional CT image and the two-dimensional MRI image, and then generate a three-dimensional model with more complete feature information and better visual effect according to the fused images of each anatomical plane.

[0100] In an optional embodiment of the present application, the generator of the multi-source heterogeneous fusion model is trained based on the following method:

[0101] Inputting the sample image combination corresponding to the sample anatomical plane into the generator to obtain a fused image of the sample anatomical plane output by the generator;

[0102] Inputting the fused image of the sample anatomical plane into the discriminator, and obtaining an output probability corresponding to the fused image of the sample anatomical plane output by the discriminator;

[0103] Inputting the fused image of the sample anatomical plane and the sample image combination corresponding to the sample anatomical plane into the image segmentation model respectively, to obtain the contour of the corresponding image output by the image segmentation model;

[0104] Determining a generation loss based on a difference between a value of 1 and an output probability corresponding to a fused image of the sample anatomical plane;

[0105] Determine a contour loss based on a difference between a contour of the fused image of the sample anatomical plane and a contour of a two-dimensional CT image and a contour of a two-dimensional MRI image in a combination of sample images corresponding to the sample anatomical plane;

[0106] Determining a structural loss based on a structural similarity between a fused image of the sample anatomical plane and a two-dimensional MRI image in a combination of sample images corresponding to the sample anatomical plane;

[0107] determining a perceptual loss based on a difference in image features between a fused image of the sample anatomical plane and a two-dimensional CT image in a combination of sample images corresponding to the sample anatomical plane;

[0108] Based on the generation loss, the contour loss, the structure loss and the perceptual loss, the parameters of the generator are adjusted.

[0109] In an optional embodiment of the present application, the training process of the generator further includes:

[0110] Determining MRI structural differences based on structural similarities between two-dimensional MRI images in a sample image combination corresponding to adjacent sample anatomical planes and two-dimensional MRI images in a sample image combination corresponding to the sample anatomical planes;

[0111] Determining CT structural differences based on structural similarities between a two-dimensional CT image in a sample image combination corresponding to adjacent sample anatomical planes and a two-dimensional CT image in a sample image combination corresponding to the sample anatomical planes;

[0112] determining a fused structural difference based on a structural similarity between a fused image of an adjacent sample anatomical plane and a fused image of the sample anatomical plane;

[0113] determining a reconstruction loss based on a difference between an average of the MRI structural difference and the CT structural difference and the fused structural difference;

[0114] Parameters of the generator are adjusted based on the generation loss, the contour loss, the structure loss, the perceptual loss and the reconstruction loss.

[0115] In an optional embodiment of the present application, determining the contour loss based on the difference between the contour of the fused image of the sample anatomical plane and the contour of the two-dimensional CT image and the contour of the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane includes:

[0116] determining contours of the fused images of adjacent sample anatomical planes;

[0117] determining a first difference between a contour of the fused image of the adjacent sample anatomical plane and a contour of the fused image of the sample anatomical plane;

[0118] Determine a second difference and a third difference between the contour of the fused image of the sample anatomical plane and the contour of the two-dimensional CT image and the contour of the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane, respectively;

[0119] Based on the first difference, the second difference and the third difference, the contour loss is determined.

[0120] In an optional embodiment of the present application, determining the structural loss based on the structural similarity between the fused image of the sample anatomical plane and the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane includes:

[0121] Calculate the structural similarity between the fused image of the sample anatomical plane and the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane based on the SSIM index;

[0122] The difference between the value 1 and the structural similarity is determined as the structural loss.

[0123] In an optional embodiment of the present application, determining the perceptual loss based on the image feature difference between the fused image of the sample anatomical plane and the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane includes:

[0124] Based on the image feature extraction model, image features of the fused image of the sample anatomical plane and image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane are respectively extracted;

[0125] The perceptual loss is determined based on the difference in image features between the image features of the fused image of the sample anatomical plane and the image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane.

[0126] In an optional embodiment of the present application, the image feature difference between the image features of the fused image of the sample anatomical plane and the image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane is determined based on the Euclidean distance between the image features of the fused image of the sample anatomical plane and the image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane.

[0127] In an optional embodiment of the present application, the discriminator of the multi-source heterogeneous fusion model is trained based on the following method:

[0128] Inputting the two-dimensional CT image and the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane into the discriminator respectively, and obtaining the output probabilities corresponding to the two-dimensional CT image and the two-dimensional MRI image in the sample image combination output by the discriminator;

[0129] Inputting the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane into the discriminator, and obtaining the output probability corresponding to the fused image of the sample anatomical plane output by the discriminator;

[0130] Determine the discrimination loss based on a first difference between the value 1 and the output probability corresponding to the two-dimensional CT image in the sample image combination, a second difference between the value 1 and the output probability corresponding to the two-dimensional MRI image in the sample image combination, and the output probability corresponding to the fused image of the sample anatomical plane;

[0131] Parameters of the discriminator are adjusted based on the discriminative loss.

[0132] In an optional embodiment of the present application, the image combination corresponding to any anatomical plane is fused based on the multi-source heterogeneous fusion model to obtain a fused image of any anatomical plane, including:

[0133] splicing the two-dimensional CT image and the two-dimensional MRI image in the image combination corresponding to any anatomical plane to obtain a spliced ​​image;

[0134] The stitched image is input into the generator of the multi-source heterogeneous fusion model to obtain a fused image of any anatomical plane.

[0135] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor (processor) 301, a communication interface (Communications Interface) 302, a memory (memory) 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304. The processor 301 can call the logic instructions in the memory 303 to execute a three-dimensional modeling method based on multi-source heterogeneous data, the method comprising: obtaining an image combination corresponding to multiple anatomical planes of the target part; wherein the image combination corresponding to any anatomical plane at least includes a registered two-dimensional CT image and a two-dimensional MRI image corresponding to any anatomical plane; for any anatomical plane, based on a multi-source heterogeneous fusion model, the image combination corresponding to any anatomical plane is fused to obtain a fused image of any anatomical plane; the fused images of multiple anatomical planes are three-dimensionally reconstructed to obtain a three-dimensional model of the target part; wherein the multi-source heterogeneous fusion model is based on The multi-source heterogeneous fusion model is constructed by a generative adversarial network; the discriminator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the sample image combination corresponding to the sample anatomical plane and the output probability of the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane; the generator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, and the difference between the two-dimensional CT image and the two-dimensional MRI image in the sample image combination.

[0136] In addition, the logic instructions in the above-mentioned memory 303 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0137] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a three-dimensional modeling method based on multi-source heterogeneous data provided by the above methods, the method comprising: obtaining an image combination corresponding to multiple anatomical planes of the target part; wherein the image combination corresponding to any anatomical plane at least includes a registered two-dimensional CT image and a two-dimensional MRI image corresponding to any anatomical plane; for any anatomical plane, fusing the image combination corresponding to any anatomical plane based on a multi-source heterogeneous fusion model to obtain a fused image of any anatomical plane; and three-dimensionally transforming the fused images of multiple anatomical planes. Reconstruct to obtain a three-dimensional model of the target part; wherein the multi-source heterogeneous fusion model is constructed based on a generative adversarial network; the discriminator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the sample image combination corresponding to the sample anatomical plane and the output probability of the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane; the generator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, and the difference between the two-dimensional CT image and the two-dimensional MRI image in the sample image combination.

[0138] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute a three-dimensional modeling method based on multi-source heterogeneous data provided by the above-mentioned methods, the method comprising: obtaining an image combination corresponding to multiple anatomical planes of a target part; wherein the image combination corresponding to any anatomical plane includes at least a registered two-dimensional CT image and a two-dimensional MRI image corresponding to the any anatomical plane; for any anatomical plane, fusing the image combination corresponding to the any anatomical plane based on a multi-source heterogeneous fusion model to obtain a fused image of the any anatomical plane; and reconstructing the fused images of the multiple anatomical planes in three dimensions to obtain a three-dimensional model of the target part. type; wherein the multi-source heterogeneous fusion model is constructed based on a generative adversarial network; the discriminator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the sample image combination corresponding to the sample anatomical plane and the output probability of the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane; the generator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, and the difference between the two-dimensional CT image and the two-dimensional MRI image in the sample image combination.

[0139] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the natural gas pipeline multi-angle leakage concentration field and flame morphology test method described in each embodiment or some parts of the embodiment.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional modeling method based on multi-source heterogeneous data, characterized in that: include: Acquire a combination of images corresponding to multiple anatomical planes of the target part; wherein the combination of images corresponding to any anatomical plane at least includes a registered two-dimensional CT image and a two-dimensional MRI image corresponding to any anatomical plane; For any anatomical plane, the image combination corresponding to the any anatomical plane is fused based on the multi-source heterogeneous fusion model to obtain a fused image of the any anatomical plane; Performing three-dimensional reconstruction on the fused images of multiple anatomical planes to obtain a three-dimensional model of the target part; Wherein, the multi-source heterogeneous fusion model is constructed based on a generative adversarial network; the discriminator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the sample image combination corresponding to the sample anatomical plane and the output probability of the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane; the generator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, and the difference between the two-dimensional CT image and the two-dimensional MRI image in the sample image combination; The generator of the multi-source heterogeneous fusion model is trained based on the following method: Inputting the sample image combination corresponding to the sample anatomical plane into the generator to obtain a fused image of the sample anatomical plane output by the generator; Inputting the fused image of the sample anatomical plane into the discriminator, and obtaining an output probability corresponding to the fused image of the sample anatomical plane output by the discriminator; Inputting the fused image of the sample anatomical plane and the sample image combination corresponding to the sample anatomical plane into the image segmentation model respectively, to obtain the contour of the corresponding image output by the image segmentation model; Determining a generation loss based on a difference between a value of 1 and an output probability corresponding to a fused image of the sample anatomical plane; Determine a contour loss based on a difference between a contour of the fused image of the sample anatomical plane and a contour of a two-dimensional CT image and a contour of a two-dimensional MRI image in a combination of sample images corresponding to the sample anatomical plane; Determining a structural loss based on a structural similarity between a fused image of the sample anatomical plane and a two-dimensional MRI image in a combination of sample images corresponding to the sample anatomical plane; determining a perceptual loss based on a difference in image features between a fused image of the sample anatomical plane and a two-dimensional CT image in a combination of sample images corresponding to the sample anatomical plane; Determining MRI structural differences based on structural similarities between two-dimensional MRI images in a sample image combination corresponding to adjacent sample anatomical planes and two-dimensional MRI images in a sample image combination corresponding to the sample anatomical planes; Determining CT structural differences based on structural similarities between a two-dimensional CT image in a sample image combination corresponding to adjacent sample anatomical planes and a two-dimensional CT image in a sample image combination corresponding to the sample anatomical planes; determining a fused structural difference based on a structural similarity between a fused image of an adjacent sample anatomical plane and a fused image of the sample anatomical plane; determining a reconstruction loss based on a difference between an average of the MRI structural difference and the CT structural difference and the fused structural difference; Parameters of the generator are adjusted based on the generation loss, the contour loss, the structure loss, the perceptual loss and the reconstruction loss.

2. A three-dimensional modeling method based on multi-source heterogeneous data according to claim 1, characterized in that: Determining contour loss based on differences between contours of the fused image of the sample anatomical plane and contours of the two-dimensional CT image and the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane, including: determining contours of the fused images of adjacent sample anatomical planes; determining a first difference between a contour of the fused image of the adjacent sample anatomical plane and a contour of the fused image of the sample anatomical plane; Determine a second difference and a third difference between the contour of the fused image of the sample anatomical plane and the contour of the two-dimensional CT image and the contour of the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane, respectively; Based on the first difference, the second difference and the third difference, the contour loss is determined.

3. The three-dimensional modeling method based on multi-source heterogeneous data according to claim 1, characterized in that: Determining a structural loss based on a structural similarity between a fused image of the sample anatomical plane and a two-dimensional MRI image in a combination of sample images corresponding to the sample anatomical plane includes: Calculate the structural similarity between the fused image of the sample anatomical plane and the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane based on the SSIM index; The difference between the value 1 and the structural similarity is determined as the structural loss.

4. The three-dimensional modeling method based on multi-source heterogeneous data according to claim 1, characterized in that: Determining the perceptual loss based on the image feature difference between the fused image of the sample anatomical plane and the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane includes: Based on the image feature extraction model, image features of the fused image of the sample anatomical plane and image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane are respectively extracted; The perceptual loss is determined based on the difference in image features between the image features of the fused image of the sample anatomical plane and the image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane.

5. A three-dimensional modeling method based on multi-source heterogeneous data according to claim 4, characterized in that: The image feature difference between the image features of the fused image of the sample anatomical plane and the image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane is determined based on the Euclidean distance between the image features of the fused image of the sample anatomical plane and the image features of the two-dimensional CT image in the sample image combination corresponding to the sample anatomical plane.

6. The three-dimensional modeling method based on multi-source heterogeneous data according to claim 1, characterized in that: The discriminator of the multi-source heterogeneous fusion model is trained based on the following method: Inputting the two-dimensional CT image and the two-dimensional MRI image in the sample image combination corresponding to the sample anatomical plane into the discriminator respectively, and obtaining the output probabilities corresponding to the two-dimensional CT image and the two-dimensional MRI image in the sample image combination output by the discriminator; Inputting the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane into the discriminator, and obtaining the output probability corresponding to the fused image of the sample anatomical plane output by the discriminator; Determine the discrimination loss based on a first difference between the value 1 and the output probability corresponding to the two-dimensional CT image in the sample image combination, a second difference between the value 1 and the output probability corresponding to the two-dimensional MRI image in the sample image combination, and the output probability corresponding to the fused image of the sample anatomical plane; Parameters of the discriminator are adjusted based on the discriminative loss.

7. The three-dimensional modeling method based on multi-source heterogeneous data according to claim 1, characterized in that: The image combination corresponding to any anatomical plane is fused based on the multi-source heterogeneous fusion model to obtain a fused image of any anatomical plane, including: splicing the two-dimensional CT image and the two-dimensional MRI image in the image combination corresponding to any anatomical plane to obtain a spliced ​​image; The stitched image is input into the generator of the multi-source heterogeneous fusion model to obtain a fused image of any anatomical plane.

8. A three-dimensional modeling device based on multi-source heterogeneous data, characterized in that: The method for executing the three-dimensional modeling method based on multi-source heterogeneous data as claimed in claim 1 comprises: An image acquisition unit, used to obtain image combinations corresponding to multiple anatomical planes of the target part; wherein the image combination corresponding to any anatomical plane at least includes a registered two-dimensional CT image and a two-dimensional MRI image corresponding to any anatomical plane; A two-dimensional image fusion unit is used for fusing the image combination corresponding to any anatomical plane based on a multi-source heterogeneous fusion model to obtain a fused image of any anatomical plane; A three-dimensional modeling unit, used for performing three-dimensional reconstruction on the fused images of multiple anatomical planes to obtain a three-dimensional model of the target part; Among them, the multi-source heterogeneous fusion model is constructed based on a generative adversarial network; the discriminator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the sample image combination corresponding to the sample anatomical plane and the output probability of the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane; the generator of the multi-source heterogeneous fusion model is trained based on the output probability of the discriminator for the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, the fused image output by the generator based on the sample image combination corresponding to the sample anatomical plane, and the difference between the two-dimensional CT image and the two-dimensional MRI image in the sample image combination.

Images