Lung three-dimensional reconstruction method, device, terminal equipment and readable storage medium
By generating point clouds of two-dimensional lung images and performing three-dimensional reconstruction, the problem of not being able to directly provide a three-dimensional reconstruction model in existing technologies is solved, enabling rapid and accurate three-dimensional lung image diagnosis and improving diagnostic efficiency and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU MINGYI MEDICAL TECH CO LTD
- Filing Date
- 2021-11-17
- Publication Date
- 2026-06-02
AI Technical Summary
Existing medical tomographic image processing solutions can only acquire two-dimensional scan images and cannot directly provide medical three-dimensional reconstruction models. This forces doctors to rely on experience and spatial imagination to perform three-dimensional reconstruction, which limits diagnostic efficiency and accuracy.
By acquiring N two-dimensional images of the lungs, a generator is used to generate point clouds of tubular organs, arteries, veins, trachea, and lung segments. Connected component extraction and region growing are then performed, and rendering techniques are combined to achieve three-dimensional reconstruction of the lungs.
It provides inexperienced doctors with rapid and accurate 3D lung imaging diagnostics, reducing the influence of doctors' subjectivity and improving the efficiency and accuracy of lung lesion diagnosis.
Smart Images
Figure CN116137024B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, terminal device, and readable storage medium for three-dimensional lung reconstruction. Background Technology
[0002] With the development of science and technology and people's increasing emphasis on their health, medical auxiliary equipment such as computed tomography (CT), magnetic resonance imaging (MRI), B-mode ultrasound, and positron emission tomography (PET) have become indispensable tools in clinical medical diagnosis. CT, ultrasound, MRI, and PET imaging systems can only provide two-dimensional images of the scanned object. Doctors must reconstruct these tomographic images into three-dimensional objects in their minds based on their experience before making a medical diagnosis. This requires doctors to have extensive experience and strong spatial reasoning skills.
[0003] Existing medical tomographic image processing solutions can only acquire two-dimensional scan images and cannot directly provide medical three-dimensional reconstruction models. There is an urgent need for a solution that can provide accurate and individualized three-dimensional reconstruction of internal targets. Summary of the Invention
[0004] In view of the above problems, this application proposes a method, apparatus, terminal device and readable storage medium for three-dimensional reconstruction of the lungs, so as to improve the diagnostic efficiency of lung diseases by performing three-dimensional reconstruction of two-dimensional images of the lungs.
[0005] In a first aspect, embodiments of this application propose a method for three-dimensional reconstruction of the lungs, the method comprising:
[0006] Obtain N two-dimensional images, including those of the lungs;
[0007] A first generator is used to generate a point cloud of the tubular organ based on the N two-dimensional images;
[0008] The second generator is used to generate arterial point clouds, vein point clouds, and tracheal point clouds based on the N two-dimensional images;
[0009] A third generator is used to generate lung segment point clouds based on the N two-dimensional images;
[0010] The lungs are reconstructed in three dimensions using the overall point cloud of the tubular organs, the point cloud of the arteries, the point cloud of the veins, the point cloud of the trachea, and the point cloud of the lung segments.
[0011] The lung three-dimensional reconstruction method described in this application, which utilizes the overall point cloud of the tubular organ, the point cloud of the artery, the point cloud of the vein, the point cloud of the trachea, and the point cloud of the lung segments to three-dimensionally reconstruct the lung, includes:
[0012] A first connected domain is extracted from the overall point cloud of the tubular organ, a second connected domain is extracted from the point cloud of the artery, a third connected domain is extracted from the point cloud of the vein, and a fourth connected domain is extracted from the point cloud of the trachea.
[0013] Region growing is performed based on the first connected region, the second connected region, the third connected region, and the fourth connected region to reconstruct arteries, veins, and trachea in three dimensions;
[0014] The lung segment point cloud is filtered to reconstruct the lung segment in three dimensions.
[0015] The lung three-dimensional reconstruction method described in this application further includes pre-training the first generator, wherein the pre-training of the first generator includes:
[0016] N two-dimensional training images are input into the first generator so that the first generator outputs a first target mask image. The first target mask image is a mask image of the whole tubular organ corresponding to the middle two-dimensional training image located in the middle position among the N two-dimensional training images.
[0017] The first generation loss value is calculated based on the first target mask image and the first mask annotation image corresponding to the intermediate two-dimensional training image;
[0018] N first mask annotation images are input into the first discriminator corresponding to the first generator, so that the first discriminator calculates the first connectivity loss value based on the input N first mask annotation images, and the N first mask annotation images correspond one-to-one with the N two-dimensional training images;
[0019] The first mask annotation image located in the middle of the N first mask annotation images is replaced with the first target mask image, so that the first discriminator calculates the first disconnection loss value based on the input N-1 first mask annotation images and the first target mask image;
[0020] The weights of the first discriminator are updated based on the first connectivity loss value and the first disconnection loss value.
[0021] The first discriminator, with updated weights, calculates the second connectivity loss value based on the input N-1 first mask annotation images and the first target mask image;
[0022] The weights of the first generator are updated based on the second connectivity loss value and the first generation loss value.
[0023] The lung three-dimensional reconstruction method described in this application further includes pre-training the second generator, wherein the pre-training of the second generator includes:
[0024] N two-dimensional training images are input into the second generator so that the second generator outputs a second target mask image. The second target mask image is a mask image of the middle two-dimensional training image located in the middle position among the N two-dimensional training images, including arteries, veins and trachea.
[0025] The second generation loss value is calculated based on the second target mask image and the second mask annotation image corresponding to the intermediate two-dimensional training image;
[0026] The N second mask annotation images are input into the second discriminator corresponding to the second generator, so that the second discriminator calculates the third connectivity loss value based on the input N second mask annotation images, and the N second mask annotation images correspond one-to-one with the N two-dimensional training images;
[0027] The second mask annotation image located in the middle of the N second mask annotation images is replaced with the second target mask image, so that the second discriminator calculates the second disconnection loss value based on the input N-1 second mask annotation images and the second target mask image;
[0028] The weights of the second discriminator are updated based on the third connectivity loss value and the second disconnection loss value.
[0029] The second discriminator, with updated weights, calculates the fourth connectivity loss value based on the input N-1 second mask annotation images and the second target mask image;
[0030] The weights of the second generator are updated based on the fourth connectivity loss value and the second generation loss value.
[0031] The lung three-dimensional reconstruction method described in this application further includes using the trained second generator as the third generator and pre-training the third generator.
[0032] The lung three-dimensional reconstruction method described in this application embodiment, wherein the pre-training of the third generator includes:
[0033] N two-dimensional training images are input into the third generator so that the third generator outputs a third target mask image, which is a mask image including the lung segment corresponding to the middle two-dimensional training image located in the middle position among the N two-dimensional training images.
[0034] The third generation loss value is calculated based on the third target mask image and the third mask annotation image corresponding to the intermediate two-dimensional training image;
[0035] The N third mask annotation images are input into the third discriminator corresponding to the third generator, so that the third discriminator calculates the fifth connectivity loss value based on the input N third mask annotation images. The N third mask annotation images correspond one-to-one with the N two-dimensional training images.
[0036] The third mask annotation image located in the middle of the N third mask annotation images is replaced with the third target mask image, so that the third discriminator calculates the third disconnection loss value based on the input N-1 third mask annotation images and the third target mask image;
[0037] The weights of the third discriminator are updated based on the fifth connectivity loss value and the third disconnection loss value.
[0038] The third discriminator, with updated weights, calculates the sixth connectivity loss value based on the input N-1 third mask annotation maps and the third target mask map;
[0039] The weights of the third generator are updated based on the sixth connectivity loss value and the third generation loss value.
[0040] The lung three-dimensional reconstruction method described in this application embodiment further includes: rendering the arteries, veins, trachea and lung segments of the three-dimensional reconstructed lung.
[0041] Secondly, embodiments of this application also propose a three-dimensional lung reconstruction device, the device comprising:
[0042] The image acquisition module is used to acquire N two-dimensional images, including those of the lungs.
[0043] The first generation module is used to generate an overall point cloud of the tubular organ based on the N two-dimensional images using a first generator.
[0044] The second generation module is used to generate arterial point clouds, vein point clouds and tracheal point clouds based on the N two-dimensional images using the second generator.
[0045] The third generation module is used to generate lung segment points based on the N two-dimensional images using a third generator.
[0046] A three-dimensional reconstruction module is used to reconstruct the lungs in three dimensions using the overall point cloud of the tubular organ, the point cloud of the artery, the point cloud of the vein, the point cloud of the trachea, and the point cloud of the lung segments.
[0047] Thirdly, an embodiment of this application provides a terminal device, including a memory and a processor. The memory stores a computer program, and the computer program executes the lung three-dimensional reconstruction method described in the embodiment of this application when it is run on the processor.
[0048] Fourthly, an embodiment of this application provides a readable storage medium storing a computer program that, when run on a processor, executes the lung three-dimensional reconstruction method described in the embodiment of this application.
[0049] This application acquires N two-dimensional images of the lungs; uses a first generator to generate a point cloud of the entire tubular organ based on the N two-dimensional images; uses a second generator to generate point clouds of arteries, veins, and trachea based on the N two-dimensional images; uses a third generator to generate point clouds of lung segments based on the N two-dimensional images; and uses the point clouds of the entire tubular organ, arteries, veins, trachea, and lung segments to reconstruct the lungs in three dimensions. This application can quickly provide inexperienced doctors with three-dimensional images of the lungs as a diagnostic basis, effectively avoiding the influence of the doctor's subjectivity in the diagnosis of lung lesions, and improving the efficiency and accuracy of lung lesion diagnosis. Attached Figure Description
[0050] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope of protection of the present invention. In the various drawings, similar components are numbered similarly.
[0051] Figure 1 A flowchart illustrating a three-dimensional lung reconstruction method according to an embodiment of this application is shown;
[0052] Figure 2 This paper presents a schematic diagram of the overall structure of a three-dimensional reconstructed tubular organ according to an embodiment of this application.
[0053] Figure 3 This illustration shows a schematic diagram of a lung segment generated by a third generator according to an embodiment of this application;
[0054] Figure 4 This illustration shows a schematic diagram of a three-dimensionally reconstructed lung segment according to an embodiment of this application;
[0055] Figure 5 A flowchart illustrating a first generator training process according to an embodiment of this application is shown;
[0056] Figure 6 A flowchart illustrating a second generator training process according to an embodiment of this application is shown;
[0057] Figure 7 A flowchart illustrating a third generator training process according to an embodiment of this application is shown.
[0058] Figure 8A schematic diagram of the structure of a three-dimensional lung reconstruction device according to an embodiment of this application is shown. Detailed Implementation
[0059] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0060] The components of the embodiments of the invention described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0061] In the following, the terms “comprising,” “having,” and their cognates, which may be used in various embodiments of the invention, are intended only to indicate a particular feature, number, step, operation, component, assembly, or combination thereof, and should not be construed as excluding, firstly, the presence of one or more other features, numbers, steps, operations, components, assemblies, or combinations thereof, or adding the possibility of one or more features, numbers, steps, operations, components, assemblies, or combinations thereof.
[0062] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.
[0063] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.
[0064] Example 1
[0065] Please see Figure 1 The first embodiment of this application proposes a method for three-dimensional reconstruction of the lungs, the method comprising:
[0066] Step S100: Obtain N two-dimensional images, including those of the lungs.
[0067] As is understandable, the lungs include veins, arteries, trachea, and 18 lung segments.
[0068] It should be noted that N two-dimensional images are a continuous sequence of two-dimensional images acquired by medical auxiliary equipment such as computed tomography (CT), magnetic resonance imaging (MRI), B-mode ultrasound imaging system (B-mode ultrasound), and positron emission tomography (PET).
[0069] The value of N should be greater than or equal to three. However, the value of N should not be too large, as this may lead to a large amount of calculation and a complex calculation process. Preferably, the value of N should be equal to three.
[0070] Step S200: Use the first generator to generate a point cloud of the tubular organ based on N two-dimensional images.
[0071] It is understandable that tubular organs as a whole include arteries, veins, and trachea.
[0072] For example, the first generator can use a U-Net network, or other networks, depending on the needs. The trained first generator can generate a complete point cloud of a tubular organ based on N two-dimensional images.
[0073] It should be noted that the overall point cloud of tubular organs is different from the point cloud of lung segments. When extracting the overall point cloud of tubular organs from N two-dimensional images, arteries, veins and trachea in the two-dimensional images are all labeled as 1, and the rest (background and lung segments) are all labeled as 0.
[0074] Step S300: Use the second generator to generate arterial point clouds, vein point clouds, and tracheal point clouds based on N two-dimensional images.
[0075] For example, the second generator can use a U-Net network, or other networks, whichever is needed. The trained second generator can generate artery point clouds, vein point clouds, and trachea point clouds based on N two-dimensional images.
[0076] It should be noted that there are differences between arterial point clouds, vein point clouds, tracheal point clouds and lung segment point clouds. When extracting arterial point clouds, vein point clouds and tracheal point clouds from N two-dimensional images, the arteries, veins and trachea in the two-dimensional images are labeled as 1, 2 and 3 respectively, and the rest (background and lung segments) are labeled as 0.
[0077] Step S400: Use the third generator to generate lung segment point clouds based on N two-dimensional images.
[0078] For example, the third generator can use a U-Net network, or other networks, depending on the needs. A trained third generator can generate lung segment point clouds based on N two-dimensional images.
[0079] It should be noted that there are a total of 18 lung segments, which are labeled as 1 to 18 to distinguish each lung segment.
[0080] Step S500: The lung is reconstructed in three dimensions using the overall point cloud of the tubular organ, the point cloud of the artery, the point cloud of the vein, the point cloud of the trachea, and the point cloud of the lung segment.
[0081] A first connected domain can be extracted from the overall point cloud of the tubular organ, a second connected domain can be extracted from the point cloud of the artery, a third connected domain can be extracted from the point cloud of the vein, and a fourth connected domain can be extracted from the point cloud of the trachea. Based on the first connected domain, the second connected domain, the third connected domain, and the fourth connected domain, region growing processing is performed to reconstruct the artery, vein, and trachea in three dimensions.
[0082] For example, such as Figure 2 As shown, the three-dimensional reconstructed tubular organ consists of an arterial portion, a venous portion, and a tracheal portion.
[0083] For further details, please see Figure 3 This paper illustrates a third generator that generates lung segment point clouds based on N two-dimensional images. These lung segment point clouds contain some noise and require filtering to obtain the desired result. Figure 4 The image shown is a three-dimensional image of a standard lung segment.
[0084] It should be noted that in this embodiment, the first generator, the second generator, and the third generator are all pre-trained. The relevant parameters in the first generator, the relevant parameters in the second generator, and the relevant weights in the third generator are all different. Therefore, the first generator, the second generator, and the third generator are all different after training.
[0085] Furthermore, in this embodiment, the arteries, veins, trachea, and lung segments of the reconstructed lungs are rendered to obtain a high-quality three-dimensional image of the lungs.
[0086] This application embodiment acquires N two-dimensional images of the lungs; uses a first generator to generate a point cloud of the entire tubular organ based on the N two-dimensional images; uses a second generator to generate point clouds of arteries, veins, and trachea based on the N two-dimensional images; uses a third generator to generate point clouds of lung segments based on the N two-dimensional images; and uses the point clouds of the entire tubular organ, arteries, veins, trachea, and lung segments to reconstruct the arteries, veins, trachea, and 18 lung segments in three dimensions. This application embodiment can quickly provide inexperienced doctors with three-dimensional images of the lungs as a diagnostic basis, effectively avoiding the influence of the doctor's subjectivity in the diagnosis of lung lesions, and improving the efficiency and accuracy of lung lesion diagnosis.
[0087] Example 2
[0088] Please see Figure 5 A second embodiment of this application proposes a process for pre-training the first generator of Embodiment 1 described above, which includes the following steps:
[0089] Step S10: Input N two-dimensional training images into the first generator so that the first generator outputs a first target mask image. The first target mask image is a mask image of the entire tubular organ corresponding to the middle two-dimensional training image located in the middle position among the N two-dimensional training images.
[0090] The N two-dimensional training images are a continuous sequence of two-dimensional images. These N images need to be pre-labeled, with arteries, veins, and trachea labeled as 1, and the remaining parts (background and lung segments) labeled as 0, to obtain the first target mask image corresponding to the N two-dimensional training images. Preferably, N equals three.
[0091] It should be noted that after every three 2D training images are stitched together, the lung region is extracted from the 2D training image using the maximum bounding region algorithm, and then normalized and standardized. For example, the normalization parameters can be: mean = [0.5, 0.5, 0.5], std = [0.5, 0.5, 0.5]. The 2D training images and their corresponding first target mask images are then randomly subjected to data augmentation such as inversion, rotation, blurring, distortion, and Gaussian filtering to fit 2D images of different qualities.
[0092] Step S11: Calculate the first generation loss value based on the first target mask map and the first mask annotation map corresponding to the intermediate two-dimensional training image.
[0093] Step S12: Input N first mask annotation images into the first discriminator corresponding to the first generator, so that the first discriminator calculates the first connectivity loss value based on the input N first mask annotation images, wherein the N first mask annotation images correspond one-to-one with the N two-dimensional training images.
[0094] Step S13: Replace the middle first mask annotation image in the N first mask annotation images with the first target mask image, so that the first discriminator calculates the first disconnection loss value based on the input N-1 first mask annotation images and the first target mask image.
[0095] Step S14: Update the weights of the first discriminator based on the first connectivity loss value and the first disconnection loss value.
[0096] Step S15: Calculate the second connectivity loss value using the first discriminator with updated weights based on the input N-1 first mask annotation images and the first target mask image.
[0097] Step S16: Update the weights of the first generator based on the second connectivity loss value and the first generation loss value.
[0098] It is understandable that when the tubular organ in the first target mask image generated by the first generator exhibits overall discontinuity, the loss ratio of terminal small blood vessels will be much smaller than that of upper-level blood vessels, and the discontinuity phenomenon of terminal blood vessels will be more severe. Therefore, the ability of using only the first generation loss value of the first generator to evaluate the discontinuity phenomenon of terminal blood vessels is limited. However, if this problem is treated as a classification problem, it becomes much simpler. Therefore, in this embodiment, the weights of the first generator are updated based on the second connectivity loss value and the first generation loss value. That is, the weights of the first generator are updated based on the segmentation loss of the first generator itself and the connectivity loss determined by the first discriminator, making the purpose of the trained first generator clearer and improving the segmentation effect of the tubular organ as a whole.
[0099] It should be noted that the above steps S10 to S16 need to be executed multiple times until the number of executions of steps S10 to S16 exceeds the set number of executions, or until the segmentation loss of the first generator itself (first generation loss value) and the connectivity loss judged by the first discriminator (second connectivity loss value) are lower than the set loss value.
[0100] For example, when training the first generator, the initial learning rate of the first generator can be set to 0.0005, the initial learning rate of the first discriminator can be set to 0.0001, and the number of training iterations can be set to 200. The learning rate can be updated using a warm-up learning rate and cosine annealing; that is, in the first iteration, the learning rate is warmed up with 0.0001, and subsequent iterations are calculated according to the cosine annealing formula. The optimizer used in training can be the SGD optimizer.
[0101] Example 3
[0102] Please see Figure 6 The third embodiment of this application proposes a process for pre-training the second generator in embodiment 1 above, which includes the following steps:
[0103] Step S20: Input N two-dimensional training images into the second generator so that the second generator outputs a second target mask image. The second target mask image is a mask image of the middle two-dimensional training image located in the middle position among the N two-dimensional training images, including arteries, veins and trachea.
[0104] The N two-dimensional training images are a continuous sequence of two-dimensional images. These N images need to be pre-labeled, with arteries, veins, and trachea labeled as 1, 2, and 3 respectively, and the remaining parts (background and lung segments) labeled as 0, to obtain a second target mask image corresponding to the N two-dimensional training images. Preferably, N equals three.
[0105] It should be noted that after every three 2D training images are stitched together, the lung region is extracted from the 2D training image using the maximum bounding region algorithm, and then normalized and standardized. For example, the normalization parameters can be: mean = [0.5, 0.5, 0.5], std = [0.5, 0.5, 0.5]. The 2D training images and their corresponding second target mask images are then subjected to random data augmentation processes such as inversion, rotation, blurring, distortion, and Gaussian filtering to fit 2D images of different qualities.
[0106] Step S21: Calculate the second generation loss value based on the second target mask map and the second mask annotation map corresponding to the intermediate two-dimensional training image.
[0107] Step S22: Input N second mask annotation images into the second discriminator corresponding to the second generator, so that the second discriminator calculates the third connectivity loss value based on the input N second mask annotation images, wherein the N second mask annotation images correspond one-to-one with the N two-dimensional training images.
[0108] Step S23: Replace the middle second mask annotation image in the N second mask annotation images with the second target mask image, so that the second discriminator calculates the second disconnection loss value based on the input N-1 second mask annotation images and the second target mask image.
[0109] Step S24: Update the weights of the second discriminator based on the third connectivity loss value and the second disconnection loss value.
[0110] Step S25: Calculate the fourth connectivity loss value using the second discriminator with updated weights based on the input N-1 second mask annotation maps and the second target mask map.
[0111] Step S26: Update the weights of the second generator based on the fourth connectivity loss value and the second generation loss value.
[0112] It is understandable that when arteries, veins, and trachea show discontinuities in the second target mask image generated by the second generator, the loss proportion of terminal small arteries, veins, and trachea will be much smaller than that of superior arteries, veins, and trachea, and the discontinuity phenomenon in terminal arteries, veins, and trachea will be more severe. Therefore, the ability of evaluating the discontinuity phenomenon in terminal arteries, veins, and trachea solely using the second generation loss value of the second generator is limited. Therefore, in this embodiment, the weights of the second generator are updated based on the fourth connectivity loss value and the second generation loss value, that is, the weights of the second generator are updated by the segmentation loss of the second generator itself and the connectivity loss determined by the second discriminator, so that the purpose of the trained second generator will be clearer and the segmentation effect of arteries, veins, and trachea will be better.
[0113] It should be noted that the above steps S20 to S26 need to be executed multiple times until the number of executions of steps S20 to S26 exceeds the set number of executions, or until the segmentation loss of the second generator itself (second generation loss value) and the connectivity loss judged by the second discriminator (fourth connectivity loss value) are lower than the set loss value.
[0114] For example, when training the second generator, the initial learning rate of the second generator can be set to 0.0005, the initial learning rate of the second discriminator can be set to 0.0001, and the number of training iterations can be set to 200. The learning rate can be updated using a warm-up learning rate and cosine annealing; that is, in the first iteration, the learning rate is warmed up with 0.0001, and subsequent iterations are calculated according to the cosine annealing formula. The optimizer used in training can be the SGD optimizer.
[0115] It should be noted that in this embodiment, the formula for calculating the evaluation index used to evaluate the performance of the second generator is Dice(P1,T)=2*∣P1∩T∣ / (∣P1∣+∣T∣), where P1 is the result predicted by the second generator, and T is the actual result. Experiments show that if the second discriminator is not introduced during the training of the second generator, the evaluation index value of the second generator is 0.744; if the second discriminator is introduced during the training of the second generator, the evaluation index value of the second generator is 0.843.
[0116] Example 4
[0117] Please see Figure 7 The fourth embodiment of this application proposes a process for pre-training the third generator in Embodiment 1 above, which includes the following steps:
[0118] Step S30: Input N two-dimensional training images into the third generator so that the third generator outputs a third target mask image, which is a mask image including the lung segment corresponding to the middle two-dimensional training image located in the middle position among the N two-dimensional training images.
[0119] The N two-dimensional training images are a continuous sequence of two-dimensional images. These images need to be pre-labeled, with 18 lung segments labeled as 1 to 18, and the remaining parts (background and lung segments) labeled as 0, to obtain a third target mask image corresponding to the N two-dimensional training images. Preferably, N equals three.
[0120] It should be noted that in the above embodiment 3, the trained second generator already has the ability to segment arteries, veins and trachea. When segmenting lung segments, it is also necessary to segment arteries, veins and trachea from all lung segments. Therefore, in order to speed up the training process of the third generator and improve the training speed, in this embodiment, the trained second generator can be used as the third generator. This embodiment is to continue training the trained second generator to obtain a third generator with the function of segmenting 18 lung segments.
[0121] It should be noted that after every three 2D training images are stitched together, the lung region is extracted from the 2D training image using the maximum bounding region algorithm, and then normalized and standardized. For example, the normalization parameters can be: mean = [0.5, 0.5, 0.5], std = [0.5, 0.5, 0.5]. The 2D training images and their corresponding third target mask images are then subjected to random data augmentation processes such as inversion, rotation, blurring, distortion, and Gaussian filtering to fit 2D images of different qualities.
[0122] Step S31: Calculate the third generation loss value based on the third target mask map and the third mask annotation map corresponding to the intermediate two-dimensional training image.
[0123] Step S32: Input N third mask annotation images into the third discriminator corresponding to the third generator, so that the third discriminator calculates the fifth connectivity loss value based on the input N third mask annotation images, and the N third mask annotation images correspond one-to-one with the N two-dimensional training images.
[0124] Step S33: Replace the middle third mask annotation image in the N third mask annotation images with the third target mask image, so that the third discriminator can calculate the third disconnection loss value based on the input N-1 third mask annotation images and the third target mask image.
[0125] Step S34: Update the weights of the third discriminator based on the fifth connectivity loss value and the third disconnection loss value.
[0126] Step S35: Calculate the sixth connectivity loss value using the third discriminator after updating the weights, based on the input N-1 third mask annotation maps and the third target mask map.
[0127] Step S36: Update the weights of the third generator based on the sixth connectivity loss value and the third generation loss value.
[0128] It should be noted that the above steps S30 to S36 need to be executed multiple times until the number of executions of steps S30 to S36 exceeds the set number of executions, or until the segmentation loss of the third generator itself (third generation loss value) and the connectivity loss judged by the third discriminator (sixth connectivity loss value) are lower than the set loss value.
[0129] For example, when training the third generator, the initial learning rate of the third generator can be set to 0.0005, the initial learning rate of the third discriminator can be set to 0.0001, and the number of training iterations can be set to 200. The learning rate can be updated using a warm-up learning rate and cosine annealing; that is, in the first iteration, the learning rate is warmed up with 0.0001, and subsequent iterations are calculated according to the cosine annealing formula. The optimizer used in training can be the SGD optimizer.
[0130] It should be noted that in this embodiment, the formula for calculating the evaluation index used to evaluate the performance of the third generator is Dice(P2,T)=2*∣P2∩T∣ / (∣P2∣+∣T∣), where P2 is the result predicted by the third generator, and T is the actual result. Experiments show that if the third generator is trained without a third discriminator, the evaluation index value is 0.677; if the third generator is trained with a third discriminator, the evaluation index value is 0.744.
[0131] Example 5
[0132] Please see Figure 8 The fifth embodiment of this application proposes a three-dimensional lung reconstruction device 10, which includes: an image acquisition module 11, a first generation module 12, a second generation module 13, a third generation module 14, and a three-dimensional reconstruction module 15.
[0133] Image acquisition module 11 is used to acquire N two-dimensional images, including those of the lungs. First generation module 12 is used to generate a point cloud of the entire tubular organ based on the N two-dimensional images using a first generator. Second generation module 13 is used to generate point clouds of arteries, veins, and trachea based on the N two-dimensional images using a second generator. Third generation module 14 is used to generate lung segment points based on the N two-dimensional images using a third generator. Three-dimensional reconstruction module 15 is used to reconstruct the lungs in three dimensions using the point cloud of the entire tubular organ, the point cloud of arteries, the point cloud of veins, the point cloud of trachea, and the point cloud of lung segments.
[0134] The lung three-dimensional reconstruction device disclosed in this embodiment is used in conjunction with the image acquisition module 11, the first generation module 12, the second generation module 13, the third generation module 14 and the three-dimensional reconstruction module 15 to perform the lung three-dimensional reconstruction method described in the above embodiment. The implementation schemes and beneficial effects involved in the above embodiments are also applicable in this embodiment, and will not be repeated here.
[0135] Example 6
[0136] The sixth embodiment of this application proposes a terminal device, including a memory and a processor. The memory stores a computer program, and the computer program executes the lung three-dimensional reconstruction method described in the embodiment of this application when it is run on the processor.
[0137] Example 7
[0138] The seventh embodiment of this application proposes a readable storage medium storing a computer program that, when run on a processor, executes the lung three-dimensional reconstruction method described in the embodiments of this application.
[0139] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, as an alternative implementation, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0140] In addition, the functional modules or units in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0141] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0142] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for three-dimensional reconstruction of the lungs, characterized in that, The method includes: Obtain N two-dimensional images, including those of the lungs; A first generator is used to generate a point cloud of the tubular organ based on the N two-dimensional images; The second generator is used to generate arterial point clouds, vein point clouds, and tracheal point clouds based on the N two-dimensional images; A third generator is used to generate lung segment point clouds based on the N two-dimensional images; The lungs are reconstructed in three dimensions using the overall point cloud of the tubular organs, the point cloud of the arteries, the point cloud of the veins, the point cloud of the trachea, and the point cloud of the lung segments; specifically including: A first connected component is extracted from the overall point cloud of the tubular organ, a second connected component is extracted from the point cloud of the artery, a third connected component is extracted from the point cloud of the vein, and a fourth connected component is extracted from the point cloud of the trachea. Region growing processing is performed based on the first, second, third, and fourth connected components to reconstruct the artery, vein, and trachea in three dimensions. The lung segment point cloud is then filtered to reconstruct the lung segment in three dimensions. The first generator, the second generator, and the third generator are all pre-trained, and the first generator, the second generator, and the third generator are all different after training. The pre-training process for each generator includes: N two-dimensional training images are input into a target generator, which outputs a corresponding target mask image. The target mask image is a mask image of the lung target region corresponding to the middle two-dimensional training image located in the middle position among the N two-dimensional training images. The corresponding generation loss value is calculated based on the target mask image and the mask annotation image corresponding to the middle two-dimensional training image. The target generator inputs N mask annotation images into the target discriminator, so that the target discriminator calculates the corresponding connectivity loss value based on the input N mask annotation images, and replaces the middle mask annotation image in the N mask annotation images with the target mask image, so that the target discriminator calculates the disconnection loss value based on the input N-1 mask annotation images and the target mask image, wherein the N mask annotation images correspond one-to-one with the N two-dimensional training images; The weights of the target discriminator are updated based on the connectivity loss value and the disconnection loss value; the target discriminator with updated weights calculates the corresponding connectivity loss value based on the input N-1 mask annotation images and the target mask image, and updates the weights of the target generator in combination with the generation loss value; The target generator is the first generator, the second generator, or the third generator, and the target discriminator is the first discriminator, the second discriminator, or the third discriminator.
2. The lung three-dimensional reconstruction method according to claim 1, characterized in that, The pre-training process of the first generator specifically includes: N two-dimensional training images are input into the first generator so that the first generator outputs a first target mask image. The first target mask image is a mask image of the whole tubular organ corresponding to the middle two-dimensional training image located in the middle position among the N two-dimensional training images. The first generation loss value is calculated based on the first target mask image and the first mask annotation image corresponding to the intermediate two-dimensional training image; N first mask annotation images are input into the first discriminator corresponding to the first generator, so that the first discriminator calculates the first connectivity loss value based on the input N first mask annotation images, and the N first mask annotation images correspond one-to-one with the N two-dimensional training images; The first mask annotation image located in the middle of the N first mask annotation images is replaced with the first target mask image, so that the first discriminator calculates the first disconnection loss value based on the input N-1 first mask annotation images and the first target mask image; The weights of the first discriminator are updated based on the first connectivity loss value and the first disconnection loss value. The first discriminator, with updated weights, calculates the second connectivity loss value based on the input N-1 first mask annotation images and the first target mask image; The weights of the first generator are updated based on the second connectivity loss value and the first generation loss value.
3. The lung three-dimensional reconstruction method according to claim 1, characterized in that, The pre-training process of the second generator specifically includes: N two-dimensional training images are input into the second generator so that the second generator outputs a second target mask image. The second target mask image is a mask image of the middle two-dimensional training image located in the middle position among the N two-dimensional training images, including arteries, veins and trachea. The second generation loss value is calculated based on the second target mask image and the second mask annotation image corresponding to the intermediate two-dimensional training image; The N second mask annotation images are input into the second discriminator corresponding to the second generator, so that the second discriminator calculates the third connectivity loss value based on the input N second mask annotation images, and the N second mask annotation images correspond one-to-one with the N two-dimensional training images; The second mask annotation image located in the middle of the N second mask annotation images is replaced with the second target mask image, so that the second discriminator calculates the second disconnection loss value based on the input N-1 second mask annotation images and the second target mask image; The weights of the second discriminator are updated based on the third connectivity loss value and the second disconnection loss value. The second discriminator, with updated weights, calculates the fourth connectivity loss value based on the input N-1 second mask annotation images and the second target mask image; The weights of the second generator are updated based on the fourth connectivity loss value and the second generation loss value.
4. The lung three-dimensional reconstruction method according to claim 1, characterized in that, The pre-training process of the third generator specifically includes: N two-dimensional training images are input into the third generator so that the third generator outputs a third target mask image, which is a mask image including the lung segment corresponding to the middle two-dimensional training image located in the middle position among the N two-dimensional training images. The third generation loss value is calculated based on the third target mask image and the third mask annotation image corresponding to the intermediate two-dimensional training image; The N third mask annotation images are input into the third discriminator corresponding to the third generator, so that the third discriminator calculates the fifth connectivity loss value based on the input N third mask annotation images. The N third mask annotation images correspond one-to-one with the N two-dimensional training images. The third mask annotation image located in the middle of the N third mask annotation images is replaced with the third target mask image, so that the third discriminator calculates the third disconnection loss value based on the input N-1 third mask annotation images and the third target mask image; The weights of the third discriminator are updated based on the fifth connectivity loss value and the third disconnection loss value. The third discriminator, with updated weights, calculates the sixth connectivity loss value based on the input N-1 third mask annotation maps and the third target mask map; The weights of the third generator are updated based on the sixth connectivity loss value and the third generation loss value.
5. The method for three-dimensional lung reconstruction according to any one of claims 1 to 4, characterized in that, Also includes: The arteries, veins, trachea, and lung segments of the reconstructed lung are rendered.
6. A three-dimensional lung reconstruction device, characterized in that, The device includes: The image acquisition module is used to acquire N two-dimensional images, including those of the lungs. The first generation module is used to generate an overall point cloud of the tubular organ based on the N two-dimensional images using a first generator. The second generation module is used to generate arterial point clouds, vein point clouds and tracheal point clouds based on the N two-dimensional images using the second generator. The third generation module is used to generate lung segment point clouds based on the N two-dimensional images using a third generator. The 3D reconstruction module is used to reconstruct the lungs in 3D using the overall point cloud of the tubular organ, the point cloud of the artery, the point cloud of the vein, the point cloud of the trachea, and the point cloud of the lung segments; specifically, it is used for: A first connected domain is extracted from the overall point cloud of the tubular organ, a second connected domain is extracted from the point cloud of the artery, a third connected domain is extracted from the point cloud of the vein, and a fourth connected domain is extracted from the point cloud of the trachea. Region growing is performed based on the first connected region, the second connected region, the third connected region, and the fourth connected region to reconstruct arteries, veins, and trachea in three dimensions; the lung segment point cloud is filtered to reconstruct lung segments in three dimensions. The first generator, the second generator, and the third generator are all pre-trained, and the first generator, the second generator, and the third generator are all different after training. The pre-training process for each generator includes: N two-dimensional training images are input into a target generator, which outputs a corresponding target mask image. The target mask image is a mask image of the lung target region corresponding to the middle two-dimensional training image located in the middle position among the N two-dimensional training images. The corresponding generation loss value is calculated based on the target mask image and the mask annotation image corresponding to the middle two-dimensional training image. The target generator inputs N mask annotation images into the target discriminator, so that the target discriminator calculates the corresponding connectivity loss value based on the input N mask annotation images, and replaces the middle mask annotation image in the N mask annotation images with the target mask image, so that the target discriminator calculates the disconnection loss value based on the input N-1 mask annotation images and the target mask image, wherein the N mask annotation images correspond one-to-one with the N two-dimensional training images; The weights of the target discriminator are updated based on the connectivity loss value and the disconnection loss value; the target discriminator with updated weights calculates the corresponding connectivity loss value based on the input N-1 mask annotation images and the target mask image, and updates the weights of the target generator in combination with the generation loss value; The target generator is the first generator, the second generator, or the third generator, and the target discriminator is the first discriminator, the second discriminator, or the third discriminator.
7. A terminal device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when run on the processor, executes the lung three-dimensional reconstruction method according to any one of claims 1 to 5.
8. A readable storage medium, characterized in that, It stores a computer program that, when run on a processor, executes the lung three-dimensional reconstruction method according to any one of claims 1 to 5.