Automatic labeling and three-dimensional reconstruction method, system and device of lung anatomical structure based on visual-semantic joint driving, storage medium and program product

Abstract

The present application relates to the technical field of image analysis, and discloses a lung anatomical structure automatic labeling and three-dimensional reconstruction method, system, device, storage medium and program product based on visual-semantic joint driving, which is used to solve the technical problem that the prior art cannot fuse visual and semantic modal information, thereby causing insufficient recognition labeling accuracy. The method comprises the following steps: acquiring chest three-dimensional CT data and performing lung lobe segmentation to obtain a lung lobe segmentation mask; inputting the chest three-dimensional CT data into a bronchus segmentation network model to perform airway segmentation and obtain a bronchus tree segmentation mask; extracting an airway skeleton based on the bronchus tree segmentation mask, extracting visual connection features of the airway skeleton, combining anatomical semantic information, and executing multi-level naming through a visual-semantic joint driving bronchus naming model to obtain a naming result; and performing automatic division labeling and three-dimensional reconstruction on the lung anatomical structure by taking the lung lobe segmentation mask as a constraint segmentation region and taking the naming result as a guide field.

Description

Technical Field

[0001] This invention relates to the field of image analysis technology, and in particular to a method, system, electronic device, computer storage medium, and computer program product for automatic annotation and three-dimensional reconstruction of lung anatomy based on vision-semantic joint driving. Background Technology

[0002] The refined anatomical division of the lung from lobes to segments and subsegments is the core foundation of modern thoracic surgery procedures, including accurate lung cancer staging, differentiation of benign and malignant pulmonary nodules, planning of sublobar resection surgery, and interventional navigation. With the widespread use of multi-slice spiral CT, how to automatically and accurately delineate the boundaries of lung segments and subsegments in three-dimensional space has become one of the core technical challenges of computer-assisted thoracic surgery systems.

[0003] Since lung segments and lung subsegments are essentially lung parenchymal regions innervated by bronchial branches of corresponding levels, the air supply range of each bronchus defines its corresponding anatomical unit of lung tissue. By accurately identifying and naming the branches of the bronchial tree at each level, the parenchymal boundaries of lung segments and lung subsegments can be inferred in reverse, which can improve the accuracy of lung anatomy labeling and reconstruction to a certain extent.

[0004] However, existing methods for analyzing the bronchial tree mainly fall into two categories. One category is airway segmentation based on U-Net variants of convolutional neural networks (CNNs). These methods excel in binary segmentation of bronchi and can accurately extract the geometry of the airways. However, they only provide "yes / no" segmentation results, completely lacking anatomical semantic information and unable to distinguish specific lung segments or subsegments, thus failing to meet the clinical need for refined anatomical localization. The other category is bronchial labeling methods, including early rule-based graph matching labeling schemes and more recent labeling schemes combining topology-aware graph neural networks (GNNs) and point cloud networks. These methods have evolved from early rule-based graph matching to topology-aware graph neural networks (GNNs) and point cloud networks, but most existing methods are limited to lung segment-level prediction or treat subsegment labeling as a flat classification task, failing to utilize the rich semantic structure embedded in standardized clinical terminology. In recent years, visual-semantic basic models (such as CLIP) have made progress in the field of semantically perceptive medical image analysis, but there is still a technological gap in applying them to the hierarchical analysis of three-dimensional anatomical tree structures such as the bronchial tree.

[0005] Therefore, existing technologies cannot achieve precise naming of lung anatomical structures from lung segments to subsegments based on the bronchial tree, ensuring hierarchical consistency. This results in the inability to utilize the "airway-lung parenchyma mapping" relationship to guide the automatic division of fine anatomical structures at the lung segment and subsegment levels. Therefore, there is an urgent need to invent a new method that can comprehensively consider the visual and prior semantic information of the bronchial tree structure to generate accurate multi-level bronchial tree naming results. Based on these naming results, precise automatic annotation and accurate 3D reconstruction of lung anatomical structures can be achieved, providing reliable technical support for clinical applications such as precision thoracic surgery. Summary of the Invention

[0006] The main objective of this invention is to solve the technical problem that existing methods for automatic annotation and three-dimensional reconstruction of lung anatomy cannot efficiently integrate and utilize visual and semantic modal information, resulting in the accuracy of the annotation not meeting the needs of clinical use.

[0007] The first aspect of this invention provides a method for automatic annotation and three-dimensional reconstruction of lung anatomy based on vision-semantic joint driving, comprising: The original three-dimensional CT data of the chest is acquired, and the lung lobe is segmented from the three-dimensional CT data of the chest to obtain the lung lobe segmentation mask; The chest 3D CT data is input into the bronchial segmentation network model for airway segmentation to obtain the bronchial tree segmentation mask. Based on the bronchial tree segmentation mask, the airway skeleton is extracted, and the visual connectivity features of the airway skeleton are extracted. Combining the visual connectivity features with pre-constructed anatomical semantic information, a multi-level naming of the bronchial tree is performed through a visual-semantic joint-driven bronchial naming model to obtain the bronchial tree naming result. The lung anatomical structure is automatically divided, labeled, and reconstructed in three dimensions using the lung lobe segmentation mask to constrain the segmented region and the bronchial tree naming result as the guiding field.

[0008] Optionally, in a first implementation of the first aspect of the present invention, the step of performing lung lobe segmentation on the chest three-dimensional CT data to obtain a lung lobe segmentation mask includes: The chest 3D CT data is input into a first initial segmentation network constructed based on a deep learning algorithm to obtain coarse segmentation masks for the left and right lungs; The coarse segmentation mask of the left and right lungs is encoded as a spatial prior feature. The spatial prior feature is then concatenated with the chest 3D CT data in the channel dimension and input into the second initial segmentation network based on deep learning. The second initial segmentation network is then driven to perform lung lobe segmentation by combining the interlobar fissure plane supervision method to obtain the lung lobe segmentation mask.

[0009] Optionally, in a second implementation of the first aspect of the present invention, the bronchial segmentation network model includes a parameter-tuned encoder and a parameter-tuned bronchial decoder. The step of inputting the chest 3D CT data into the bronchial segmentation network model for airway segmentation to obtain the bronchial tree segmentation mask includes: The chest 3D CT data is input into a parameter-tuned encoder to extract multi-scale 3D CT features; The multi-scale 3D CT features are input into the bronchial decoder after parameter tuning. By upsampling at each level and fusing with the skip connection features of the corresponding level, the segmentation logic value corresponding to each voxel in the chest 3D CT data is output, and the bronchial tree segmentation mask is calculated based on the segmentation logic value. Furthermore, noise and abnormal regions in the bronchial tree segmentation mask are eliminated by removing small-volume isolated connected domains, performing morphological closing operations on local gaps, and retaining the largest component connected to the main bronchial tree.

[0010] Optionally, in a third implementation of the first aspect of the invention, before inputting the chest 3D CT data into a parameter-tuned encoder to extract multi-scale 3D CT features, the method further includes: Construct a third initial segmentation network comprising an encoder, a bronchial decoder, and an arteriovenous decoder, wherein the encoder may be shared by the bronchial decoder and the arteriovenous decoder; The third initial segmentation network is trained in two stages by calling the first joint optimization function based on region overlap loss, topological continuity loss and anatomical companion loss, and the encoder, bronchus decoder and arteriovenous decoder in the third initial segmentation network are tuned. After training, the arteriovenous decoder of the third initial segmentation network is removed, and a bronchial segmentation network model is constructed based on the parameter-tuned encoder and the parameter-tuned bronchial decoder.

[0011] Optionally, in a fourth implementation of the first aspect of the present invention, the step of extracting the airway skeleton based on the bronchial tree segmentation mask, extracting the visual connectivity features of the airway skeleton, combining the visual connectivity features with pre-constructed anatomical semantic information, and performing multi-level naming on the bronchial tree through a visual-semantic jointly driven bronchial naming model to obtain the bronchial tree naming result includes: The bronchial tree segmentation mask is input into the bronchial naming model to extract three-dimensional centerline points and construct an airway point cloud set. Based on the airway point cloud set, centerline map structure data is constructed, and spatial connection relationships and branch topology are extracted to obtain visual connection features. A text library containing standardized clinical terms is constructed as anatomical semantic information, and cue words are constructed based on the names of anatomical structures of the airways in the lung anatomy. The semantic features of the lung anatomy are encoded based on the anatomical semantic information and the cue words. The semantic features of the lung anatomy are embedded and mapped into the same processing space compatible with the visual connectivity features to obtain the aligned projection text features of the lung anatomy. Based on the aligned projection of the lung anatomical structure text features and the visual connectivity features, a multi-level naming prediction is performed, and the bronchial naming model outputs the bronchial tree naming results.

[0012] Optionally, in a fifth implementation of the first aspect of the present invention, before extracting three-dimensional centerline points based on the bronchial tree segmentation mask, constructing an airway point cloud set, constructing centerline map structure data based on the airway point cloud set, and extracting spatial connectivity and branch topology to obtain visual connectivity features, the method further includes: An initial naming model is constructed, and a second joint optimization function based on lung anatomy classification loss, semantic alignment constraint loss, and hierarchical consistency loss is called to perform three-stage dynamic scheduling training on the constructed initial naming model to obtain a bronchial naming model. The three-stage dynamic scheduling training includes a geometric feature warm-up stage, a multimodal fusion and constraint introduction stage, and a fine-tuning and consistency enhancement stage.

[0013] Optionally, in a sixth implementation of the first aspect of the present invention, the lung anatomical structure includes lung segment structures, and the bronchial tree naming result includes segmental bronchial naming results; The automatic segmentation, annotation, and 3D reconstruction of the lung anatomy structure using the lung lobe segmentation mask to constrain the segmented region and the bronchial tree naming result as the guiding field includes: Obtain the lung lobe segmentation mask and segment bronchus naming results; wherein, the lung lobe segmentation mask includes the region number of each lung lobe, and the segment bronchus naming results include the anatomical name of the segment bronchus, its parent lung lobe, central path, branch number, distal direction vector, and segment bronchus number; The lung lobe range to which each bronchus segment belongs is determined based on the naming results of each bronchus segment, and the lung lobe range to which it belongs is used as the seed region of the lung segment to constrain the segmentation region of the lung segment. Construct a segmental airway-guided cost map, perform three-dimensional watershed expansion within the segmented regions of the lung segments, and generate initial watershed lung segment results; Using the parent lung lobe of the lung lobe to which the segmental bronchus belongs as a consistency constraint, the initial watershed lung segments generated within the lung lobe are merged, checked, and corrected to obtain the automatic segmentation and labeling results of the lung segment structure and the three-dimensional reconstruction results.

[0014] Optionally, in a seventh implementation of the first aspect of the present invention, the lung anatomical structure further includes lung subsegment structures, and the bronchial tree naming result further includes subsegment bronchial naming results; After obtaining the automatic segmentation and annotation results of the lung segment structure and the three-dimensional reconstruction results, the following is also included: The lung subsegment range to which each subsegment bronchus belongs is determined based on the naming results of each subsegment bronchus, and the lung subsegment range is used as a seed region for the lung subsegment to constrain the segmentation region of the lung subsegment. Construct a subsegment-level airway-guided cost map, perform three-dimensional watershed expansion within the segmented regions of the lung subsegments, and generate initial watershed lung subsegment results; Using the parent lung segment of the subsegment bronchus as a consistency constraint, the initial lung subsegment results generated within the lung segment are merged, checked, and corrected to obtain the automatic division and labeling results of the lung subsegment structure and the three-dimensional reconstruction results.

[0015] Optionally, in an eighth implementation of the first aspect of the present invention, after acquiring the original chest three-dimensional CT data, the method further includes: The voxel intensity of the chest 3D CT data is normalized and mapped to the airway sensitive window range; The chest 3D CT data was then cropped to remove the chest wall and mediastinal background, resulting in chest 3D CT data containing the region of interest in the lungs.

[0016] A second aspect of this invention provides an automatic annotation and 3D reconstruction system for lung anatomy based on vision-semantic joint driving, comprising: The data acquisition module is used to acquire raw chest 3D CT data; The lung lobe segmentation module is used to segment the lung lobes of the chest 3D CT data to obtain a lung lobe segmentation mask. The airway segmentation module is used to input the chest 3D CT data into the bronchial segmentation network model for airway segmentation and obtain the bronchial tree segmentation mask. The airway naming module is used to extract the airway skeleton based on the bronchial tree segmentation mask, extract the visual connectivity features of the airway skeleton, and combine the visual connectivity features with pre-constructed anatomical semantic information to perform multi-level naming on the bronchial tree through a visual-semantic joint-driven bronchial naming model to obtain the bronchial tree naming result. The structural segmentation and reconstruction module is used to automatically segment, label, and reconstruct the lung anatomical structure using the lung lobe segmentation mask to constrain the segmented region and the bronchial tree naming result as a guiding field.

[0017] A third aspect of the present invention provides an automatic annotation and three-dimensional reconstruction device for lung anatomy based on vision-semantic joint driving, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the automatic annotation and three-dimensional reconstruction device for lung anatomy based on vision-semantic joint driving to perform the steps of the above-described automatic annotation and three-dimensional reconstruction method for lung anatomy based on vision-semantic joint driving.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described method for automatic annotation and three-dimensional reconstruction of lung anatomy based on a vision-semantic joint drive.

[0019] A fifth aspect of the present invention provides a computer program product comprising a computer program / instruction that, when executed by a processor, implements the steps of the above-described method for automatic annotation and three-dimensional reconstruction of lung anatomy based on a vision-semantic joint drive.

[0020] The technical solution provided by this invention involves acquiring raw 3D chest CT data and performing lung lobe segmentation on the 3D chest CT data to obtain a lung lobe segmentation mask. The 3D chest CT data is then input into a bronchial segmentation network model for airway segmentation to obtain a bronchial tree segmentation mask. Based on the bronchial tree segmentation mask, the airway skeleton is extracted, and its visual connectivity features are extracted. Combining these visual connectivity features with pre-constructed anatomical semantic information, a multi-level naming process is performed on the bronchial tree using a vision-semantic joint-driven bronchial naming model to obtain the bronchial tree naming result. The segmented regions are constrained by the lung lobe segmentation mask, and the bronchial tree naming result is used as a guiding field to automatically classify, label, and 3D reconstruct the lung anatomical structure. This method comprehensively considers the visual information and prior semantic information of the bronchial tree structure, generating accurate multi-level bronchial tree naming results. Based on the bronchial tree naming results, it achieves accurate automatic labeling and accurate 3D reconstruction of the lung anatomical structure. Furthermore, it can provide reliable technical support for clinical applications such as precision thoracic surgery.

[0021] Furthermore, the system, electronic device, computer-readable storage medium, and computer program product provided by this invention also solve the corresponding technical problems. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1This is a schematic diagram of the first process of the first embodiment of the automatic annotation and three-dimensional reconstruction method for lung anatomy based on vision-semantic joint driving in this invention application; Figure 2 This is a second flowchart illustrating the first embodiment of the automatic annotation and 3D reconstruction method for lung anatomy based on vision-semantic joint driving in this invention application. Figure 3 This is a flowchart illustrating the second embodiment of the automatic annotation and 3D reconstruction method for lung anatomy based on vision-semantic joint driving in this invention application; Figure 4 This is a schematic diagram of the training architecture of the bronchial segmentation network model in the second embodiment of the method for automatic annotation and 3D reconstruction of lung anatomy based on vision-semantic joint driving in this invention application. Figure 5 This is a schematic diagram of the bronchial naming model in the second embodiment of the method for automatic annotation and three-dimensional reconstruction of lung anatomy based on vision-semantic joint driving in this invention application. Figure 6 This is a schematic diagram of an embodiment of the automatic annotation and 3D reconstruction system for lung anatomy based on vision-semantic joint driving in this invention application; Figure 7 This is a schematic diagram of an embodiment of the automatic annotation and 3D reconstruction device for lung anatomy based on vision-semantic joint drive in this invention application; Figure 8 This is a schematic diagram of a computer-readable medium according to the present invention application. Detailed Implementation

[0023] Exemplary embodiments of the invention will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limiting the invention to the embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention more comprehensive and complete, and to facilitate a full communication of the inventive concept to those skilled in the art. The same reference numerals in the drawings denote the same or similar elements, components, or parts, and therefore repeated descriptions of them will be omitted.

[0024] Subject to the technical concept of this invention, the features, structures, characteristics or other details described in a particular embodiment may be combined in one or more other embodiments in a suitable manner.

[0025] In the description of specific embodiments, the features, structures, characteristics, or other details described in this invention are intended to enable those skilled in the art to fully understand the embodiments. However, it is not excluded that those skilled in the art can practice the technical solutions of this invention without one or more of the specific features, structures, characteristics, or other details.

[0026] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0027] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0028] The terms “and / or” or “and / or” include all combinations of any one or more of the listed items.

[0029] See Figure 1 and Figure 2 The first embodiment of the method for automatic annotation and three-dimensional reconstruction of lung anatomy based on vision-semantic joint driving in this invention application is described in detail below.

[0030] It is understood that the executing entity of this invention can be an automatic annotation and 3D reconstruction system for lung anatomy based on vision-semantic joint drive, or it can be a terminal or a server; the specific implementation is not limited here. This embodiment of the invention will be described using a server as an example.

[0031] S101, Lobe Segmentation: After receiving a request for automatic annotation and 3D reconstruction of lung anatomy, the server first needs to acquire the original chest 3D CT (Computer Tomography) data. After preprocessing the chest 3D CT data, initial lobe segmentation is performed to obtain the lobe segmentation results. In this embodiment, the chest 3D CT data acquired has all been obtained with the explicit consent of the subjects from whom the data was collected and has undergone anonymization processing.

[0032] In one specific implementation, during preprocessing, the voxel intensity of the chest 3D CT data is normalized and mapped to the airway sensitive window range, and the chest 3D CT data is cropped to remove the chest wall and mediastinal background, resulting in chest 3D CT data containing the lung region of interest. In these embodiments, the airway refers to the bronchial tree.

[0033] Subsequently, the preprocessed chest 3D CT data is input into a first initial segmentation network based on deep learning to perform coarse segmentation of the left and right lungs, obtaining coarse segmentation masks for the left and right lungs. Then, the coarse segmentation masks of the left and right lungs are embedded, encoding them into spatial prior features. These spatial prior features are then concatenated with the chest 3D CT data along the channel dimension and input into a second initial segmentation network based on deep learning to perform initial lobe segmentation, obtaining the lobe segmentation result. The lobe segmentation result is output in the form of a lobe segmentation mask. The first initial segmentation network used for coarse segmentation of the left and right lungs in this step is pre-trained based on pre-collected 3D CT data samples, which are pre-annotated with embedding information for the left and right lungs. Similarly, the second initial segmentation network used for lobe segmentation in this step is also pre-trained based on pre-collected 3D CT data samples, but the samples used to train the second initial segmentation network are pre-annotated with left and right lung information and lobe information.

[0034] In a preferred embodiment, when performing lung lobe segmentation, multiple downsampling and upsampling levels of the second initial segmentation network are respectively supplemented with interlobar segmentation auxiliary supervision signals. The interlobar segmentation auxiliary supervision signals are used to assist and constrain multiple intermediate levels of the second initial segmentation network, driving the second initial segmentation network to output a lung lobe segmentation mask containing five lung lobe categories.

[0035] The method in this step improves the accuracy of lung lobe segmentation, resulting in accurate lung lobe segmentation, which can provide accurate spatial constraints for subsequent lung segment and subsegment segmentation.

[0036] S102, Airway Segmentation: In this step, the preprocessed 3D CT data of the same chest from S101 is acquired, and this 3D CT data is input into the bronchial segmentation network model to perform airway segmentation, thereby obtaining the segmentation result of the bronchial tree. The bronchial tree segmentation result described in this step is a 3D bronchial tree segmentation mask.

[0037] In one specific implementation, a bronchial segmentation network model is constructed using an improved nnUnet (Neural Network U-Net) framework to ensure topological integrity from the main bronchus to the terminal bronchioles.

[0038] Specifically, the bronchial segmentation network model can be obtained by pre-constructing a third initial segmentation network and training it based on collected data samples. This third initial segmentation network adopts a shared encoder-dual decoder architecture, consisting of a shared encoder, a bronchial decoder, and an arteriovenous decoder. During training, a two-stage training method is used to reduce the problem of gray-level entanglement between arteries / venous vessels and airways. In the first training stage, only the shared encoder and bronchial decoder are trained, focusing on restoring the airway topology. In the second training stage, an arteriovenous decoder is added to jointly optimize the loss, and the shared encoder is purified using vascular gradients to improve the detection rate of distal small airways and the ability to distinguish between blood vessels and airways, thereby improving the topological integrity and boundary accuracy of bronchial segmentation. After training, a bronchial segmentation network model is constructed based on the trained encoder and bronchial decoder. The preprocessed 3D CT data of the same chest from S101 is input into the bronchial segmentation network model for processing to obtain a bronchial probability map. The bronchial probability map is then subjected to thresholding or argmax function processing to obtain a 3D bronchial tree segmentation mask. The three-dimensional bronchial tree segmentation mask is output in the form of a binary mask.

[0039] S103, Airway Labeler: In this step, the airway skeleton is first extracted based on the 3D bronchial tree segmentation mask, and its visual connectivity features are extracted. Anatomical semantic information of the lung anatomy is then constructed. A visual-semantic jointly driven bronchial naming model is invoked, combining the visual connectivity features and the anatomical semantic information to perform multi-level naming of the bronchial tree, obtaining naming results at different levels. The visual connectivity features of the airway skeleton include the geometric connections and branch topology contained in the bronchial centerline point cloud.

[0040] Specifically, the 3D bronchial tree segmentation mask is input into the bronchial naming model to extract 3D centerline points and construct an airway point cloud set. Based on the airway point cloud set, centerline map structure data is constructed, and spatial connection relationships and branch topology are extracted to obtain visual connection features.

[0041] In this embodiment, an anatomical terminology text library is pre-constructed as anatomical semantic information, and prompt words are constructed based on the anatomical names of airways in the lung anatomy. The semantic features of the lung anatomy are encoded based on the anatomical semantic information and the prompt words. This allows for the automatic assignment of standard names (such as RS1, RB1a, etc.) to each level of bronchus by combining prior knowledge of the anatomical text, and the establishment of hierarchical parent-child relationships.

[0042] When using the vision-semantic joint-driven bronchial naming model, the semantic features of the lung anatomy are first embedded and mapped into the same processing space compatible with the visual connectivity features, resulting in aligned and projected textual features of the lung anatomy. Based on these aligned and projected textual features and the visual connectivity features, multi-level naming prediction is performed to obtain the bronchial tree naming result. This bronchial tree naming result is a three-dimensional bronchial tree naming result, output as an airway multi-label mask.

[0043] In a preferred embodiment, the naming results of the three-dimensional bronchial tree at different levels mentioned in this step include the naming results of segmental bronchi (tertiary bronchi) and the naming results of subsegmental bronchi (corresponding to quaternary bronchi).

[0044] S104. Segmentation and Reconstruction of Lung Segments and Subsegments: In this embodiment, the automatic annotation and three-dimensional reconstruction of the lung anatomy mainly targets the division and reconstruction of lung segments and lung subsegments. Specifically, lung segments correspond to the segmental bronchus naming results included in the three-dimensional bronchial tree naming results output in step S103, while lung subsegments correspond to the subsegmental bronchus naming results.

[0045] During segmentation and reconstruction, the lung lobe segmentation results output in step S101 are obtained. The initial seed region is determined based on these lung lobe segmentation results, and the segmented region is constrained. Simultaneously, the 3D bronchial tree naming results are used as a guiding field to automatically segment, label, and reconstruct the lung anatomy. The lung anatomy includes lung segments and subsegments. In this step, segmental bronchi and subsegmental bronchi are classified to achieve automatic segmentation and labeling of lung segments and subsegments, and the corresponding 3D reconstruction results are output.

[0046] Specifically, using a lung lobe segmentation mask as the initial seed region to constrain the segmentation area, and combining it with the named segmented bronchi as the guiding field, an improved 3D watershed algorithm is applied to slice the lung parenchyma layer by layer along the airway direction, ultimately obtaining 18 lung segment reconstructions. Then, using the lung segment results as the initial seed region, and combining it with the branching names of the subsegmental bronchi within the lung, the 3D watershed algorithm is used to obtain the 3D reconstruction results of the lung subsegments.

[0047] The method provided in this invention enables fully automated construction of a structured, semantically meaningful, and surgically plannable 3D digital twin of the lungs from 3D CT data, achieving automatic annotation and 3D reconstruction of lung anatomical structures. Specifically, this method improves the recognition effect of the tracheal tree by using multiple models in combination, and greatly improves the recognition accuracy of lung anatomical structures by comprehensively considering the spatial information and prior semantic information of the tracheal tree; further enhancing the recognition accuracy and clinical usability of anatomical structures in the 3D reconstruction results.

[0048] See Figures 3-5 The second embodiment of the method for automatic annotation and 3D reconstruction of lung anatomy based on vision-semantic joint driving in this invention application consists of two stages: the first stage is the model building and training stage, and the second stage is the model forward inference stage. Specifically, it includes: I. Model Building and Training Phase; S201. Construct and train a deep learning segmentation model for performing lung lobe segmentation to obtain a first initial segmentation network and a second initial segmentation network. First, three-dimensional CT data samples are acquired, and the three-dimensional CT data samples are labeled with embedding information tags for the left and right lungs and the lung lobes. In this embodiment, all training samples are obtained with the explicit consent of the subjects from whom the data is collected and are desensitized.

[0049] A first initial segmentation network is constructed based on a deep learning algorithm. The first initial segmentation network is trained and its parameters are adjusted by calling three-dimensional CT data samples containing embedded information labels of the left and right lungs. The first initial segmentation network that can segment the left and right lungs based on the three-dimensional CT data is obtained. A second initial segmentation network is constructed based on a deep learning algorithm and an interlobar segmentation-assisted supervised algorithm. The second initial segmentation network is trained and its parameters are tuned using 3D CT data samples containing embedded information labels of labeled lung lobes, resulting in a second initial segmentation network that can segment five lung lobe categories based on 3D CT data.

[0050] S202. Construct and train the bronchial segmentation network model used for airway segmentation to obtain the trained bronchial segmentation network model. In this step, a third initial segmentation network is first constructed, comprising an encoder, a bronchial decoder, and an arteriovenous decoder. The encoder can be shared by the bronchial decoder and the arteriovenous decoder. A two-stage training is performed on the third initial segmentation network using a first joint optimization function based on region overlap loss, topological continuity loss, and anatomical accompanying loss, and the encoder, bronchial decoder, and arteriovenous decoder in the third initial segmentation network are then tuned. After training, the arteriovenous decoder of the third initial segmentation network is removed, and a bronchial segmentation network model is constructed based on the tuned encoder and the tuned bronchial decoder.

[0051] The following is a step-by-step explanation: S2021. Obtain the original chest CT images and perform data preprocessing and training sampling. Raw chest CT images from multiple cases were collected as training samples, which included annotations or coarse segmentation results. Subsequently, the chest CT images underwent data preprocessing and training sampling.

[0052] In this embodiment, all training samples collected were obtained with the explicit consent of the subjects from whom the samples were collected, and were also desensitized.

[0053] In one specific implementation, the data preprocessing and training sampling include the following: (1) CT intensity normalization and lung region clipping: Voxel intensity is mapped to an airway sensitive window (e.g., -1200HU to 600HU, where HU represents Henle units). During training, random perturbation is applied to the upper limit of the window to enhance cross-device generalization. Then, the ROI (Region of Interest) is clipped according to the lung parenchyma to reduce the interference of the chest wall and mediastinal background on the distal small airways.

[0054] (2) Establishment of bronchial index: The results of offline scanning of each case annotation or coarse segmentation are converted into local equivalent diameter according to the voxel spacing. The voxel coordinates of bronchial tubes with a diameter of less than 2 mm are written into the lightweight index table for quick query during the training phase, avoiding repeated full-volume distance transformation in each iteration.

[0055] (3) Priority sampling of bronchioles: Since the airways (especially bronchioles) only account for a very small part of the lung volume, in order to prevent the small bronchioles from being missed by random sampling, this step uses a priority sampling strategy when calling the data loader to perform the training process.

[0056] Specifically, let the oversampling ratio of the foreground be... When a sub-patch needs to cover the foreground, it is done with probability. The sub-block center is anchored to the index coordinates of the bronchioles; otherwise, sampling is performed according to the normal planning. This strategy increases the expected frequency of terminal bronchi in mini-batches to alleviate the extreme imbalance between large and small airways.

[0057] (4) Multi-source balanced sampling: assign sampling weights to multi-center cases according to the data source. ,in, Number the data source. The number of cases contained in this source. To balance the strength coefficient, this invention takes This makes the contributions of each scan domain tend to be more balanced in gradient updates.

[0058] S2022, Construct a three-dimensional bronchial segmentation network; In plain CT images, bronchi, pulmonary arteries, and pulmonary veins exhibit significant overlap in CT values ​​(grayscale). Traditional single-decoder segmentation networks, when processing such "grayscale entanglement," require the shared feature layer to simultaneously respond to both low-density regions of the airway lumen and medium-density regions of the blood vessel lumen. This leads to feature competition, where the network tends to learn larger vascular features while suppressing weaker airway responses. Ultimately, this results in missed detections of distal small airways and false positives where blood vessels are misidentified as airways. Therefore, to address these issues, this embodiment employs a shared encoder-dual-decoder decoupled architecture in its constructed 3D bronchial segmentation network.

[0059] The shared encoder-dual decoder decoupling architecture of the three-dimensional bronchial segmentation network consists of a shared encoder ( ), Bronchial decoder ( ) and arteriovenous decoder ( It consists of three parts. Among them, the shared encoder ( ) is responsible for receiving raw CT sub-blocks and extracting multi-scale anatomical features; the bronchial decoder ( The decoder is responsible for decoding shared features into bronchial segmentation maps; the arteriovenous decoder ( The system is responsible for decoding the shared features into artery and vein segmentation maps. This arteriovenous decoder only participates in loss calculation during the training phase and will be removed during the inference phase.

[0060] The bronchial decoder and the arteriovenous decoder have the same structure but independent parameters. That is, they have the same number of network layers, convolutional kernel size, and upsampling method, but they store and learn weight parameters independently. They use the same encoder, but use different final layer classification heads (number of output channels) to match the 2-class classification task (background / airway) of the bronchi and the 3-class classification task (background / artery / vein) of the arteriovenous system, respectively.

[0061] (1) Shared encoder: For shared encoders ( The input data it receives is a sub-block of 3D CT. And satisfy: ; in, , , Sub-blocks Dimensions along the three voxel axes; The shared encoder ( A three-dimensional residual convolutional neural network structure is adopted, and multi-scale skip connection features are output through step-by-step downsampling operations, denoted as... ;in, For the first Level-skip connection features; This represents the number of downsampling layers for the encoder. .

[0062] (2) Decoupling of dual decoders: The three-dimensional bronchial segmentation network described in this embodiment uses multi-scale features output from a shared encoder, which are then input into a bronchial decoder. ) and arteriovenous decoder ( In this process, the bronchial segmentation logic values ​​(logits) are obtained respectively. Arterial and venous segmentation logic values .

[0063] Among them, the bronchial segmentation logic value = arteriovenous segmentation logic value And satisfy: ; ; Right now The dimension is 2 indicates that there are two channels, with the two channels representing the prediction score for each voxel belonging to the "background" and the prediction score for each voxel belonging to the "airway". The dimension is 3 represents the number of channels, with the three channels representing the prediction score for each voxel belonging to "background", the prediction score for each voxel belonging to "artery", and the prediction score for each voxel belonging to "vein".

[0064] S2023, Construct the loss function; This invention compresses the optimization objective into three types of constraints: region overlap, topological continuity, and anatomical companions, and constructs a total loss function. In this embodiment, training is divided into two stages, specifically as follows: Phase 1: ; Phase 2: ; in, For bronchial branch loss, Loss of monitoring of arterial and venous branches, For the loss of anatomical restraint, This is the weighting coefficient for arterial and venous branch loss. For the weighting coefficients of the anatomical accompanying loss; in a specific example, The value is 1.0. The value is 0.15.

[0065] The loss of each branch is explained below: (1) Bronchial branching loss : In this embodiment, the bronchial branching loss considers both regional overlap loss and topological continuity loss terms, and the specific expression is as follows: ; This is the region overlap loss term, used to constrain the voxel-level classification accuracy and hard case boundaries; This is a topological continuity loss term used to constrain the soft skeleton connectivity and the completeness of terminal detection; As the topological term weight coefficient, in a preferred embodiment, take .

[0066] The following section discusses the regional overlap loss term. and topological continuous loss term Please provide a detailed explanation: (a) Regional overlap loss term : In a preferred embodiment, the region overlap item To balance voxel classification and hard case mining, the expression is: ; in, For weighted Dice loss, The cross-entropy loss is TopK, and the coefficients of both terms are set to 0.5, representing an equally weighted combination of the two terms.

[0067] First, within each voxel in the sub-block Constructing spatial weights : ; in, This represents the weighting coefficient of the bronchioles. In a specific example, The value is 2.0; For the local equivalent diameter, This is the normalized distance to the local centerline; This indicates an indicator function, which is activated when the condition within the parentheses (i.e., ...) is met. The value is 1 if the condition is true, and 0 otherwise. This represents the centerline distance weighting coefficient. In a specific example, The value is 1.5; This represents the exponentially decaying term.

[0068] By adjusting the exponential decay term, the exponential term of voxels near the centerline is close to 1, and the weight is close to 1.5. For voxels far from the centerline (near the airway wall or edge), the exponential term decays rapidly, and the weight approaches 0. This allows the model to focus on learning the core trunk region of the airway and weaken the influence of blurred boundary regions, thereby improving the topological integrity of the segmentation.

[0069] Using the probability of the bronchial prospect express: ; Truth mask ; but The expression is: ; TopK cross-entropy is only for the highest CE value (cross-entropy loss value). Voxel backpropagation gradients, and the set of hard examples consisting of the K voxels with the highest CE values ​​(cross-entropy loss values) in the sub-block. ,but The expression is: ; in, Used to focus on the boundary voxels that are most difficult to classify; Indicates the number of hard voxels; It is the natural logarithm.

[0070] (b) Topological continuity loss term : In a preferred embodiment, a topological continuity loss term is used. The clDice (centerlineDice loss) and differentiable topological distance (Soft-TD) are merged into a unified topological constraint, with the specific expression as follows: ; For the probability of bronchial prospect and truth mask Apply Microsoft skeletonization This allows for the use of differentiable mathematical operations (such as convolution, pooling, softmax, and smooth approximations of distance transform) to simulate the skeleton extraction process, extracting the centerline skeleton of monomer widths from a 3D tubular structure volume mask. Based on this, Defined as: ; in, Represented as a product of individual elements. For voxel summation, the differentiable topological distance (Soft-TD) makes the discrete topological detection rate differentiable: ; in, This represents the predicted mass of the soft skeleton within the true airway lumen and can be considered a surrogate term for the "correctly detected topological length". The total mass of the truth-value soft skeleton is represented as a reference for length normalization. The stop-gradient operator is used to block the gradient backpropagation of this term, so that the denominator only performs the function of normalization; Indicates will Cut off to interval (here) In the expression above ); The smaller the value, the higher the effective detection rate of the predicted soft skeleton within the true lumen. Emphasis is placed on the overall overlap of the skeletal structure. The focus is on the effective detection ratio of the skeleton within the true lumen, and the two complement each other to constrain the topological integrity of the bronchial tree.

[0071] (2) Arterial and venous monitoring items : For arteriovenous segmentation logic values This invention constructs a specialized arteriovenous monitoring loss. Specifically, in this embodiment, the loss of arteriovenous monitoring We use the sum of weighted Dice loss and cross-entropy loss.

[0072] For cases involving only partial arteriovenous labeling, this invention calculates the loss of arteriovenous surveillance. To address this, an ignore mask mechanism is introduced. Specifically, for each voxel, except for regions already labeled as arteries, veins, or background, unlabeled regions do not participate in the gradient backpropagation of the loss function (i.e., the loss value is set to zero) or are assigned extremely low weight coefficients. By ignoring the mask, the unlabeled regions are avoided from being simply and strongly classified as background, thus eliminating the supervision noise introduced by sparse labeling. Through this marginal relaxation constraint, the model can fully utilize limited partially labeled data to learn the general morphological features of blood vessels while retaining the ability to correctly predict unlabeled vascular regions.

[0073] (3) Anatomical accompanying constraints : In lung anatomy, the pulmonary artery and bronchi are symbiotically associated, spatially adjacent and running in the same direction along the same path. Pulmonary veins, on the other hand, mostly run in the periphery of the lung parenchyma or in larger spaces between bronchial trees, exhibiting low symbiosis and significant spatial separation from the bronchi. Furthermore, on plain CT scans, the grayscale values ​​of all three are similar. Existing segmentation networks using a single segmentation head are prone to misidentifying accompanying arteries as airways or attaching venous segments far from the bronchi to the airway tree, resulting in false-positive errors in airway segmentation. Therefore, this step introduces an anatomical symbiosis constraint loss to explicitly incorporate the prior anatomical knowledge of the symbiotic relationship between the pulmonary artery and bronchi and the distance of pulmonary veins from the bronchi during the training phase. Through the soft constraints of the loss function, the shared encoder is guided to learn the joint feature representations that can effectively distinguish the three.

[0074] For any voxel The shared encoder outputs the model's prediction of the probability that the voxel belongs to the bronchus via the bronchus decoder and arteriovenous decoder, respectively. The model predicts the probability that the voxel belongs to an artery. The model predicts the probability that the voxel belongs to a vein. .

[0075] Define anatomical accompanying constraints The two sub-losses include companionship loss. and venous separation loss In one specific implementation, the accompanying consistency loss... The expression is: ; in, The scope of the anatomical constraints includes true airway voxels and voxels predicted by the model as airways with high confidence. For the high confidence threshold of the bronchus, when At that time, voxels The area considered by the model to belong to the airway; For set The number of mesosomes.

[0076] In one specific implementation, venous separation loss The expression is: ; In this embodiment, In the high-confidence region of the bronchus, the probability of artery and bronchus are encouraged to increase synergistically, and the "artery accompanied by bronchus" prior is used to correct vascular-airway confusion. The co-occurrence of veins and bronchi on the same voxel is penalized by using the a priori premise that "veins are far from bronchi" to suppress erroneous adhesion of veins to the airway tree.

[0077] The two sub-losses mentioned above are weighted and combined into a single anatomical accompanying constraint loss, namely: ; in, This is a weighting coefficient for the vein separation loss, used to balance the relative strength of the two constraints. In a preferred embodiment, .

[0078] In this embodiment, Enabled only during the joint fine-tuning phase, and only if the current training batch contains arteriovenous annotations; the gradient of this constraint loss is backpropagated via the shared encoder, prompting... Learn the combined characteristics that can distinguish between accompanying arteries and entangled veins without altering the bronchial decoder. The learned bronchiolar topological sensitivity.

[0079] S2024, Two-stage training; After the network structure and constraints described above are constructed, multiple raw CT image data are acquired as training samples, and training is conducted in two phases. The first training phase is the airway-specific training phase, and the second training phase is the joint fine-tuning phase. Specifically: (1) First training phase - specialized airway training: In the first training phase, only the shared encoder is deployed during training. ) and bronchial decoder ( During training, bronchial branching loss was optimized on multicenter bronchial annotation data. Training was conducted in conjunction with the priority sampling of bronchioles in step S201.

[0080] This stage focuses on restoring the complete bronchial tree topology and establishing a characteristic base that is sensitive to small airways.

[0081] (2) Second training phase - joint fine-tuning: After loading in the first training phase, the shared encoder ( ) and bronchial decoder ( The weight of the replicated bronchial decoder ( ) and initialize the arteriovenous decoder ( Forward propagation and loss calculations are performed using joint data with complete or partial annotation of bronchi and arteries / venous vessels. The calculated total loss function is the optimization function described in step 203 above (Step 2: ).

[0082] Specifically, initialize the arteriovenous decoder ( When the first phase of training of the bronchial decoder is completed, the bronchial decoder ( All network weights except for the final layer classification head are copied and assigned to the arteriovenous decoder. The corresponding structural layer of the arteriovenous decoder. This initialization strategy enables the arteriovenous decoder ( It eliminates the need to learn the general geometric representation of tubular structures from scratch, and only requires fine-tuning its last-layer-specific features during joint training, thereby significantly accelerating second-stage convergence and avoiding the forgetting of shared encoder features due to random initialization.

[0083] In this embodiment, the third initial segmentation network structure with two-stage training is as follows: Figure 4 As shown. After completing two stages of training, the original image data is input into the parameter-tuned third initial segmentation network structure to output the bronchial probability map and the arteriovenous probability map. After thresholding or processing with the argmax function, the arteriovenous segmentation mask and the bronchial segmentation mask can be obtained respectively. In a specific implementation, before generating the arteriovenous segmentation mask and the bronchial segmentation mask, a post-processing step is also included; this post-processing step includes steps such as removing small-volume isolated connected components, performing morphological closure operations on local gaps, and retaining the largest component connected to the main bronchial tree.

[0084] In a preferred embodiment, after completing the two phases, the arteriovenous decoder of the third initial segmentation network is removed. ), based on the trained shared encoder ( ) and bronchial decoder ( Construct a bronchial segmentation network model for the actual reasoning stage.

[0085] This step achieved significant synergistic gain through a two-stage training strategy. After completing the second stage of joint fine-tuning, the bronchial decoder ( The segmentation performance on plain CT data shows a comprehensive improvement compared to the first stage of training alone. Specifically, this is manifested in two ways: firstly, the anatomical accompaniment data transmitted back by the arteriovenous decoder is significantly enhanced. The constrained gradient and shared encoder effectively decoupled airway-vascular features, resulting in a significant enhancement of the response of distal bronchioles and a substantial improvement in the topological integrity of the bronchial tree (measured by centerline recall). Secondly, since the encoder eliminates grayscale interference from accompanying arteries, the accuracy of bronchial segmentation boundaries (measured by average surface distance) is significantly improved, and the false positive rate of misidentifying blood vessels as airways is greatly reduced.

[0086] S203. Construct and train the bronchial naming model for performing multi-level naming of the bronchial tree to obtain the trained bronchial naming model. In this step, an initial naming model is first constructed, and a second joint optimization function based on lung anatomy classification loss, semantic alignment constraint loss, and hierarchical consistency loss is called to perform three-stage dynamic scheduling training on the constructed initial naming model to obtain a bronchial naming model. The three-stage dynamic scheduling training includes a geometric feature warm-up stage, a multimodal fusion and constraint introduction stage, and a fine-tuning and consistency enhancement stage.

[0087] S2031. Construct the initial naming model; The initial naming model includes a geometric encoding branch, a semantic encoding branch, and a hierarchical decoding branch. The geometric encoding branch comprises a graph encoder and a point cloud encoder, used to extract the topological features of the centerline graph structure and the spatial geometric features of the point cloud set, and then fused them through a fusion module to obtain visual fusion features. The semantic encoding branch comprises a pre-trained text encoder and a multilayer perceptron projector, used to encode anatomical terminology text and project it onto a feature space aligned with the visual features. The hierarchical decoding branch comprises a segment decoder and a sub-segment decoder, which use a cross-attention mechanism to perform semantic enhancement decoding on visual features guided by text features, outputting segment-level prediction results and sub-segment-level prediction results respectively.

[0088] S2032, Construct the loss function; To effectively train the bronchial naming model proposed in this invention and to address challenges such as multi-objective optimization, semantic alignment, and hierarchical logical consistency in the bronchial tree hierarchical labeling task, this invention designs a composite loss function and employs a dynamic scheduling strategy to balance the importance of each loss term at different training stages.

[0089] (1) Composition of composite loss function; The total loss function is a weighted sum of four core components: segment and subsegment classification loss, semantic alignment constraint loss, and hierarchical consistency loss. The specific expression is as follows: ; in, Represents classification loss, This indicates a loss of hierarchical consistency. This represents the semantic alignment constraint loss.

[0090] (a) Classification loss ; This loss is used to supervise the model's basic classification performance at both the segment and subsegment levels. Specifically, it employs the standard weighted cross-entropy loss, directly optimizing the match between the model's predicted class labels and the gold standard labels. Class weights are introduced to mitigate the extreme class imbalance problem among different anatomical structures in the dataset (especially the 61 subsegments), ensuring that even the minority classes are adequately learned. The specific expression is as follows: ; in, , These represent the logits of the true labels and the model predictions, respectively. , These represent the segmental bronchus and subsegmental bronchus, respectively. , These represent the weights for lung segment loss and lung subsegment loss, respectively.

[0091] (b) Hierarchical consistency loss ; This invention innovatively proposes a hierarchical consistency loss function based on Jensen-Shannon divergence, aiming to ensure that the model's predicted fine-grained lung subsegment classification results are logically strictly subordinate to the coarse-grained lung segment classification results, thereby eliminating anatomical hierarchical contradictions. The specific implementation steps are as follows: (b-1) Constructing the anatomical mapping matrix: A binary mapping matrix is ​​pre-established to explicitly define the parent-child membership relationship between the 61 subsegments and the 19 lung segments (i.e., if a subsegment belongs to a lung segment, the corresponding position is marked as 1, otherwise it is marked as 0). This matrix is ​​embedded into the calculation process as fixed anatomical prior knowledge.

[0092] (b-2) Calculate the probability distribution of the "parent-child" node branches: The distribution of the parent node is the probability distribution of the lung segments (19 classes) directly predicted by the model. The child node aggregation distribution is the distribution of the probabilities of the subsegments (61 classes) predicted by the model. The equivalent lung segment probability distribution is obtained by mapping and summing the anatomical parent-child membership relationships, as detailed in the section on hierarchical consistency loss below. The expression.

[0093] (b-3) Calculate the JS divergence loss, as well as the following hierarchical consistency loss. As shown in the expression, by minimizing the JS divergence between the two distributions mentioned above, the model is guided to learn an intrinsic hierarchical compatible feature representation: that is, if the model considers a point to belong to the "upper left lobe," then the predicted probability of that point at the subsegment level must be mainly concentrated in the subsegments included in the "upper left lobe" (such as the apical-posterior segment, anterior segment, etc.), and cannot be dispersed to subsegments of other lobes. Compared with the traditional KL divergence, the JS divergence has symmetry and boundedness, and its values ​​are more stable. It can effectively avoid gradient explosion or NaN (Not a Number) problems during training. At the same time, as a "soft constraint," it allows the model to retain reasonable uncertainty in regions with ambiguous boundaries, rather than forcibly truncating the data.

[0094] The following section discusses hierarchical consistency loss. The specific expression is: ; Where B is the batch size and N is the number of points in the point cloud. For segmental bronchi, probability of lung involvement. This method maps and accumulates the subsegment outputs based on anatomical parent-child relationships to obtain the equivalent lung segment probability distribution. This loss function implements end-to-end hierarchical logical constraints, effectively preventing errors that violate anatomical common sense, such as "the parent node is predicted as the left lung, but the child node is predicted as the right lung subsegment," and significantly improving the clinical logical credibility of the automatic labeling results.

[0095] (c) Semantic alignment constraint loss ; This loss function aims to deeply align visual features with linguistic semantics. Constructed using a contrastive learning approach, the loss function forces the model to learn point cloud local feature vectors that are closer to their corresponding text features in the feature space, while simultaneously distancing them from other irrelevant text prototypes. This allows the model to leverage rich prior knowledge of clinical terminology to guide the learning of geometric features, especially when faced with geometric feature ambiguity caused by airway deformation, where semantic information can compensate for this. The specific expression is as follows: ; in, It is the first Point cloud features, It is the corresponding real-world category text prototype. It is the text prototype of all C categories. Represents cosine similarity. It is a temperature coefficient used to adjust the sharpness of the distribution.

[0096] (2) Dynamic loss function weight scheduling strategy; Since the objectives and optimization difficulties of the aforementioned losses differ, using fixed weights throughout the training process may lead to model convergence difficulties or getting stuck in suboptimal solutions. Therefore, this invention proposes a three-stage dynamic scheduling strategy that adaptively adjusts the weights of each loss term based on the training epoch. The weights are adjusted using the loss function coefficients.

[0097] (a) First stage: Geometric feature warm-up stage (Epochs 0 - T1); In the first phase, set The model first learns robust basic geometric and topological representations. Semantic alignment and hierarchical constraints are temporarily disabled at this stage to avoid complex multi-task interference with the learning of basic features.

[0098] (b) Second stage: Multimodal fusion and constraint introduction stage (Epochs T1 - T2); In the second phase, set , It grows linearly from 0 to a preset maximum value (e.g., 0.5). The model grows linearly from 0 to a suitable value (such as 0.1). Once the basic features have stabilized, semantic information begins to play a role, and hierarchical constraints are applied to enable the model to learn fine-grained classification while satisfying anatomical logic. The goal is to gradually introduce semantic priors and hierarchical logic to guide the model to perform multimodal feature fusion.

[0099] (c) Third stage: Fine-tuning and consistency enhancement stage (Epochs T2 - T3); In the third phase, , The value is continuously increased linearly from 0.1 to the maximum value (e.g., 0.2). The purpose is to further enhance the semantic alignment effect and the consistency of hierarchical logic while ensuring high classification accuracy.

[0100] (d) Training completion phase (Epochs T3 - end): When the weights of each loss function are maintained at a high level (e.g., 1.0, 1.2, 0.5, 0.1), the model has a good foundation. At this point, all objectives are fully optimized to obtain the final best performance. Through this carefully designed dynamic weight scheduling mechanism, this invention effectively coordinates multiple interrelated but potentially conflicting optimization objectives, ensuring the stability of the model training process and obtaining a trained bronchial naming model.

[0101] S204. Obtain the raw chest 3D CT data and preprocess the chest 3D CT data; The voxel intensity of the chest 3D CT data is normalized and mapped to the airway sensitive window range, and the chest 3D CT data is cropped to remove the chest wall and mediastinal background, resulting in chest 3D CT data containing the lung region of interest.

[0102] S205. Acquire chest 3D CT data and perform lung lobe segmentation on the chest 3D CT data to obtain the lung lobe segmentation mask. First, a deep learning model is used to coarsely segment the left and right lungs based on chest CT volume data, obtaining left and right lung information. Then, the left and right lung information is used as embedding features, along with the target CT image information, and input into the deep learning segmentation model for initial lobe segmentation, resulting in a lobe segmentation mask. As a result of the division, k represents the lung lobe number.

[0103] In particular, when performing lung lobe segmentation, the interlobar fissure plane depth supervision is used to improve the accuracy of lung lobe segmentation, so as to provide more accurate spatial constraints for subsequent lung segmentation and subsegment segmentation.

[0104] S206. Input the three-dimensional chest CT data into the bronchial segmentation network model to perform airway segmentation and obtain the bronchial tree segmentation mask. In this embodiment, the bronchial segmentation network model includes a parameter-tuned encoder and a parameter-tuned bronchial decoder. During specific airway segmentation, the chest 3D CT data is input into the parameter-tuned encoder to extract multi-scale 3D CT features; these features are then input into the parameter-tuned bronchial decoder, where they are upsampled level by level and fused with skip connection features at the corresponding level to output segmentation logic values ​​corresponding to each voxel in the chest 3D CT data. A 3D bronchial tree segmentation mask is then calculated based on these segmentation logic values.

[0105] In one specific implementation, airway segmentation is performed based on the bronchial segmentation model trained in step S201: preprocessed chest 3D CT image data is input into the trained bronchial segmentation network model to perform forward computation, and the bronchial decoder outputs a 3D bronchial segmentation logits tensor. ;in, tensor It contains two channels: the first channel corresponds to the background, and the second channel corresponds to the bronchi. A Softmax normalization operation is performed on the output logic value along the channel dimension to obtain the bronchial foreground probability map. Each voxel has a value between 0 and 1, representing the confidence level that the voxel belongs to the bronchial lumen. The probability map is binarized to obtain a binary mask of the bronchus. The binary mask generated in this step is used in the airway naming process of step S207 for subsequent processing.

[0106] During binarization, either thresholding or argmax methods are used. When using thresholding, a probability threshold is set. (Preferably 0.5), The voxels are identified as bronchi, and the rest as background. When using the argmax method, the logical tensor can be directly obtained. The bronchogenic factor is the one with the largest response in the second channel.

[0107] In a preferred embodiment, after the segmentation is performed, a post-processing step is further included, specifically including: Removing small-volume isolated connected components: Perform 3D connected component labeling on the binary mask, calculate the number of voxels contained in each connected component, and remove those with a volume smaller than a preset threshold. Isolated connected regions (preferably 50 voxels) are removed as noise; Morphological closing operation: A morphological closing operation (dilation followed by erosion) is performed on a binary mask using three-dimensional structuring elements to fill tiny pores and fissures in local areas of the bronchial wall, thereby improving the continuity of the airway lumen. Main connectivity preservation: Preserve the largest connectivity component of the bronchial tree that connects to the main trachea (the largest connectivity component starting below the larynx), and remove free and isolated regions caused by vascular adhesions or false positives in segmentation, to ensure that the output mask is consistent with the structure of a single bronchial tree in anatomy.

[0108] S207. Based on the bronchial tree segmentation mask, extract the airway skeleton and extract the visual connectivity features of the airway skeleton. Combine the visual connectivity features with the pre-constructed anatomical semantic information, and perform multi-level naming on the bronchial tree through a visual-semantic joint-driven bronchial naming model to obtain the bronchial tree naming results. This step proposes a hierarchical labeling method for segmental and subsegmental bronchi based on visual-language guidance and semantic alignment point cloud learning. Specifically, the 3D bronchial tree segmentation mask is input into the bronchial naming model to extract 3D centerline points and construct an airway point cloud set. Based on the airway point cloud set, a centerline map structure data is constructed, and spatial connectivity and branch topology are extracted to obtain visual connectivity features. A text library containing standardized clinical terms is constructed as anatomical semantic information, and prompt words are constructed based on the anatomical names of airways in the lung anatomy. The semantic features of the lung anatomy are encoded based on the anatomical semantic information and the prompt words. The semantic features of the lung anatomy are embedded and mapped into the same processing space compatible with the visual connectivity features to obtain aligned and projected lung anatomy text features. Multi-level naming prediction is performed based on the aligned and projected lung anatomy text features and the visual connectivity features, and the bronchial naming model outputs the 3D bronchial tree naming results.

[0109] In this embodiment, multi-level naming is performed based on the bronchial naming model constructed and trained in step S203. The specific steps are as follows: S2071, Data Preparation and Multimodal Input Construction; (1) Airway Point Cloud Construction; Based on the bronchus segmentation mask obtained from the bronchus segmentation module, the 3D centerline of the bronchial tree structure is first extracted to obtain a set of candidate centerline points located on the central axis of the airway. Then, the coordinate set (Nx3) of the centerline point cloud is obtained using the farthest distance sampling strategy to ensure the spatial coverage and uniform distribution of the sampling points on the entire bronchial tree.

[0110] Specifically, an initial point is first selected from the set of candidate centerline points. Then, the candidate point with the largest distance from the current set of sampled points is iteratively selected until a preset number of centerline sampling points are obtained. In one specific implementation, the number of sampling points is set to (N=4096), and each sampling point contains at least three-dimensional spatial coordinate features (x,y,z), thereby forming a set of airway centerline point cloud coordinates of size (Nx3).

[0111] Furthermore, each sampling point can be associated with its corresponding airway branch label, segment label, subsegment label, normalized coordinates, local neighborhood connectivity, or centerline topological connectivity for subsequent geometric feature extraction, graph structure modeling, and airway anatomical naming prediction. The resulting airway point cloud retains both the spatial course information of the bronchial centerline and reflects the local geometric morphological features of the airway branches.

[0112] (2) Construction of the airway centerline graph; A graph structure is constructed based on nodes and edges extracted from the airway skeleton. Specifically, neighborhood analysis is first performed on the centerline sampling points to identify endpoints, branch points, and ordinary centerline points based on their number of connections within the three-dimensional neighborhood. Endpoints are typically connected to only one adjacent centerline point, branch points are typically connected to three or more adjacent centerline points, and ordinary centerline points are typically located between two adjacent centerline points. In one implementation, sampling points on the centerline can be used as graph nodes. Each node contains at least its three-dimensional spatial coordinates (x, y, z) and may further include features such as its associated airway branch, segment or sub-segment label, local radius, centerline direction vector, and normalized coordinates. Edges represent the spatial connections between centerline points and can be constructed based on adjacency relationships, continuous path relationships on the centerline skeleton, or local connections obtained through nearest neighbor search. The resulting airway centerline diagram can be represented as G=(V,E), where (V) is the set of airway centerline nodes and (E) is the set of connecting edges between nodes, used to explicitly or approximately encode the spatial connectivity and branch topology of the bronchial tree.

[0113] (3) Semantic Anatomical Prompts Construction; A text library containing standardized clinical terminology is constructed. For example, segmental bronchi include "left upper lobe apical bronchus" and "left upper lobe apical posterior bronchus," while subsegmental bronchi include "left upper lobe apical posterior segmental subsegmental bronchus" and "left upper lobe apical posterior posterior subsegmental bronchus," etc. This text, as anatomical semantic information, is input into the language branch as prior knowledge. Cue words are constructed based on the anatomical names of airways within the lung anatomy. Finally, the semantic features of the lung anatomy are encoded based on the anatomical semantic information and the cue words.

[0114] S2072, Multimodal Feature Encoding Fusion and Cross-Modal Projection; (1) Graph Encoder; See Figure 5 The top left section contains information related to the graph encoder. In this step, the constructed airway centerline map is input into the graph neural network (GNN). Neighborhood node information is aggregated through graph convolution operations, and the geometric and topological features of the nodes are extracted. This step ensures that the model can perceive the global connectivity relationships of the airway branches.

[0115] (2) Point cloud encoder and fusion module (Fuse); This step employs a multi-task learning approach for segment and sub-segment naming; see [link / reference]. Figure 5As shown in the left-middle section, the airway point cloud P is input into the backbone of a point convolutional network (Point Conv) to extract point cloud features. Subsequently, a fusion module (Fuse) concatenates and fuses the point cloud features with the geometric and topological features output by the graph encoder. After passing through three point cloud encoding layers, a comprehensive point cloud feature rich in geometric and topological information is obtained. The fusion module (Fuse) can be implemented based on a 1×1 convolutional layer.

[0116] (3) Language Branch and Hints Project; like Figure 5 As shown in the lower left corner, a pre-trained CLIP text encoder with frozen parameters is used to process semantic anatomical cues. Based on the anatomical names of bronchial segments and subsegments, cue words are generated and input into the CLIP model to obtain semantic features of lung segments and subsegments.

[0117] Because the feature space of the CLIP model differs from that of medical image feature space, this invention designs a segmental and subsegmental multi-layer perceptron (MLP) projection head to map the text embedding onto a space compatible with image features, thereby obtaining the projected lung segment text features. and sub-segment text features This step enables cross-modal feature alignment.

[0118] (4) Text-guided layered decoding and modulation; The multi-task learning framework consists of two decoding parts: one for naming lung segments and the other for naming lung subsegments. A cross-attention mechanism deeply injects semantic information into both the segment decoder and the subseg decoder. This mechanism allows the decoder to dynamically consider the semantics of relevant anatomical terms when generating subsegment features, thereby using linguistic priors to correct feature biases caused by geometric deformations. As shown in the lower right corner of the framework diagram, taking the subsegment decoder as an example, the subsegment features output from the last layer of encoding and the corresponding scale of projected lung subsegment text features are input into the cross-attention module to achieve guided fusion of point cloud features by textual semantics.

[0119] Specifically, the subsegment point cloud features are used as the query vector (Q), and the lung subsegment text features are used as the key vector (K) and value vector (V). The subsegment point cloud features are represented as follows: Where N is the number of centerline sampling points, C is the point cloud feature dimension, and the lung subsegment text feature is represented as... M represents the number of sub-segment categories, and D represents the text embedding dimension. First, a linear projection layer maps the point cloud features and text features to the same attention dimension, resulting in: ; in, , , These are learnable linear transformation parameters. Then, the attention weights between the point cloud features and the text features of each sub-segment are calculated: Where d is the dimension of the attention feature. Then, the text value vector is weighted and summed using the attention weights to obtain the semantic enhancement features guided by the text: Finally, the semantically enhanced features are added, concatenated, or linearly fused with the original sub-segment point cloud features to obtain the fused sub-segment features: ; Fusion can include residual connections, normalization layers, splicing operations, and multilayer perceptron mapping. Therefore, the cross-attention module can adaptively select the associated semantic information of the lung subsegment based on the geometric features of each centerline sampling point, thereby enhancing the semantic discrimination capability of subsegmental bronchi.

[0120] Similarly, through a decoding layer that integrates multi-scale text embeddings, multi-granular semantic guidance is achieved, ultimately leading to sub-segment prediction. Similar procedures can be followed to perform lung segment prediction and obtain lung segment predictions. This will not be elaborated upon here.

[0121] In this embodiment, after obtaining the prediction results, an automatic label for the bronchial tree is generated, including the anatomical name of each bronchial segment, its parent lobe, and its central path. Branch number and far-end direction vector Where i represents the segmental bronchus number; and the anatomical name of each subsegmental bronchus, its parent lung segment, and its central path. Branch number and far-end direction vector .

[0122] S208. Using a lung lobe segmentation mask to constrain the segmented region and the bronchial tree naming result as a guiding field, the lung anatomical structure is automatically divided, labeled, and reconstructed in three dimensions.

[0123] In this embodiment, the lung anatomy includes lung segment structures and lung subsegment structures, and the three-dimensional bronchial tree naming results include segmental bronchial naming results. The automatic segmentation and labeling, as well as the three-dimensional reconstruction, specifically include two parts: automatic segmentation and labeling of lung segments within the basin based on lobar constraints and bronchial segment naming, and automatic segmentation and labeling of lung subsegments within the basin based on lung segment constraints and bronchial segment naming.

[0124] Part 1: Watershed lung segment reconstruction based on lobar constraints and segmental bronchial nomenclature: Obtain the lung lobe segmentation mask and segmental bronchus naming results; wherein, the segmental bronchus naming results include the anatomical name of the segmental bronchus and its parent lung lobe; when performing automatic segmentation and labeling and 3D reconstruction, specifically including: determining the lung lobe range to which each segmental bronchus belongs based on the segmental bronchus naming results, and using the lung lobe range as the seed region of the lung segment to constrain the segmentation region of the lung segment; constructing a segmental airway guided cost map, performing 3D watershed expansion within the segmentation region of the lung segment to generate initial watershed lung segment results; using the parent lung lobe of the lung lobe range to which the segmental bronchus belongs as a consistency constraint, merging, checking and correcting each initial watershed lung segment generated within the lung lobe to obtain the automatic segmentation and labeling results and 3D reconstruction results of the lung segment structure.

[0125] Specifically, in this embodiment, a "watershed lung segment" refers to a lung parenchyma region within a corresponding lung lobe, guided by named bronchial segments. Each watershed lung segment corresponds to an anatomical name of a bronchial segment, and multiple watershed lung segments within the same lung lobe should be consistent with their parent lobe region after merging. This embodiment proposes an airway-guided three-dimensional watershed lung segment reconstruction method using a lung lobe segmentation mask and bronchial segment naming results as inputs. Unlike traditional watershed algorithms that mainly rely on image grayscale gradients or distance transformations, this scheme uses the lung lobe mask as a hard spatial constraint, named bronchial segments as anatomical seeds and directional guides, and introduces parent lobe consistency constraints during watershed expansion to obtain watershed lung segments that conform to the anatomical relationship of bronchial segments. The specific steps include the following: S2081. Obtain input data; Lung lobe segmentation mask obtained from 3D chest CT images Where k represents the lung lobe number; simultaneously, the anatomical name of each bronchus segment, its parent lung lobe, and its central path are obtained from the airway naming module. Branch number and far-end direction vector , where i represents the segment number of the bronchus.

[0126] S2082. Determine the lung segmentation constraint domain; Based on the anatomical nomenclature of each bronchial segment and the lobe to which it belongs, bronchial segment i is assigned to its corresponding parent lobe. For segmental bronchus i, its watershed expansion is only permitted within the parent lobe mask. Internal process, that is: ; S2083, Constructing lung segment seed regions; For each named segmental bronchus i, select its distal centerline point set or points within a preset length range distal to the centerline as the seed center, and expand it three-dimensionally around it according to a preset radius to form the lung segment seed region. The seed region is confined within its parent lobe mask, i.e.: .

[0127] S2084. Construct a segmental airway guidance cost map; Mask in each lung lobe Within the lung parenchyma, for any lung parenchyma v, only the bronchial segment to which it belongs within that lobe is calculated. The cost of expansion, among which This represents the set of segmental bronchi belonging to the k-th lobe of the lung. The dilation cost can be expressed as: ; in, This represents the shortest Euclidean distance from voxel v to the centerline of the segmental bronchus. G(v) represents the angle between the direction of the voxel relative to the centerline of the bronchus segment and the direction vector of the distal end of the bronchus segment; (G(v)) represents the gradient magnitude of the CT image at voxel v, used to enhance the response of natural boundaries such as lung fissures. Indicates the penalty term for lung lobe boundaries; These are adjustable weight parameters.

[0128] S2085, Performing three-dimensional watershed expansion controlled by intralobar markers; Seed region of segmental bronchus within the same lung lobe As markers, in the lung lobe mask Three-dimensional watershed expansion is performed internally. During the expansion, voxel v is assigned to the segmental bronchial watershed with the lowest cost: When the expansion fronts of adjacent lung segments meet, the candidate lung segment boundaries are determined based on cost differences, image gradients, or spatial boundary constraints.

[0129] S2086, Execute parent-level lobe consistency constraints; All watershed lung segments generated within the same lung lobe should be examined together to meet the following requirements: ; For unassigned voxels or regions with unstable boundaries, they can be assigned to the corresponding watershed lung segment based on the minimum cost, the nearest bronchial centerline, or the connectivity of adjacent regions; for voxels that exceed the parent-level lung lobe, lung lobe boundaries are trimmed.

[0130] S2087, Output watershed lung segment results; The final output is a set of watershed lung segment regions that correspond one-to-one with the segmental bronchus naming results. In routine bronchial segmental anatomy, 10 segments can be obtained in the right lung and 8 segments in the left lung, for a total of 18 segmental regions. When anatomical variations exist, the corresponding number of segmental regions can be output based on the actual number of named bronchial segments.

[0131] Part Two: Watershed Lung Segment Reconstruction Based on Segmental Constraints and Subsegmental Bronchial Nomenclature: In this embodiment, the lung anatomy structure also includes lung subsegment structures, and the three-dimensional bronchial tree naming results also include subsegment bronchial naming results. Specifically, the automatic segmentation and labeling and three-dimensional reconstruction process includes: obtaining subsegment bronchial naming results; wherein, the subsegment bronchial naming results include the anatomical name of the subsegment bronchus and its parent lung segment; determining the lung subsegment range to which the subsegment bronchus belongs based on the naming results of each subsegment bronchus, and using the lung subsegment range as a seed region to constrain the segmentation region of the lung subsegment; constructing a subsegment-level airway guided cost map, performing three-dimensional watershed expansion within the segmentation region of the lung subsegment to generate initial watershed lung subsegment results; using the parent watershed lung segment to which the subsegment bronchus belongs as a consistency constraint, merging, checking, and correcting the initial watershed lung subsegment results generated within the watershed lung segment to obtain the automatic segmentation and labeling results and three-dimensional reconstruction results of the lung subsegment structure.

[0132] In this embodiment, the "watershed lung subsegment" refers to a subregion of lung parenchyma within a corresponding watershed lung segment, guided by a named subsegmental bronchus. Each watershed lung subsegment corresponds to an anatomical name of a subsegmental bronchus, and multiple watershed lung subsegments within the same parent lung segment should be consistent with their parent watershed lung segment after merging. After obtaining the watershed lung segment results, this invention further uses the watershed lung segment and bronchial subsegment naming results as input to construct a hierarchical constrained airway-guided three-dimensional watershed lung subsegment reconstruction method. Unlike directly dividing lung subsegments within the entire lung lobe, this scheme uses the reconstructed watershed lung segment as a rigid spatial constraint, ensuring that each lung subsegment expands only within its corresponding parent lung segment, thereby guaranteeing the anatomical hierarchy consistency between lung lobe, lung segment, and lung subsegment. Specifically, it includes the following steps: S2088. Obtain input data; Input the reconstructed watershed lung segment region The airway naming module obtains the anatomical name of each subsegment bronchus, its parent segment bronchus, and its central path. Branch number and far-end direction vector , where j represents the subsegment bronchus number.

[0133] S2089. Determine the lung subsegment segmentation constraint domain; Based on the parent-child relationship in the bronchial nomenclature, subsegmental bronchus j is assigned to its parent segmental bronchus i=q(j). For subsegmental bronchus j, its watershed expansion is only permitted within the parent segmental watershed. Internal process, that is: ; S20810, Constructing lung subsegment seed regions; For each named subsegmental bronchus j, a set of its distal centerline points or a centerline point within a predetermined distal length range is selected as the seed center, and three-dimensional expansion is performed around it to form the lung subsegment seed region. The seed region is confined within its parent watershed lung segment, i.e.: S20811. Construct a subsegment-level airway guidance cost map; In each parent watershed lung segment Within this range, for any lung parenchyma v, only the subsegmental bronchus to which it belongs is calculated. The cost of expansion, among which This represents the collection of subsegmental bronchi belonging to the i-th lung segment. The dilation cost can be expressed as: ; in, This represents the shortest distance from voxel v to the centerline of the subsegmental bronchus. G(v) represents the angle between the direction of the voxel relative to the centerline of the bronchus in that subsegment and its distal direction vector; G(v) represents the gradient magnitude of the CT image. Indicates the penalty item at the lung segment boundary; These are adjustable weight parameters.

[0134] S20812, Performing three-dimensional watershed expansion controlled by intra-segmental markers; Subsegmental seed regions within the same parental watershed lung segment As markers, in the lung segment of the parent watershed Three-dimensional watershed expansion is performed internally. During the expansion, voxel v is assigned to the subsegment bronchial watershed with the lowest cost. When the expansion fronts of adjacent lung subsegments meet, the boundaries of candidate lung subsegments are determined based on cost differences, image gradients, or spatial boundary constraints.

[0135] S20813, Implement parent-level lung segment consistency constraints; All subsegments of the same parent lung segment are examined together to ensure that: For unassigned voxels or regions with unstable boundaries, they can be assigned to the corresponding watershed lung subsegment based on the minimum cost, the nearest subsegment bronchial centerline, or connectivity with adjacent regions; for voxels that exceed the parent lung segment's range, lung segment boundaries are trimmed.

[0136] S20814, Output results of lung subsegments in the watershed; The final output is a set of watershed lung subsegment regions that correspond one-to-one with the subsegmental bronchus naming results. The lung subsegment results can be used for subsegmental localization of lung nodules or lesions, bronchoscopic navigation, planning of lung segment or subsegment resection, and preoperative surgical pathway assessment.

[0137] The method provided in the second embodiment of this invention pioneers a "visual-language" dual-branch semantic alignment mechanism. It introduces the CLIP pre-trained language model as a prior knowledge engine to construct an independent language branch. Standard anatomical nomenclature is converted into text embedding vectors. A semantic alignment constraint loss is designed to force the alignment of the point cloud geometric feature space with the text semantic space. Even if the airway geometry is severely distorted due to lesions, the model can still "associate" the correct anatomical location through the mapping relationship in the semantic space, significantly improving robustness in complex pathological scenarios.

[0138] Furthermore, the method provided in the second embodiment of this invention innovatively proposes a deep fusion architecture of graph topology and point cloud geometry, and designs a dual-stream coupled coding architecture: it utilizes a graph convolutional network (GCN) to extract global topological features of the centerline skeleton, and uses PointNetconv to extract local geometric features of the airway surface. Then, it performs deep interaction in the feature channel dimension through a feature fusion module, and introduces a text-guided cross-attention mechanism in the decoding stage, using text prototypes to dynamically modulate geometric features. This achieves three-dimensional complementarity of "global topology + local geometry + semantic prior", solving the problem of insufficient expression by a single feature.

[0139] Furthermore, the method provided in the second embodiment of this invention proposes a hierarchical consistency loss function based on JS divergence, coupled with a dynamic weight scheduling strategy. An anatomical mapping matrix is ​​constructed to aggregate the probability distributions of child nodes (subsegments) into the distributions of parent nodes (lung segments). The JS divergence between the "directly predicted parent distribution" and the "parent distribution aggregated from child nodes" is minimized. A three-stage dynamic scheduling approach is adopted: geometric convergence is emphasized in the early stage of training, semantic alignment is introduced in the middle stage, and hierarchical logic is strengthened in the later stage. End-to-end logically self-consistent optimization is achieved, avoiding the gradient vanishing problem of hard constraints and utilizing the symmetry and stability of JS divergence to completely eliminate the contradiction between parent and child hierarchical predictions, further ensuring the clinical logical correctness of the results.

[0140] Furthermore, the method provided in this embodiment of the invention automatically improves the granularity of labeling from the traditional "lung segment level" to the "lung subsegment level." Through high-precision reconstruction, it can automatically output results conforming to anatomical standards end-to-end, eliminating the subjective differences and time-consuming bottlenecks of manual labeling. This provides important reference for surgical teaching, thoracic surgery planning, and interventional magnetic navigation; it can also be used for electromagnetic navigation bronchoscopy (ENB). bronchoscopy provides a high-confidence "virtual map" to improve the success rate and safety of bronchoscopy navigation, significantly increasing the success rate of single-pass biopsies of deep nodules and reducing the risk of complications such as pneumothorax caused by path planning errors. It introduces visual-linguistic semantic priors, not relying solely on geometric features. Even in complex pathological scenarios such as tumor compression, airway stenosis, or distortion, it can still accurately infer airway affiliation based on semantic knowledge, solving the pain point of traditional algorithms being prone to failure in lesion areas and expanding the scope of clinical application. It can also support the mainstream anatomical lung subsegment resection in modern thoracic surgery, automatically generating detailed three-dimensional resection plans to help doctors maximize the preservation of healthy lung tissue while radically removing tumors, reducing preoperative planning time from hours to minutes.

[0141] The above describes the automatic annotation and 3D reconstruction method for lung anatomy based on vision-semantic joint driving in this invention application. The following describes the automatic annotation and 3D reconstruction system for lung anatomy based on vision-semantic joint driving in this invention application. (See reference...) Figure 6 An embodiment of the automatic annotation and 3D reconstruction system for lung anatomy based on vision-semantic joint driving in this invention application includes: The data acquisition module 601 is used to acquire raw chest 3D CT data; Lung lobe segmentation module 602 is used to segment the lung lobes of the chest 3D CT data to obtain a lung lobe segmentation mask. The airway segmentation module 603 is used to input the chest 3D CT data into the bronchial segmentation network model for airway segmentation to obtain a bronchial tree segmentation mask; wherein, in this embodiment, the bronchial tree segmentation mask is a 3D bronchial tree segmentation mask. The airway naming module 604 is used to extract the airway skeleton based on the bronchial tree segmentation mask, extract the visual connectivity features of the airway skeleton, and combine the visual connectivity features with pre-constructed anatomical semantic information to perform multi-level naming on the bronchial tree through a visual-semantic joint-driven bronchial naming model to obtain the bronchial tree naming result; wherein, in this embodiment, the bronchial tree naming result is a three-dimensional bronchial tree naming result; The structural segmentation and reconstruction module 605 is used to automatically segment, label, and reconstruct the lung anatomical structure using the lung lobe segmentation mask to constrain the segmented region and the bronchial tree naming result as a guiding field.

[0142] The system provided in this invention realizes fully automated construction of a structured, semantic, and surgically plannable 3D digital twin of the lungs from 3D CT data, achieving automatic annotation and 3D reconstruction of lung anatomical structures. Specifically, this method improves the recognition effect of the tracheal tree by using multiple models in combination, and greatly improves the recognition accuracy of lung anatomical structures by comprehensively considering the spatial information and prior semantic information of the tracheal tree; further enhancing the recognition accuracy and clinical usability of anatomical structures in the 3D reconstruction results.

[0143] In another embodiment of this application, the lung lobe segmentation module 602 includes: The first lung segmentation unit is used to input the chest 3D CT data into the first initial segmentation network constructed based on a deep learning algorithm to obtain coarse segmentation masks for the left and right lungs. The second lung lobe segmentation unit is used to encode the coarse segmentation mask of the left and right lungs into spatial prior features. After concatenating the spatial prior features with the chest 3D CT data in the channel dimension, the concatenation is input into the second initial segmentation network based on deep learning. The second initial segmentation network is then driven to perform lung lobe segmentation in conjunction with the interlobar fissure plane supervision method to obtain the lung lobe segmentation mask.

[0144] In another embodiment of this application, the airway segmentation module 603 includes: The feature extraction unit is used to extract multi-scale three-dimensional CT features from the chest three-dimensional CT data by inputting it into the parameter-tuned encoder. The airway segmentation unit is used to input the multi-scale three-dimensional CT features into the bronchial decoder after parameter tuning, and to output the segmentation logic value corresponding to each voxel in the chest three-dimensional CT data by upsampling step by step and fusing with the skip connection features of the corresponding level, and to calculate the bronchial tree segmentation mask based on the segmentation logic value. The post-processing unit is used to eliminate noise and abnormal regions in the bronchial tree segmentation mask by removing small-volume isolated connected domains, performing morphological closing operations on local gaps, and retaining the largest component connected to the main bronchial tree.

[0145] In another embodiment of this application, a bronchial segmentation network construction module is further included, specifically used to train and construct the bronchial segmentation network model; The bronchial segmentation network construction module specifically includes: The first network construction unit is used to construct a third initial segmentation network including an encoder, a bronchial decoder, and an arteriovenous decoder, wherein the encoder can be shared by the bronchial decoder and the arteriovenous decoder; The first joint training unit is used to call the first joint optimization function based on the region overlap loss, topological continuity loss and anatomical accompanying loss to perform two-stage training on the third initial segmentation network, and to tune the encoder, bronchus decoder and arteriovenous decoder in the third initial segmentation network. The first model generation unit is used to remove the arteriovenous decoder of the third initial segmentation network after training is completed, and to construct a bronchial segmentation network model based on the parameter-tuned encoder and the parameter-tuned bronchial decoder.

[0146] In another embodiment of this application, the airway naming module 604 further includes: The visual feature extraction unit is used to input the bronchial tree segmentation mask into the bronchial naming model, extract three-dimensional centerline points, construct an airway point cloud set, construct centerline map structure data based on the airway point cloud set, and extract spatial connection relationships and branch topology to obtain visual connection features. The semantic feature extraction unit is used to construct a text library containing standardized clinical terms as anatomical semantic information, and to construct cue words based on the names of the anatomical structures of the airways in the lung anatomy, and to encode the semantic features of the lung anatomy based on the anatomical semantic information and the cue words. Alignment units are used to embed and map the semantic features of the lung anatomy into the same processing space compatible with the visual connectivity features, so as to obtain the aligned and projected text features of the lung anatomy. The naming execution unit is used to perform multi-level naming prediction based on the aligned projection of the lung anatomical structure text features and the visual connectivity features, and the bronchial naming model outputs the bronchial tree naming results.

[0147] In another embodiment of this application, a bronchial naming model construction module is also included, specifically used to train and construct the bronchial naming model; The bronchial naming model construction module specifically includes: The second network building unit is used to build the initial naming model; The second joint training unit is used to call the second joint optimization function constructed based on lung anatomy classification loss, semantic alignment constraint loss, and hierarchical consistency loss to perform three-stage dynamic scheduling training on the constructed initial naming model to obtain the bronchial naming model; wherein, the three-stage dynamic scheduling training includes a geometric feature warm-up stage, a multimodal fusion and constraint introduction stage, and a fine-tuning and consistency enhancement stage.

[0148] In another embodiment of this application, the lung anatomy includes lung segment structures, and the bronchial tree naming results include segmental bronchial naming results; The structural partitioning and reconstruction module 605 includes: The information acquisition unit is used to acquire the lung lobe segmentation mask and the segment bronchus naming results; wherein, the lung lobe segmentation mask includes the region number of each lung lobe, and the segment bronchus naming results include the anatomical name of the segment bronchus, its parent lung lobe, the central line path, the branch number, the distal direction vector, and the segment bronchus number. The constraint construction unit is used to determine the range of the lung lobe to which the segmental bronchus belongs based on the naming result of each segmental bronchus, and to use the range of the lung lobe to which it belongs as the seed region of the lung segment to constrain the segmentation region of the lung segment. The segmentation execution unit is used to construct a segment-level airway guided cost map, perform three-dimensional watershed expansion within the segmented region of the lung segment, and generate the initial watershed lung segment results. The segmentation correction unit is used to merge, check and correct each initial watershed lung segment generated within the lung lobe, using the parent lung lobe of the lung lobe to which the segmental bronchus belongs as a consistency constraint, to obtain the automatic segmentation and labeling results of the lung segment structure and the three-dimensional reconstruction results.

[0149] In another embodiment of this application, the lung anatomy structure further includes lung subsegment structures, and the bronchial tree naming results further include subsegmental bronchial naming results; in the structure division and reconstruction module: The information acquisition unit is also used to acquire the subsegmental bronchus naming results; wherein, the subsegmental bronchus naming results include the anatomical name of the subsegmental bronchus, its parent lung segment, central path, branch number, distal direction vector, and subsegmental bronchus number. The constraint construction unit is also used to determine the range of the lung subsegment to which the subsegment bronchus belongs based on the naming results of each subsegment bronchus, and to use the range of the lung subsegment to which it belongs as the seed region of the lung subsegment to constrain the segmentation region of the lung subsegment. The segmentation execution unit is also used to construct subsegment-level airway guided cost maps, perform three-dimensional watershed expansion within the segmented regions of lung subsegments, and generate initial watershed lung subsegment results. The segmentation correction unit is used to merge, check and correct the initial lung subsegment results generated within the lung subsegment by using the parent lung segment to which the subsegment bronchus belongs as a consistency constraint, so as to obtain the automatic segmentation and labeling results of the lung subsegment structure and the three-dimensional reconstruction results.

[0150] In another embodiment of this application, the automatic annotation and three-dimensional reconstruction system for lung anatomy based on vision-semantic joint drive further includes a preprocessing module, which is used to normalize and map the voxel intensity of the chest three-dimensional CT data to the airway sensitive window range; and to crop the chest three-dimensional CT data to remove the chest wall and mediastinal background, so as to obtain chest three-dimensional CT data containing the lung region of interest.

[0151] The system provided in this invention realizes fully automated construction of a structured, semantic, and surgically plannable 3D digital twin of the lungs from 3D CT data, achieving automatic annotation and 3D reconstruction of lung anatomical structures. Specifically, this method improves the recognition effect of the tracheal tree by using multiple models in combination, and greatly improves the recognition accuracy of lung anatomical structures by comprehensively considering the spatial information and prior semantic information of the tracheal tree; further enhancing the recognition accuracy and clinical usability of anatomical structures in the 3D reconstruction results. Specifically, the system provided in this embodiment of the invention improves the automatic labeling granularity of lung anatomy from the traditional "lung segment level" to the "subsegment level" by accurately identifying the location of segmental and subsegmental bronchi, achieving more precise segmentation. Through high-precision reconstruction, it can automatically output results that conform to anatomical standards end-to-end, eliminating the subjective differences and time-consuming bottlenecks of manual annotation. Furthermore, by introducing visual-linguistic semantic priors, this system does not rely solely on geometric features. Even in complex pathological scenarios such as tumor compression, airway stenosis, or distortion, it can still accurately infer airway attribution based on semantic knowledge. This solves the pain point of traditional algorithms being prone to failure in lesion areas, expands the scope of clinical application, and provides important reference for applications in scenarios such as surgical teaching, thoracic surgery planning, and interventional magnetic navigation.

[0152] Based on the same inventive concept, this specification also provides an electronic device for automatic annotation and three-dimensional reconstruction of lung anatomical structures driven by vision-semantics. The electronic device for automatic annotation and three-dimensional reconstruction of lung anatomical structures driven by vision-semantics in this invention application will be described in detail below from the perspective of hardware processing.

[0153] Figure 7 This is a schematic diagram of an electronic device provided as an embodiment of this specification. Refer to the following... Figure 7 To describe the electronic device 700 according to this embodiment of the invention. Figure 7 The electronic device 700 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0154] like Figure 7As shown, the electronic device 700 is presented in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including storage unit 720 and processing unit 710), a display unit 740, etc.

[0155] The storage unit stores program code that can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the processing method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 710 can perform actions such as... Figure 1 or Figure 3 The steps are shown.

[0156] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 7201 and / or a cache storage unit 7202, and may further include a read-only memory unit (ROM) 7203.

[0157] The storage unit 720 may also include a program / utility 7204 having a set (at least one) program module 7205, such program module 7205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0158] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0159] Electronic device 700 can also communicate with one or more external devices 100 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. Network adapter 760 can communicate with other modules of electronic device 700 via bus 730. It should be understood that, although... Figure 7As not shown in the diagram, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0160] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described in this invention can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the method described above according to this invention. When the computer program is executed by a data processing device, it enables the computer-readable medium to implement the method described above, i.e.: as... Figure 1 or Figure 3 The steps of the method shown.

[0161] Figure 8 This is a schematic diagram of a computer-readable medium provided for embodiments of this specification.

[0162] accomplish Figure 1 or Figure 3 The computer program of the method shown can be stored on one or more computer-readable media. A computer-readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0163] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0164] In addition, the present invention also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the method for automatic annotation and three-dimensional reconstruction of lung anatomy based on visual-semantic joint drive as described in any of the above embodiments.

[0165] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0166] In summary, the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that in practice, general-purpose data processing devices such as microprocessors or digital signal processors (DSPs) can be used to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0167] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0168] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0169] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, on the personal information processing device, the personal information processing rules are clearly indicated / informed, and authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0170] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for automatic annotation and 3D reconstruction of lung anatomy based on vision-semantic joint driving, characterized in that, include: The original three-dimensional CT data of the chest is acquired, and the lung lobe is segmented from the three-dimensional CT data of the chest to obtain the lung lobe segmentation mask; The chest 3D CT data is input into the bronchial segmentation network model for airway segmentation to obtain the bronchial tree segmentation mask. Based on the bronchial tree segmentation mask, the airway skeleton is extracted, and the visual connectivity features of the airway skeleton are extracted. Combining the visual connectivity features with pre-constructed anatomical semantic information, a multi-level naming of the bronchial tree is performed through a visual-semantic joint-driven bronchial naming model to obtain the bronchial tree naming result. The lung anatomical structure is automatically divided, labeled, and reconstructed in three dimensions using the lung lobe segmentation mask to constrain the segmented region and the bronchial tree naming result as the guiding field.

2. The method for automatic annotation and three-dimensional reconstruction of lung anatomy based on vision-semantic joint driving as described in claim 1, characterized in that, The step of performing lung lobe segmentation on the chest 3D CT data to obtain the lung lobe segmentation mask includes: The chest 3D CT data is input into a first initial segmentation network constructed based on a deep learning algorithm to obtain coarse segmentation masks for the left and right lungs; The coarse segmentation mask of the left and right lungs is encoded as a spatial prior feature. The spatial prior feature is then concatenated with the chest 3D CT data in the channel dimension and input into the second initial segmentation network based on deep learning. The second initial segmentation network is then driven to perform lung lobe segmentation by combining the interlobar fissure plane supervision method to obtain the lung lobe segmentation mask.

3. The method for automatic annotation and three-dimensional reconstruction of lung anatomy based on vision-semantic joint driving as described in claim 1, characterized in that, The bronchial segmentation network model includes a parameter-tuned encoder and a parameter-tuned bronchial decoder. The step of inputting the chest 3D CT data into the bronchial segmentation network model for airway segmentation to obtain the bronchial tree segmentation mask includes: The chest 3D CT data is input into a parameter-tuned encoder to extract multi-scale 3D CT features; The multi-scale 3D CT features are input into the bronchial decoder after parameter tuning. By upsampling at each level and fusing with the skip connection features of the corresponding level, the segmentation logic value corresponding to each voxel in the chest 3D CT data is output, and the bronchial tree segmentation mask is calculated based on the segmentation logic value. Furthermore, noise and abnormal regions in the bronchial tree segmentation mask are eliminated by removing small-volume isolated connected domains, performing morphological closing operations on local gaps, and retaining the largest component connected to the main bronchial tree.

4. The method for automatic annotation and three-dimensional reconstruction of lung anatomy based on vision-semantic joint driving as described in claim 3, characterized in that, Before inputting the chest 3D CT data into the parameter-tuned encoder to extract multi-scale 3D CT features, the method further includes: Construct a third initial segmentation network comprising an encoder, a bronchial decoder, and an arteriovenous decoder, wherein the encoder may be shared by the bronchial decoder and the arteriovenous decoder; The third initial segmentation network is trained in two stages by calling the first joint optimization function based on region overlap loss, topological continuity loss and anatomical companion loss, and the encoder, bronchus decoder and arteriovenous decoder in the third initial segmentation network are tuned. After training, the arteriovenous decoder of the third initial segmentation network is removed, and a bronchial segmentation network model is constructed based on the parameter-tuned encoder and the parameter-tuned bronchial decoder.

5. The method for automatic annotation and three-dimensional reconstruction of lung anatomy based on vision-semantic joint driving as described in claim 1, characterized in that, The process involves extracting the airway skeleton based on the bronchial tree segmentation mask, extracting the visual connectivity features of the airway skeleton, and combining these visual connectivity features with pre-constructed anatomical semantic information. A multi-level naming process is then performed on the bronchial tree using a vision-semantic joint-driven bronchial naming model, resulting in the following bronchial tree naming results: The bronchial tree segmentation mask is input into the bronchial naming model to extract three-dimensional centerline points and construct an airway point cloud set. Based on the airway point cloud set, centerline map structure data is constructed, and spatial connection relationships and branch topology are extracted to obtain visual connection features. A text library containing standardized clinical terms is constructed as anatomical semantic information, and cue words are constructed based on the names of anatomical structures of the airways in the lung anatomy. The semantic features of the lung anatomy are encoded based on the anatomical semantic information and the cue words. The semantic features of the lung anatomy are embedded and mapped into the same processing space compatible with the visual connectivity features to obtain the aligned projection text features of the lung anatomy. Based on the aligned projection of the lung anatomical structure text features and the visual connectivity features, a multi-level naming prediction is performed, and the bronchial naming model outputs the bronchial tree naming results.

6. The method for automatic annotation and three-dimensional reconstruction of lung anatomy based on vision-semantic joint driving as described in claim 5, characterized in that, Before extracting three-dimensional centerline points based on the bronchial tree segmentation mask, constructing an airway point cloud set, constructing centerline map structure data based on the airway point cloud set, and extracting spatial connectivity and branch topology to obtain visual connectivity features, the method further includes: An initial naming model is constructed, and a second joint optimization function based on lung anatomy classification loss, semantic alignment constraint loss, and hierarchical consistency loss is called to perform three-stage dynamic scheduling training on the constructed initial naming model to obtain a bronchial naming model. The three-stage dynamic scheduling training includes a geometric feature warm-up stage, a multimodal fusion and constraint introduction stage, and a fine-tuning and consistency enhancement stage.

7. The method for automatic annotation and three-dimensional reconstruction of lung anatomy based on vision-semantic joint driving as described in claim 1, characterized in that, The lung anatomy includes lung segment structures, and the bronchial tree naming results include segmental bronchial naming results. The automatic segmentation, annotation, and 3D reconstruction of the lung anatomy structure using the lung lobe segmentation mask to constrain the segmented region and the bronchial tree naming result as the guiding field includes: Obtain the lung lobe segmentation mask and segmental bronchus naming results; wherein, the segmental bronchus naming results include the anatomical name of the segmental bronchus and its parent lung lobe; The lung lobe range to which each bronchus segment belongs is determined based on the naming results of each bronchus segment, and the lung lobe range to which it belongs is used as the seed region of the lung segment to constrain the segmentation region of the lung segment. Construct a segmental airway-guided cost map, perform three-dimensional watershed expansion within the segmented regions of the lung segments, and generate initial watershed lung segment results; Using the parent lung lobe of the lung lobe to which the segmental bronchus belongs as a consistency constraint, the initial watershed lung segments generated within the lung lobe are merged, checked, and corrected to obtain the automatic segmentation and labeling results of the lung segment structure and the three-dimensional reconstruction results.

8. The method for automatic annotation and three-dimensional reconstruction of lung anatomy based on vision-semantic joint driving as described in claim 7, characterized in that, The lung anatomy structure also includes lung subsegmental structures, and the bronchial tree naming results also include subsegmental bronchial naming results; After obtaining the automatic segmentation and annotation results of the lung segment structure and the three-dimensional reconstruction results, the following is also included: Obtain the subsegmental bronchus naming results; wherein, the subsegmental bronchus naming results include the anatomical name of the subsegmental bronchus and its parent lung segment; The lung subsegment range to which each subsegment bronchus belongs is determined based on the naming results of each subsegment bronchus, and the lung subsegment range is used as a seed region for the lung subsegment to constrain the segmentation region of the lung subsegment. Construct a subsegment-level airway-guided cost map, perform three-dimensional watershed expansion within the segmented regions of the lung subsegments, and generate initial watershed lung subsegment results; Using the parent lung segment of the subsegment bronchus as a consistency constraint, the initial lung subsegment results generated within the lung segment are merged, checked, and corrected to obtain the automatic division and labeling results of the lung subsegment structure and the three-dimensional reconstruction results.

9. The method for automatic annotation and three-dimensional reconstruction of lung anatomy based on visual-semantic joint driving according to any one of claims 1-8, characterized in that, After acquiring the raw chest 3D CT data, the process also includes: The voxel intensity of the chest 3D CT data is normalized and mapped to the airway sensitive window range; The chest 3D CT data was then cropped to remove the chest wall and mediastinal background, resulting in chest 3D CT data containing the region of interest in the lungs.

10. A system for automatic annotation and 3D reconstruction of lung anatomy based on vision-semantic joint driving, characterized in that, include: The data acquisition module is used to acquire raw chest 3D CT data; The lung lobe segmentation module is used to segment the lung lobes of the chest 3D CT data to obtain a lung lobe segmentation mask. The airway segmentation module is used to input the chest 3D CT data into the bronchial segmentation network model for airway segmentation and obtain the bronchial tree segmentation mask. The airway naming module is used to extract the airway skeleton based on the bronchial tree segmentation mask, extract the visual connectivity features of the airway skeleton, and combine the visual connectivity features with pre-constructed anatomical semantic information to perform multi-level naming on the bronchial tree through a visual-semantic joint-driven bronchial naming model to obtain the bronchial tree naming result. The structural segmentation and reconstruction module is used to automatically segment, label, and reconstruct the lung anatomical structure using the lung lobe segmentation mask to constrain the segmented region and the bronchial tree naming result as a guiding field.

11. An automatic annotation and 3D reconstruction device for lung anatomy based on vision-semantic joint driving, characterized in that, include: A memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the vision-semantic joint-driven automatic annotation and 3D reconstruction device for lung anatomy to perform the steps of the vision-semantic joint-driven automatic annotation and 3D reconstruction method for lung anatomy as described in any one of claims 1-9.

12. A computer-readable storage medium storing a computer program / instructions thereon, characterized in that, When the program / instruction is executed by the processor, it implements the steps of the automatic annotation and three-dimensional reconstruction method for lung anatomy based on vision-semantic joint drive as described in any one of claims 1-9.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the method for automatic annotation and three-dimensional reconstruction of lung anatomy based on visual-semantic joint drive as described in any one of claims 1-9.

Images