Small airway segmentation method and device, electronic equipment and nonvolatile storage medium

Abstract

This application discloses a method, apparatus, electronic device, and non-volatile storage medium for small airway segmentation. The method includes: segmenting a lung CT image using an image segmentation network to obtain a target region image, wherein the target region image is an image of the lung parenchyma region excluding the main airway portion; extracting features from the target region image using a convolutional neural network to obtain a first feature map; determining a first attention weight of the first feature map using a Transformer network, and determining a second feature map based on the first attention weight, wherein the first attention weight is used to characterize the degree of influence between features at various locations in the first feature map; and determining a target segmented image based on the first and second feature maps, wherein the target segmented image contains airways with a diameter smaller than a preset diameter threshold.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and non-volatile storage medium for small airway segmentation. Background Technology

[0002] Currently, mainstream navigation systems based on CT images have insufficient ability to identify small airways smaller than a preset diameter threshold, resulting in poor identification accuracy. When using navigation bronchoscopy to diagnose and treat peripheral pulmonary nodules, blind spots exist in some small airway areas, causing problems such as failure or inaccuracy in preoperative bronchoscopic path planning in some areas.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and non-volatile storage medium for small airway segmentation, enabling accurate identification of small airways. This addresses the technical problem of inaccurate bronchoscopic path planning caused by the insufficient ability of current mainstream CT image-based navigation systems to identify small airways smaller than a preset diameter threshold.

[0005] This application provides a method for small airway segmentation, comprising: segmenting a lung CT image using an image segmentation network to obtain a target region image, wherein the target region image is an image of the lung parenchyma region excluding the main airway portion, and the main airway is an airway with a diameter not less than a preset diameter threshold; extracting features from the target region image using a convolutional neural network to obtain a first feature map; determining a first attention weight of the first feature map using a Transformer network, and determining a second feature map based on the first attention weight, wherein the first attention weight is used to characterize the degree of influence between features at various positions in the first feature map; and determining a target segmented image based on the first feature map and the second feature map, wherein the target segmented image contains airways with a diameter less than a preset diameter threshold.

[0006] Optionally, the lung parenchyma region includes multiple sub-regions; determining the first attention weight corresponding to the first feature map includes: dividing the first feature map into multiple first feature sub-maps based on the sub-regions, wherein each sub-region corresponds to one first feature sub-map; and using a Transformer network to determine the first attention weight corresponding to each first feature sub-map.

[0007] Optionally, determining the second feature map based on the first attention weight includes: using the encoding module in the Transformer network to extract features from the first feature sub-map to obtain a third feature sub-map; performing linear transformations on the third feature sub-map with the query weight matrix, key weight matrix, and value weight matrix respectively to obtain the query vector, key vector, and value vector; determining the first attention weight based on the query vector and key vector; performing weighted calculations based on the first attention weight and value vector to obtain the second feature sub-map corresponding to each first feature sub-map; and concatenating the various second feature sub-maps to obtain the second feature map.

[0008] Optionally, determining the first attention weight based on the query vector and the key vector includes: using a fully connected network to perform linear transformation and nonlinear mapping on the query vector and the key vector respectively; performing inversion operations on the query vector and the key vector after linear transformation and nonlinear mapping respectively; determining the similarity between the query vector and the key vector after inversion operations, and determining the first attention weight based on the similarity.

[0009] Optionally, the image segmentation network includes: a first segmentation sub-network and a second segmentation sub-network; segmenting the lung CT image using the image segmentation network to obtain the target region image includes: using the first segmentation sub-network to determine the second attention weight of the CT image, and segmenting the CT image according to the second attention weight to obtain an airway tree trunk mask, wherein the airway tree trunk mask contains images of the main airways; using the second segmentation sub-network to segment the CT image to obtain a lung lobe mask, wherein the lung lobe mask contains various sub-regions, including: left upper lobe, left lower lobe, right upper lobe, right middle lobe, and right lower lobe; and determining the target region image based on the airway tree trunk mask, the lung lobe mask, and the CT image.

[0010] Optionally, determining the target segmentation image based on the first feature map and the second feature map includes: performing deconvolution operations on the first feature map and the second feature map respectively, so that the size of the first feature map and the second feature map are consistent with the size of the CT image; determining the small airway image based on the first feature map and the second feature map after deconvolution operation, wherein the small airway image is an image of an airway with a diameter smaller than a preset diameter threshold; and generating the target segmentation image based on the small airway image and the airway tree trunk mask.

[0011] Optionally, determining the target region image based on the airway tree trunk mask, lung lobe mask, and CT image includes: inverting the airway tree trunk mask to obtain a mask excluding the main airways; determining a lung parenchyma mask based on the lung lobe mask, wherein the lung parenchyma mask contains the lung parenchyma region; and multiplying the pixel values ​​at corresponding positions in the mask excluding the main airways, the lung parenchyma mask, and the CT image to obtain the target region image.

[0012] Optionally, before determining the target region image based on the airway tree trunk mask, lung lobe mask, and CT image, the method further includes: using a super-resolution network to perform super-resolution reconstruction on the airway tree trunk mask, lung lobe mask, and CT image, respectively, wherein the resolution of the image after super-resolution reconstruction is higher than the resolution of the image without super-resolution reconstruction.

[0013] This application embodiment also provides a small airway segmentation device, comprising: a preliminary segmentation module, used to segment a lung CT image using an image segmentation network to obtain a target region image, wherein the target region image is an image of the lung parenchyma region excluding the main airway portion, and the main airway is an airway with a diameter not less than a preset diameter threshold; a first feature extraction module, used to extract features from the target region image using a convolutional neural network to obtain a first feature map; a second feature extraction module, used to use a Transformer network to determine a first attention weight corresponding to the first feature map, and determine a second feature map based on the first attention weight, wherein the first attention weight is used to characterize the degree of influence between features at various positions in the first feature map; and a target image determination module, used to determine a target segmented image based on the first feature map and the second feature map, wherein the target segmented image contains airways with a diameter less than a preset diameter threshold.

[0014] This application also provides a non-volatile storage medium storing a computer program, which, when executed by a processor, implements a small airway segmentation method.

[0015] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a small airway segmentation method.

[0016] Based on the above scheme, this application uses an image segmentation network to segment lung CT images to obtain a target region image. The target region image is an image of the lung parenchyma region excluding the main airway, where the main airway is an airway with a diameter not less than a preset diameter threshold. A convolutional neural network is used to extract features from the target region image to obtain a first feature map. A Transformer network is used to determine the first attention weight of the first feature map, and based on the first attention weight, a second feature map is determined. The first attention weight is used to characterize the degree of influence between features at various locations in the first feature map. Based on the first and second feature maps, a target segmented image is determined, which includes airways with a diameter less than the preset diameter threshold. By improving the self-attention mechanism in the Transformer network using a graph network, the feature extraction effect of the Transformer network is optimized. Combining the convolutional neural network and the optimized Transformer network to segment and identify the lung parenchyma region in lung CT images achieves accurate identification of small airways. This solves the technical problem of inaccurate bronchoscopic path planning caused by the insufficient ability of current mainstream CT image-based navigation systems to identify small airways smaller than the preset diameter threshold. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a small airway segmentation method in an exemplary embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the structure of an image segmentation network in an exemplary embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the structure of a 3D-VNet network model with a self-attention mechanism in an exemplary embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the structure of a traditional 3D-VNet network model without a self-attention mechanism in an exemplary embodiment of this application;

[0022] Figure 5 This is a schematic diagram of a small airway segmentation network in an exemplary embodiment of this application;

[0023] Figure 6 This is a schematic diagram of an improved self-attention mechanism based on graph networks in an exemplary embodiment of this application;

[0024] Figure 7 This is a schematic diagram of a self-attention mechanism in a related technology according to an exemplary embodiment of this application;

[0025] Figure 8 This is a schematic diagram of a small airway splitting device in an exemplary embodiment of this application;

[0026] Figure 9 A flowchart illustrating a lung treatment method in an exemplary embodiment of this application;

[0027] Figure 10 This is a schematic diagram of the structure of an electronic device in an exemplary embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] The technical solutions of this application will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0031] Currently, navigation-guided bronchoscopy has limitations in the diagnosis and treatment of peripheral pulmonary nodules. Data shows that the overall diagnostic rate of peripheral pulmonary nodule biopsy using navigation-guided bronchoscopy is currently only 80%, lower than that of traditional CT-guided percutaneous lung biopsy, especially for lesions near the pleura and lesions smaller than 1 cm in diameter. The main reasons are as follows:

[0032] In the relevant technologies, the mainstream navigation system based on CT images is not capable enough to identify small airways smaller than a preset diameter threshold (in this embodiment, the preset diameter threshold is 3mm as an example). During the operation, there are blind spots in some small airway areas, which leads to failure or inaccuracy in the preoperative bronchoscopic path planning in some areas.

[0033] Therefore, the existing technologies cannot effectively process small airways smaller than a preset diameter threshold. In response to the above reasons, this application proposes an airway segmentation method, device, electronic device and non-volatile storage medium to achieve accurate identification of small airways.

[0034] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0035] This application provides a method for small airway segmentation. Figure 1 A flowchart of a small airway segmentation method provided in this application embodiment is shown below. Figure 1 As shown, the specific steps include:

[0036] Step S102: The lung CT image is segmented using an image segmentation network to obtain the target region image. The target region image is the image of the lung parenchyma region excluding the main airway. The main airway is an airway with a diameter not less than a preset diameter threshold. The lung parenchyma region includes multiple sub-regions.

[0037] Figure 2 This is a schematic diagram of the structure of an image segmentation network in an exemplary embodiment of this application, as shown below. Figure 2 As shown.

[0038] The image segmentation network described above includes: a first segmentation sub-network (i.e. Figure 2 The airway tree trunk segmentation network shown) and the second segmentation subnetwork (i.e. Figure 2 (The lung lobe segmentation network shown in the figure)

[0039] CT images typically contain a large amount of data. To improve the processing efficiency of image segmentation networks, lung CT images (data) are first segmented into blocks before using image segmentation networks to segment lung CT images, thereby reducing the computational load of the lung lobe segmentation network and the airway tree trunk segmentation network.

[0040] In some embodiments of this application, segmenting a lung CT image using an image segmentation network to obtain a target region image includes the following steps: using a first segmentation sub-network to determine a second attention weight for the CT image, and segmenting the CT image according to the second attention weight to obtain an airway tree trunk mask, wherein the airway tree trunk mask contains images of the main airways; using a second segmentation sub-network to segment the CT image to obtain a lung lobe mask, wherein the lung lobe mask contains various sub-regions, including: left upper lobe, left lower lobe, right upper lobe, right middle lobe, and right lower lobe; and determining the target region image based on the airway tree trunk mask, the lung lobe mask, and the CT image.

[0041] In this embodiment, the aforementioned first segmentation sub-network (i.e. Figure 2 The airway tree trunk segmentation network shown employs a 3D-VNet network model with a self-attention mechanism, such as... Figure 3 As shown, the traditional airway segmentation task is transformed into a connectivity prediction problem. The network's expressive power is enhanced by adding a self-attention mechanism. The formula for this self-attention mechanism is shown below:

[0042]

[0043] in, This represents the value of the attention received. For activation functions, Q, K, and V are the query vector matrix, key vector matrix, and value vector matrix, respectively. Each row of these three matrices represents a corresponding vector. Q, K, and V are generally generated by processing the input sequence X (i.e., ... Figure 3 The output of each convolutional layer shown is multiplied by the query weight matrix. Key weight matrix Value weight matrix We obtain the formula, The calculated result is the second attention weight mentioned above.

[0044] Through the first segmented subnetwork (i.e. Figure 2 The airway tree trunk segmentation network shown in the figure segments and identifies the lung CT image after it is divided into blocks, and obtains the airway tree trunk segmentation mask (i.e. the airway tree trunk mask above). The airway tree trunk (main airway) included therein refers to an airway with a diameter greater than or equal to 3 mm (preset diameter threshold).

[0045] In this embodiment, the aforementioned second segmentation sub-network (i.e. Figure 2 The lung lobe segmentation network shown uses a traditional 3D-VNet network model without a self-attention mechanism, such as... Figure 4As shown, the lung CT image after being segmented is segmented and identified by the second segmentation network to obtain the lung lobe segmentation mask (i.e. the lung lobe mask above). The lung lobe segmentation mask is marked with the left upper lobe, left lower lobe, right upper lobe, right middle lobe and right lower lobe respectively. The sub-regions of the left upper lobe, left lower lobe, right upper lobe, right middle lobe and right lower lobe are collectively referred to as the lung parenchyma region.

[0046] It should be noted that, as some optional implementation methods, the first segmentation sub-network (airway tree trunk segmentation network) and the second segmentation sub-network (lung lobe segmentation network) can both adopt a 3D-VNet network with self-attention mechanism, thereby making the segmentation results of the airway tree trunk and the lung lobe segmentation results more accurate, but requiring more computational resources; or, the first segmentation sub-network (airway tree trunk segmentation network) and the second segmentation sub-network (lung lobe segmentation network) can both be traditional 3D-VNet network models without self-attention mechanism, thereby saving computational resources and improving computational speed.

[0047] In this embodiment, the preferred approach is that the first segmentation sub-network (airway tree trunk segmentation network) adopts a 3D-VNet network with self-attention mechanism, and the second segmentation sub-network (lung lobe segmentation network) adopts a traditional 3D-VNet network without self-attention mechanism. This approach ensures the accuracy of the segmentation results of the airway tree trunk while saving computational resources, thereby achieving a balance between computational accuracy and computational resource consumption.

[0048] After obtaining the airway tree trunk segmentation mask (i.e., the airway tree trunk mask) and lung lobe segmentation mask (i.e., the lung lobe mask) through the above steps, it is necessary to combine the original CT image of the lung to determine the lung parenchyma region (target region image) excluding the main airway in the original CT image, so as to further segment and identify the small airways in the image.

[0049] In some embodiments of this application, determining the target region image based on the airway tree trunk mask, the lung lobe mask, and the CT image includes the following steps: inverting the airway tree trunk mask to obtain a mask excluding the main airways; determining the lung parenchyma mask based on the lung lobe mask, wherein the lung parenchyma mask contains the lung parenchyma region; and multiplying the pixel values ​​at corresponding positions in the mask excluding the main airways, the lung parenchyma mask, and the CT image to obtain the target region image.

[0050] Specifically, the airway tree trunk segmentation mask (i.e., the airway tree trunk mask image mentioned above) is inverted to obtain a mask without the airway tree trunk (i.e., the mask image without the main airway). Based on the identification information of each part in the lung lobe segmentation mask (i.e., the lung lobe mask image mentioned above, such as the left upper lobe, left lower lobe, right upper lobe, right middle lobe, and right lower lobe), the lung lobe segmentation mask (i.e., the lung lobe mask image mentioned above) is processed to obtain a lung parenchyma mask (i.e., the mask image without the main airway). Then, a multiplication operation is performed on the airway tree trunk mask (i.e., the mask image without the main airway), the lung parenchyma mask (i.e., the mask image without the main airway), and the original CT image to obtain an image of the lung parenchyma region in the original CT image excluding the main airway (i.e., the target region image mentioned above). The pixel values ​​of all parts of the lung parenchyma region excluding the main airway in this image are set to 0.

[0051] To further improve the accuracy of small airway identification, as an optional implementation, before determining the target region image based on the airway tree trunk mask, lung lobe mask, and CT image, the method may also include the following steps: using a super-resolution network to perform super-resolution reconstruction on the airway tree trunk mask, lung lobe mask, and CT image, respectively, wherein the resolution of the image after super-resolution reconstruction is higher than the resolution of the image without super-resolution reconstruction.

[0052] Specifically, super-resolution networks (including Figure 2 The super-resolution network (airway tree super-resolution network, lung lobe super-resolution network, and CT super-resolution network shown) performs super-resolution reconstruction on the airway tree trunk mask (airway tree trunk segmentation mask), lung lobe mask (lung lobe segmentation mask), and CT image respectively to improve the resolution of the airway tree trunk segmentation mask, lung lobe segmentation mask, and original CT, thereby improving the accuracy of subsequent processing. The super-resolution network consists of an encoder, a learnable upsampling layer, and a decoder.

[0053] After obtaining the target region image through the above steps, it is necessary to further segment and identify the small airways in the image. Specifically, the lung parenchyma region excluding the main airway (i.e., the target region image) is used as the input of the small airway segmentation network to perform small airway segmentation.

[0054] When performing small airway segmentation, this application employs two types of networks to process the target region image to improve the accuracy of small airway recognition. One type is based on convolutional neural networks (CNNs), such as ADAUNet, and the other is based on Transformer networks, such as UNERT (a complete Transformer network in the encoder stage and a deconvolutional network in the decoder stage) and Swin-UNet (a complete Transformer network in both the encoder and decoder stages). The advantage of CNNs is that they have fewer parameters and are easier to train, but they cannot capture long-range dependencies due to their limited receptive field. The advantage of Transformers is that they can capture long-range dependencies, but they have more parameters and are more difficult to train.

[0055] The small airway segmentation network in this application adopts a structure of CNN small modules + Transformer small modules, which improves the accuracy of small airway segmentation and recognition while controlling the training difficulty of the entire network. This will be further introduced below.

[0056] Figure 5 This is a schematic diagram of a small airway segmentation network in an exemplary embodiment of this application, as shown below. Figure 5 As shown, convolutional modules and Transformer encoding modules are used to extract features from the target region image. In the decoder part of the small airway segmentation network, Transformer decoding modules and deconvolution modules are used. The specific steps are as follows.

[0057] Step S104: Use a convolutional neural network to extract features from the target region image to obtain the first feature map;

[0058] Step S106: Using a Transformer network, determine the first attention weight of the first feature map, and determine the second feature map based on the first attention weight, wherein the first attention weight is used to characterize the degree of influence between features at each position in the first feature map;

[0059] In some embodiments of this application, determining the first attention weight corresponding to the first feature map includes the following steps: dividing the first feature map into multiple first feature sub-maps based on the sub-region, wherein each sub-region corresponds to one first feature sub-map; and using a Transformer network to determine the first attention weight corresponding to each first feature sub-map.

[0060] Specifically, the lung parenchyma region excluding the main airway (i.e., the target region image mentioned above) is input into the convolution module (convolutional neural network) for convolution processing to obtain the first feature map. Then, before inputting it into the Transformer module (Transformer network), the lung lobe region block information in the lung lobe segmentation mask is combined to map the features output by the convolution module (i.e., the first feature map mentioned above) to different lung lobe regions (i.e., the sub-regions mentioned above: left upper lobe, left lower lobe, right upper lobe, right middle lobe, and right lower lobe) to obtain the first feature sub-map corresponding to each sub-region. Then, the features corresponding to different lung lobes (i.e., the first feature sub-maps mentioned above) are input into the Transformer module (Transformer network) for processing.

[0061] Before employing the Transformer module (Transformer network), a convolutional neural network is used first. This facilitates feature extraction, reduces the computational load of the subsequent Transformer module, and lowers the overall training difficulty of the network. The aforementioned Transformer module (Transformer network) includes... Figure 5 The Transformer encoding module, graph attention module, and Transformer decoding module are shown in the diagram.

[0062] It should be noted that after using a convolutional neural network to extract features and obtain the first feature map mentioned above, before inputting the first feature map into the Transformer module (Transformer network), the processing of the first feature map is not limited to the method of combining the lung lobe region block information in the lung lobe segmentation mask to map the features output by the convolutional module (i.e., the first feature map mentioned above) to different lung lobe regions (i.e., the sub-regions mentioned above). In other words, it is not limited to the method of dividing the first feature map into multiple first feature sub-maps based on the sub-regions.

[0063] As an alternative implementation, at least one option is to not divide the first feature map and directly input the entire first feature map into the Transformer module (Transformer network) to perform subsequent execution steps; or, another option is to divide the first feature map into a preset number of first feature sub-maps and then input the first feature sub-maps into the Transformer module (Transformer network) in sequence to perform subsequent execution steps.

[0064] In some embodiments of this application, determining the second feature map based on the first attention weight includes the following steps: using the encoding module in the Transformer network to extract features from the first feature sub-map to obtain a third feature sub-map; performing linear transformations on the third feature sub-map with the query weight matrix, key weight matrix, and value weight matrix respectively to obtain the query vector, key vector, and value vector; determining the first attention weight based on the query vector and key vector; performing weighted calculations based on the first attention weight and value vector to obtain the second feature sub-map corresponding to each first feature sub-map; and concatenating the various second feature sub-maps to obtain the second feature map.

[0065] In some embodiments of this application, determining the first attention weight based on the query vector and the key vector includes the following steps: using a fully connected network to perform linear transformation and nonlinear mapping on the query vector and the key vector respectively; performing inversion operations on the query vector and the key vector after linear transformation and nonlinear mapping respectively; determining the similarity between the query vector and the key vector after the inversion operation, and determining the first attention weight based on the similarity.

[0066] Specifically, after extracting features using a convolutional neural network to obtain the first feature sub-map, a Transformer module encoder and a Transformer module decoder are then used in combination. Figure 5 The graph attention module shown in the diagram further extracts features from the first feature sub-map in sequence, which can capture the long-distance dependence of CT and obtain more effective features (i.e., the second feature sub-map mentioned above). The second feature map is obtained by stitching together the second feature sub-maps of each sub-region.

[0067] Specifically, for the first feature sub-map of the input, a multi-layer Transformer encoding module is used to extract features from it to obtain a third feature sub-map. Each Transformer encoding module corresponds to a third feature sub-map, and the third feature sub-map extracted by each Transformer encoding module is different (e.g., the size of the feature map is different). As the number of layers increases, the encoding module can better capture the long-range dependencies between different positions in the input feature map, thereby improving the encoding module's ability to model different feature subspaces.

[0068] After obtaining the third feature sub-map, the attention weights are calculated using the corresponding graph attention module of the encoding module at that layer. Finally, the second feature sub-map is obtained by fusing the second feature sub-maps output by the Transformer decoding modules at each layer based on the attention weights.

[0069] In this embodiment, the graph attention module employs a self-attention mechanism based on graph networks, such as... Figure 6 As shown.

[0070] The formula for the autonomous power mechanism is shown below:

[0071]

[0072] in, This represents the value of the attention received. For activation functions, Q, K, and V are the query vector matrix, key vector matrix, and value vector matrix, respectively. Each row of these three matrices represents a corresponding vector. Q, K, and V are typically generated by processing the input sequence X ( Figure 5 The outputs of the convolutional module or Transformer encoding module shown are multiplied by the query weight matrix respectively. Key weight matrix Value weight matrix get.

[0073] Based on the message passing formula of graph networks, the self-attention mechanism is improved, resulting in the improved self-attention mechanism, the formula of which is shown below:

[0074]

[0075] Where F represents Figure 6 The fully connected network MLP shown in the figure, The sign indicates Figure 6 The inversion module (inversion operation) shown in this embodiment can be implemented using a one-dimensional fully connected network since Q, K, and V are all vectors.

[0076] Compared to self-attention mechanisms in related technologies, the improved self-attention mechanism based on graph networks, such as... Figure 7 As shown, by adding a fully connected network and an inverted network (module), the features that need to be highlighted can be strengthened, enabling the small airway segmentation network to better capture the distribution pattern and long-distance dependence of small airway segmentation label values.

[0077] Step S108: Determine the target segmentation image based on the first feature map and the second feature map, wherein the target segmentation image contains airways with a diameter smaller than a preset diameter threshold.

[0078] In some embodiments of this application, determining the target segmentation image based on the first feature map and the second feature map includes the following steps: performing deconvolution operations on the first feature map and the second feature map respectively, so that the size of the first feature map and the second feature map are consistent with the size of the CT image; determining the small airway image based on the first feature map and the second feature map after deconvolution operation, wherein the small airway image is an image of an airway with a diameter smaller than a preset diameter threshold; and generating the target segmentation image based on the small airway image and the airway tree trunk mask.

[0079] Specifically, deconvolution is used to match the CT dimension, ensuring that the dimensions of both the first and second feature maps are consistent with the CT image dimensions. This yields segmentation results for multiple organs and their complex vascular structures. Skip connections and a U-shaped structure are used to integrate the effective features of each layer to determine the small airway image. Combined with the previously obtained airway tree trunk mask, the final target segmentation image is generated.

[0080] Through the above steps, the self-attention mechanism in the transformer network is improved by using a graph network, which optimizes the feature extraction effect of the transformer network. By combining a convolutional neural network and the optimized transformer network, the lung parenchyma region in the lung CT image is segmented and identified, achieving the goal of accurately identifying small airways with a diameter of less than 3 mm. Thus, the technical problem of inaccurate bronchoscopic path planning caused by the insufficient recognition capability of current mainstream navigation systems based on CT images for airways smaller than 3 mm is solved.

[0081] This application also provides an embodiment of a small airway splitting device. Figure 8 This is a schematic diagram of a small airway splitting device according to an exemplary embodiment of this application. The device includes:

[0082] The preliminary segmentation module 80 is used to segment the lung CT image using an image segmentation network to obtain the target region image, wherein the target region image is the image of the lung parenchyma region excluding the main airway, and the main airway is an airway with a diameter not less than a preset diameter threshold.

[0083] In some embodiments of this application, segmenting a lung CT image using an image segmentation network to obtain a target region image includes the following steps: using a first segmentation sub-network to determine a second attention weight for the CT image, and segmenting the CT image according to the second attention weight to obtain an airway tree trunk mask, wherein the airway tree trunk mask contains images of the main airways; using a second segmentation sub-network to segment the CT image to obtain a lung lobe mask, wherein the lung lobe mask contains various sub-regions, including: left upper lobe, left lower lobe, right upper lobe, right middle lobe, and right lower lobe; and determining the target region image based on the airway tree trunk mask, the lung lobe mask, and the CT image.

[0084] In some embodiments of this application, determining the target region image based on the airway tree trunk mask, the lung lobe mask, and the CT image includes the following steps: inverting the airway tree trunk mask to obtain a mask excluding the main airways; determining the lung parenchyma mask based on the lung lobe mask, wherein the lung parenchyma mask contains the lung parenchyma region; and multiplying the pixel values ​​at corresponding positions in the mask excluding the main airways, the lung parenchyma mask, and the CT image to obtain the target region image.

[0085] Before determining the target region image based on the airway tree trunk mask, lung lobe mask, and CT image, the preliminary segmentation module 80 is further used to: employ a super-resolution network to perform super-resolution reconstruction on the airway tree trunk mask, lung lobe mask, and CT image, respectively, wherein the resolution of the image after the super-resolution reconstruction operation is higher than the resolution of the image without the super-resolution reconstruction operation.

[0086] The first feature extraction module 82 is used to extract features from the target region image using a convolutional neural network to obtain a first feature map;

[0087] The second feature extraction module 84 is used to use a Transformer network to determine the first attention weight of the first feature map and to determine the second feature map based on the first attention weight, wherein the first attention weight is used to characterize the degree of influence between features at each position in the first feature map;

[0088] In some embodiments of this application, determining the first attention weight corresponding to the first feature map includes the following steps: dividing the first feature map into multiple first feature sub-maps based on the sub-region, wherein each sub-region corresponds to one first feature sub-map; and using a Transformer network to determine the first attention weight corresponding to each first feature sub-map.

[0089] In some embodiments of this application, determining the second feature map based on the first attention weight includes the following steps: using the encoding module in the Transformer network to extract features from the first feature sub-map to obtain a third feature sub-map; performing linear transformations on the third feature sub-map with the query weight matrix, key weight matrix, and value weight matrix respectively to obtain the query vector, key vector, and value vector; determining the first attention weight based on the query vector and key vector; performing weighted calculations based on the first attention weight and value vector to obtain the second feature sub-map corresponding to each first feature sub-map; and concatenating the various second feature sub-maps to obtain the second feature map.

[0090] In some embodiments of this application, determining the first attention weight based on the query vector and the key vector includes the following steps: using a fully connected network to perform linear transformation and nonlinear mapping on the query vector and the key vector respectively; performing inversion operations on the query vector and the key vector after linear transformation and nonlinear mapping respectively; determining the similarity between the query vector and the key vector after the inversion operation, and determining the first attention weight based on the similarity.

[0091] The target image determination module 86 is used to determine a target segmentation image based on the first feature map and the second feature map, wherein the target segmentation image contains airways with a diameter smaller than a preset diameter threshold.

[0092] In some embodiments of this application, determining the target segmentation image based on the first feature map and the second feature map includes the following steps: performing deconvolution operations on the first feature map and the second feature map respectively, so that the size of the first feature map and the second feature map are consistent with the size of the CT image; determining the small airway image based on the first feature map and the second feature map after deconvolution operation, wherein the small airway image is an image of an airway with a diameter smaller than a preset diameter threshold; and generating the target segmentation image based on the small airway image and the airway tree trunk mask.

[0093] It should be noted that each module in the above-mentioned small airway segmentation device can be a program module (for example, a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0094] It should be noted that the small airway splitting device provided in this embodiment can be used to perform... Figure 1 The small airway segmentation method shown above is also applicable to the embodiments of this application, and will not be repeated here.

[0095] This application also provides an embodiment of a lung treatment method. Figure 9 This is a flowchart of a lung treatment method in an exemplary embodiment of this application, such as... Figure 9 As shown, the method includes the following steps:

[0096] Step S902: The lung CT image is segmented using an image segmentation network to obtain the target region image. The target region image is the image of the lung parenchyma region excluding the main airway. The main airway is an airway with a diameter not less than a preset diameter threshold.

[0097] Step S904: Use a convolutional neural network to extract features from the target region image to obtain the first feature map;

[0098] Step S906: Using a Transformer network, determine the first attention weight of the first feature map, and determine the second feature map based on the first attention weight, wherein the first attention weight is used to characterize the degree of influence between features at each position in the first feature map;

[0099] Step S908: Determine the target segmentation image based on the first feature map and the second feature map, wherein the target segmentation image contains airways with a diameter smaller than a preset diameter threshold.

[0100] Step S910: Based on the target segmentation image, generate prompt information, wherein the prompt information is used to assist in the path planning of the airway tube in order to achieve the treatment of lung diseases.

[0101] It should be noted that the lung treatment method provided in this embodiment is similar to... Figure 1 The method embodiments corresponding to the small airway segmentation method shown are as described above. Therefore, the relevant explanations and descriptions of the above small airway segmentation method also apply to the embodiments of this application, and will not be repeated here.

[0102] This application also provides an electronic device. Figure 10 This is a schematic diagram of the structure of an electronic device in an exemplary embodiment of this application. Figure 10 As shown, the electronic device includes a memory 1001 and a processor 1002.

[0103] The memory 1001 stores an executable computer program 1003. The processor 1002, coupled to the memory 1001, calls the executable computer program 1003 stored in the memory to execute the small airway segmentation method provided in the above embodiment.

[0104] For example, the computer program 1003 can be divided into one or more modules / units, which are stored in the memory 1001 and executed by the processor 1002 to complete this application. The one or more modules / units may include various modules in the small airway segmentation device in the above embodiments, such as: a preliminary segmentation module, a first feature extraction module, a second feature extraction module, and a target image determination module.

[0105] Furthermore, the device also includes at least one input device and at least one output device.

[0106] The processor 1002, memory 1001, input devices, and output devices mentioned above can be connected via a bus.

[0107] The input device can be a camera, touch panel, physical buttons, or mouse, etc. The output device can be a display screen.

[0108] Furthermore, the device may include more components than illustrated, or combine certain components, or different components, such as network access devices, sensors, etc.

[0109] The processor 1002 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0110] The memory 1001 can be, for example, a hard disk drive, non-volatile memory (such as flash memory or other electronically programmable erasure-restricted memory used to form a solid-state drive), volatile memory (such as static or dynamic random access memory), etc., and this application embodiment is not limited thereto. Specifically, the memory 1001 can be an internal storage unit of the electronic device, such as the hard disk or RAM of the electronic device. The memory 1001 can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device. Further, the memory 1001 can also include both internal storage units and external storage devices of the electronic device. The memory 1001 is used to store computer programs and other programs and data required by the terminal. The memory 1001 can also be used to temporarily store data that has been output or will be output.

[0111] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the electronic device of the above embodiments, and the computer-readable storage medium may be the aforementioned... Figure 10 The memory 1001 in the illustrated embodiment stores a computer program on the computer-readable storage medium. When executed by a processor, the program implements the image-based target position adjustment method described in the foregoing embodiments. Furthermore, the computer-readable storage medium can also be various media capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0113] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0115] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, external hard drives, ROM, RAM, magnetic disks, or optical disks.

[0116] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0117] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0118] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for small airway segmentation, characterized in that, include: An image segmentation network is used to segment lung CT images to obtain target region images. The target region images are images of the lung parenchyma region excluding the main airway. The main airway is an airway with a diameter not less than a preset diameter threshold. The image segmentation network includes: a first segmentation subnetwork and a second segmentation subnetwork. The process of segmenting a lung CT image using an image segmentation network to obtain a target region image includes: using a first segmentation sub-network to determine a second attention weight for the CT image, and segmenting the CT image according to the second attention weight to obtain an airway tree trunk mask; using the second segmentation sub-network to segment the CT image to obtain a lung lobe mask, wherein the lung lobe mask contains multiple sub-regions, including: left upper lobe, left lower lobe, right upper lobe, right middle lobe, and right lower lobe; and determining the target region image based on the airway tree trunk mask, the lung lobe mask, and the CT image. The determination of the target region image based on the airway tree trunk mask, the lung lobe mask, and the CT image includes: inverting the airway tree trunk mask to obtain a mask excluding the main airways; determining a lung parenchyma mask based on the lung lobe mask, wherein the lung parenchyma mask includes the lung parenchyma region; and multiplying the pixel values ​​at corresponding positions in the mask excluding the main airways, the lung parenchyma mask, and the CT image to obtain the target region image. A convolutional neural network is used to extract features from the target region image to obtain a first feature map; A Transformer network is used to determine the first attention weight corresponding to the first feature map, and a second feature map is determined based on the first attention weight, wherein the first attention weight is used to characterize the degree of influence between features at each position in the first feature map; Determining a target segmentation image based on the first feature map and the second feature map includes: performing deconvolution operations on the first feature map and the second feature map respectively, so that the sizes of the first feature map and the second feature map are consistent with the size of the CT image; determining small airway images based on the first feature map and the second feature map after the deconvolution operation, wherein the small airway images are images of airways with a diameter smaller than the preset diameter threshold; generating the target segmentation image based on the small airway images and the airway tree trunk mask, wherein the airway tree trunk mask contains images of the main airways, and the target segmentation image contains airways with a diameter smaller than the preset diameter threshold.

2. The small airway segmentation method according to claim 1, characterized in that, The lung parenchyma region includes multiple sub-regions; determining the first attention weight corresponding to the first feature map includes: Based on the sub-regions, the first feature map is divided into multiple first feature sub-maps, wherein each sub-region corresponds to one first feature sub-map; The Transformer network is used to determine the first attention weights corresponding to each of the first feature sub-graphs.

3. The small airway segmentation method according to claim 2, characterized in that, Based on the first attention weight, the second feature map is determined as follows: The first feature sub-map is extracted using the encoding module in the Transformer network to obtain the third feature sub-map; The third feature subgraph is linearly transformed with the query weight matrix, key weight matrix and value weight matrix respectively to obtain the query vector, key vector and value vector; The first attention weight is determined based on the query vector and the key vector; The second feature sub-maps corresponding to each first feature sub-map are obtained by weighting the first attention weight and the value vector. The individual second feature sub-images are concatenated to obtain the second feature image.

4. The small airway segmentation method according to claim 3, characterized in that, Determining the first attention weight based on the query vector and the key vector includes: A fully connected network is used to perform linear transformations and nonlinear mappings on the query vector and the key vector, respectively. The query vector and the key vector, after the linear transformation and nonlinear mapping, are respectively inverted; The similarity between the query vector and the key vector after the inversion operation is determined, and the first attention weight is determined based on the similarity.

5. The small airway segmentation method according to claim 2, characterized in that, Before determining the target region image based on the airway tree trunk mask, the lung lobe mask, and the CT image, the method further includes: A super-resolution network is used to perform super-resolution reconstruction on the airway tree trunk mask, the lung lobe mask, and the CT image, respectively. The resolution of the image after the super-resolution reconstruction operation is higher than that of the image without the super-resolution reconstruction operation.

6. A small airway splitting device, characterized in that, include: The preliminary segmentation module is used to segment lung CT images using an image segmentation network to obtain target region images. The target region images are images of lung parenchyma regions excluding the main airway portion. The main airway is an airway with a diameter not less than a preset diameter threshold. The image segmentation network includes: a first segmentation subnetwork and a second segmentation subnetwork. The process of segmenting a lung CT image using an image segmentation network to obtain a target region image includes: using a first segmentation sub-network to determine a second attention weight for the CT image, and segmenting the CT image according to the second attention weight to obtain an airway tree trunk mask; using the second segmentation sub-network to segment the CT image to obtain a lung lobe mask, wherein the lung lobe mask contains multiple sub-regions, including: left upper lobe, left lower lobe, right upper lobe, right middle lobe, and right lower lobe; and determining the target region image based on the airway tree trunk mask, the lung lobe mask, and the CT image. The determination of the target region image based on the airway tree trunk mask, the lung lobe mask, and the CT image includes: inverting the airway tree trunk mask to obtain a mask excluding the main airways; determining a lung parenchyma mask based on the lung lobe mask, wherein the lung parenchyma mask includes the lung parenchyma region; and multiplying the pixel values ​​at corresponding positions in the mask excluding the main airways, the lung parenchyma mask, and the CT image to obtain the target region image. The first feature extraction module is used to extract features from the target region image using a convolutional neural network to obtain a first feature map; The second feature extraction module is used to use a Transformer network to determine the first attention weight corresponding to the first feature map, and to determine the second feature map based on the first attention weight, wherein the first attention weight is used to characterize the degree of influence between features at each position in the first feature map; The target image determination module is used to determine a target segmentation image based on the first feature map and the second feature map, including: performing deconvolution operations on the first feature map and the second feature map respectively, so that the size of the first feature map and the second feature map are consistent with the size of the CT image; determining small airway images based on the first feature map and the second feature map after the deconvolution operation, wherein the small airway images are images of airways with a diameter smaller than the preset diameter threshold; generating the target segmentation image based on the small airway images and the airway tree trunk mask, wherein the airway tree trunk mask contains images of the main airways, and the target segmentation image contains airways with a diameter smaller than the preset diameter threshold.

7. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, which, when executed by a processor, implements the small airway segmentation method according to any one of claims 1 to 5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the small airway segmentation method as described in any one of claims 1 to 5.

Images