Pulmonary embolism detection system combining CT pulmonary arteriography and neural network

Abstract

The invention discloses a pulmonary embolism detection system combining CT pulmonary arteriography and a neural network, and relates to the technical field of angiography image processing. Comprising five parts, namely CT pulmonary arteriography data collection, data set annotation and division, CT pulmonary arteriography image enhancement, data set amplification and data normalization, and is used for processing a patient image, wherein the CT pulmonary arteriography sequence image cleaning and enhancement network is composed of five parts, namely CT pulmonary arteriography data collection, data set annotation and division, CT pulmonary arteriography image enhancement, data set amplification and data normalization; the CT pulmonary arteriography multi-scale feature extraction network is composed of a residual transformation network and a bidirectional feature pyramid network and is used for extracting data features; the pulmonary embolus potential position detection network is composed of an independent self-attention mechanism and is used for detecting the position area of pulmonary embolus; the pulmonary embolism type identification and positioning network is composed of two identification modules connected in series and is used for pulmonary embolism type judgment and position positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of contrast image processing, in particular to a pulmonary embolism detection system combining CT pulmonary angiography and neural network. BACKGROUND

[0002] Pulmonary embolism refers to a clinical pathophysiological syndrome caused by endogenous or exogenous embolus blocking pulmonary artery or its branches, leading to pulmonary circulation disorder. In common parlance, common thrombus, fat, air and other "embolus" reach the lung with blood flow, block the pulmonary artery, and cause pulmonary circulation disorder and a series of pathological reactions. Due to the similarity of the clinical features of pulmonary embolism and other respiratory diseases, misdiagnosis and missed diagnosis often occur.

[0003] Hospitals mainly use CT pulmonary angiography (CTPA) technology to diagnose pulmonary embolism. As the gold standard for the diagnosis of pulmonary vascular disease, CTPA highlights the pulmonary artery and its branches by intravenous injection of high-density contrast agent. Since the contrast agent is dissolved in the blood, the blood vessels are very bright, and the embolus does not absorb the contrast agent, so the embolus appears as a dark area in the bright pulmonary artery area in the CTPA image. In clinical practice, pulmonary embolism appears as a low-density filling defect (HU value about -50~100) in CTPA, which forms a density difference with the surrounding enhanced pulmonary artery (HU value about 200~300). Usually, radiologists rely on manual reading to diagnose pulmonary embolism, and the lung structure is usually composed of hundreds of CTPA images, each image representing a lung image. Manual reading to diagnose pulmonary embolism is a very time-consuming and labor-intensive task, and the accuracy of the judgment is also affected by subjective factors such as the doctor's subjective experience and attention.

[0004] Therefore, an accurate pulmonary embolism auxiliary detection technology is needed to help doctors accurately identify and diagnose pulmonary embolism, and to minimize the probability of misdiagnosis and missed diagnosis, and to save lives. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a pulmonary embolism detection system combining CT pulmonary angiography and neural network, which realizes the detection, identification and positioning of pulmonary embolism.

[0006] The present application adopts the following technical scheme to achieve the purpose of the application: A pulmonary embolism detection system combining CT pulmonary angiography and neural network, characterized in that it comprises: a CT pulmonary angiography sequence image cleaning and enhancement network, a CT pulmonary angiography multi-scale feature extraction network, a pulmonary embolus potential position detection network, and a pulmonary embolism type identification and positioning network. The CT pulmonary angiography sequence image cleaning and enhancement network specifically comprises five parts: CT pulmonary angiography data collection, data set labeling and division, CT pulmonary angiography image enhancement, data set expansion, and data normalization. The CT pulmonary angiography multi-scale feature extraction network is composed of a residual transformation network and a bidirectional feature pyramid network. The pulmonary embolus potential position detection network is composed of multiple independent self-attention mechanisms. The pulmonary embolism type identification and positioning network comprises multiple identification modules connected in series.

[0007] As a further limitation of the technical solution, the CT pulmonary angiography image enhancement step is: S111: Load the CT pulmonary angiography sequence image, and determine the center point coordinates of the lung blood vessel region; S112: Take the center point as the origin, and perform cropping based on the open source computer vision library to crop the original 512x512 size image to 319x319 size; S113: The cropped image is enlarged to the original size by bilinear interpolation, and image enhancement is completed.

[0008] The data set expansion step is: S121: Perform horizontal mirroring, vertical mirroring, and left-right rotation by 90 degrees on the CT pulmonary angiography image to generate images of various angles and shapes; S122: The expanded images and the original images together form a new training set and a validation set.

[0009] As a further limitation of the technical solution, the data normalization step is: S131: Obtain the pixel matrix of the CT pulmonary angiography image; S132: Calculate the maximum and minimum values of the pixels; S133: Normalize the pixel values to the [0, 1] interval through normalization operation to optimize the input data distribution of the model.

[0010] As a further limitation of the technical solution, the residual transformation network introduces a grouped convolution technique based on a residual network, which improves the model's detection capability for small pulmonary emboli without significantly increasing the model's computation and parameter amounts through grouped convolution; The residual transformation network is stacked by multiple residual blocks, each of which is composed of three convolution layers, wherein the second convolution layer adopts a grouped convolution technique to divide the input channels into multiple groups for convolution operation, and then splices the outputs of each group.

[0011] As a further limitation of the technical solution, the bidirectional feature pyramid network is constructed on the basis of the output feature map of the residual transformation network residual block group, and contains a top-down path and a bottom-up path; S21: The output feature maps of the residual transformation network residual block group 2-5 are processed respectively, the channel number is adjusted through 1×1 convolution, the channel number is unified to 256, and then the upsampling operation is performed to gradually restore the feature map size; S22: The output feature maps of each stage of the residual transformation network are subjected to 1×1 convolution to adjust the channel number, and then subjected to downsampling operation to gradually reduce the feature map size; S23: For each level of the bidirectional feature pyramid network, the feature maps of the bidirectional paths are adjusted, and a learnable weight parameter is assigned to the feature maps of each path, the weight parameter is normalized by the Softmax function, so that its value is distributed between [0, 1], and the sum of the weights of all paths is 1; S24: The feature maps of each path are multiplied by their corresponding weights, and then element-wise addition is performed to obtain the final fusion feature map, the specific formula is as follows: (1); Wherein: The output feature map of the residual transformation network residual block group 2 is taken as the input of the bidirectional feature pyramid network; The intermediate process output of the second layer from top to bottom of the bidirectional feature pyramid network; is a learnable weight parameter of the network, is a very small constant to prevent the denominator from being 0; is a downsampling operation, is a 1×1 convolution operation, is the output of the bidirectional feature pyramid network corresponding to the residual transformation network residual block group 2; (2); Wherein, The output feature map of the residual transformation network residual block group 3 is taken as the input of the bidirectional feature pyramid network; is a learnable weight parameter of the network; is the output of the bidirectional feature pyramid network corresponding to the residual transformation network residual block group 3; (3); Wherein: The output feature map of the residual transformation network residual block group 4 is taken as the input of the bidirectional feature pyramid network, The intermediate process output of the first layer from top to bottom of the bidirectional feature pyramid network, is a learnable weight parameter of the network, is the output of the residual block group 4 of the residual transform network corresponding to the bidirectional feature pyramid network; (4); wherein: represents the output feature map of the residual block group 5 of the residual transform network, as the input of the bidirectional feature pyramid network, is a network learnable weight parameter, is the output of the residual block group 5 of the residual transform network corresponding to the bidirectional feature pyramid network.

[0012] As a further limitation of the technical solution, the independent self-attention mechanism calculation step is: S311: extracting a local neighborhood centered on a pixel point from the input feature map, represents the input feature dimension, and the query vector and the key matrix and value matrix of each pixel point on the local neighborhood are obtained through the attention kernel; S312: obtaining the output center pixel of the center pixel by performing pixel-by-pixel dot product and summation on the weight matrix and the value matrix, that is, one pixel of the output feature map; S313: adding the relative distance information of each pixel point in the softmax matrix of the independent self-attention mechanism, defined as the relative distance from the center pixel to other pixel points in the neighborhood; For a two-dimensional feature map, there are two dimensions of relative distance between two pixel points, row offset and column offset The formula for calculating the independent self-attention mechanism is as follows: (5); (6); (7); (8); wherein: represents the size of the square local neighborhood; are the query attention kernel, the key attention kernel and the value attention kernel, respectively, used to map the pixel in the local neighborhood to the query vector , the key vector and the value vector , represents the output feature dimension; represents the row offset relative distance, represents the column offset relative distance; is a learnable row offset position embedding, is a learnable column offset position embedding; is obtained by matrix multiplication and summation with the value vector after matrix multiplication and summation, the output of the local neighborhood is obtained , and the same operation is sequentially performed on other local neighborhoods of the input feature map to obtain the final output feature map.

[0013] As a further limitation of the technical solution, the pulmonary embolus potential position detection network uses the 4-scale feature maps output by the upper-level network as input, and each scale of feature map is subjected to the same detection processing. The following takes a 128x128x256 feature map as an example for description, and the steps are as follows: S321: 4 square detection boxes of different sizes are set for each pixel point of the CT pulmonary angiography image , and the side length is used to preliminarily predict the potential position of the pulmonary embolus; S322: The feature map is subjected to an independent self-attention mechanism layer with a local neighborhood size K=3x3 and a channel number of 512 to complete CT pulmonary angiography long-distance spatiotemporal structure modeling; S323: Two parallel independent self-attention mechanism layers are used, wherein the attention layer of the upper branch is used for target classification, i.e., whether there is a pulmonary embolus in the detection box. Since there are 4 detection boxes for each detection point and each detection box has two probability values , the output corresponding to each detection point is an 8-dimensional vector, i.e., the upper branch has 8 channels; the attention layer of the lower branch is used to obtain the offset of the detection box position relative to the true position, and each detection box has three offset values , so the output corresponding to each detection point is a 12-dimensional vector, i.e., the lower branch has 12 channels; S324: In the training stage, 512 detection boxes are selected for classification and position regression, and the selection process is as follows: For each real label, select a detection box with the largest intersection-over-union value as a positive sample; Then select a detection box with an intersection-over-union value >0.7 with any real label as a positive sample from the remaining detection boxes, and the total number of positive samples is 256; Finally, randomly select a detection box with an intersection-over-union value <0.3 with any real label as a negative sample, and the remaining detection boxes do not participate in training. The total number of negative samples and positive samples is 512; The objective function used is as follows: (9); Wherein: is the entire objective function; is the binary cross-entropy loss; to smooth loss; denotes the bounding box label; denotes the size of the batch sample number of each iteration; denotes the number of anchor points with positive labels in each iteration; The weight parameter is used to balance the sample number; denotes the network predicted probability of embolus existence; denotes the true probability of the corresponding bounding box; denotes the network predicted bounding box offset; denotes the offset of the corresponding positive sample bounding box; Each has three values, defined as follows: (10); wherein: and denote the center point coordinates of the predicted bounding box; and denote the center point coordinates of the set bounding box; and denote the center point coordinates of the true labeled pulmonary embolus bounding box; denote the edge length of the predicted, set and labeled bounding box respectively; are the network predicted bounding box relative to the set bounding box coordinate and edge length offset respectively; are the true labeled bounding box relative to the set bounding box coordinate and edge length offset respectively; and are defined as follows: (11); (12); S325: In the inference stage, the generated pulmonary embolus existence probability is sorted from large to small, the top 20% corresponding bounding boxes are selected, and then the generated coordinate offset The selected bounding boxes are position corrected, the bounding boxes exceeding the image boundary are coordinate cropped, and the bounding boxes with unreasonable size are discarded; S326: Finally, non-maximum suppression is performed on all bounding boxes, and the remaining bounding box coordinates are output, completing the detection of all potential positions of pulmonary embolus.

[0014] As a further limitation of the technical solution, the steps of the pulmonary embolism type recognition and positioning network are: S411: First, the different scale feature maps and the generated pulmonary embolism detection box are precisely aligned and sampled using a region of interest alignment module, and all the different size regions are down-sampled to the same scale; S412: When all the detection boxes of the different scale feature maps are down-sampled to the same size feature map, all the feature maps are added element by element for fusion, and then the feature map is converted into a one-dimensional vector and sent into two layers of fully connected layers, and then divided into two branches connected to the fully connected layer, the upper branch is used for classification to predict which category the pulmonary embolism belongs to, and the lower branch is used for regression to predict the position coordinates of the pulmonary embolism, and each category has 3 position parameters; S413: The pulmonary embolism detection box obtained by the lower branch is sent to the next recognition module for the same operation to further iteratively improve the recognition and positioning accuracy of the pulmonary embolism; In the training stage, a large number of detection boxes generated by the pulmonary embolism potential position detection network are processed: S421: First, the intersection over union value between each detection box and the real sample is calculated, the intersection over union value > 0.7 is classified as a positive sample, otherwise it is a negative sample, and 256 samples are selected according to the ratio of 3:1 and sent to the region of interest alignment module and the subsequent fully connected layer for pulmonary embolism classification and regression; S422: The second recognition module increases the intersection over union value to 0.9, and performs the same classification and regression processing to generate the fine-tuned pulmonary embolism classification and position positioning; S423: The loss function of the network also adopts the binary cross-entropy loss and smooth Loss.

[0015] Compared with the related art, the pulmonary embolism detection system provided by the application has the following beneficial effects: The CTPA sequence image cleaning and enhancement network is used for processing the CTPA sequence image of the patient; a large number of invalid regions (such as background tissues and non-related anatomical structures) are contained in the original CTPA image, which interferes with the learning of the key pulmonary embolism features by the model. By clipping the invalid regions to the maximum extent, the core region where the pulmonary embolism is located is focused, and the learning efficiency of the model can be improved. For small targets such as pulmonary embolism, the enhanced image makes the relative size of the target larger and the features more prominent, which helps to improve the detection accuracy.

[0016] The CTPA multi-scale feature extraction network is composed of a residual conversion network ResNeXt50 and a bidirectional feature pyramid network BiFPN, and is used for pixel-level filtering analysis and feature extraction on the original CTPA image, and outputs a CTPA multi-scale feature map; the pulmonary embolus potential position detection network is composed of a plurality of stand-alone attention mechanisms Stand-AloneAttention, and is used for detecting the position area of the pulmonary embolus on the CTPA multi-scale feature map, and outputs the existence probability and possible position of the pulmonary embolus; the pulmonary embolism type recognition and positioning network iteratively optimizes all possible positions of the pulmonary embolus through a plurality of connected recognition modules, gradually improves the detection accuracy of the network for small target pulmonary embolus, and finally completes the type identification and accurate positioning of the pulmonary embolism. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 It is a schematic diagram of the system structure of the application.

[0018] Figure 2 It is a schematic diagram of the grouping convolution technology structure of the application.

[0019] Figure 3 It is a BiFPN bidirectional feature fusion network of the application.

[0020] Figure 4 It is a flowchart of the stand-alone self-attention mechanism of the application.

[0021] Figure 5 It is a flowchart of the pulmonary embolus potential position detection network of the application.

[0022] Figure 6 It is a flowchart of the pulmonary embolism type recognition and positioning network of the application. DETAILED DESCRIPTION

[0023] The application will be further described below in combination with the drawings and embodiments.

[0024] Reference Figures 1-6 A pulmonary embolism detection system combining CT pulmonary angiography and neural network, comprising: a CT pulmonary angiography (CTPA) sequence image cleaning and enhancement network, a CTPA multi-scale feature extraction network, a pulmonary embolus potential position detection network, and a pulmonary embolism type recognition and positioning network; The CT pulmonary angiography sequence image cleaning and enhancement network is used for processing the CTPA sequence image of the patient, and specifically includes five parts: CT pulmonary angiography data collection, data set labeling and division, CTPA image enhancement, data set expansion, and data normalization. The CT pulmonary angiography multi-scale feature extraction network is composed of a residual network (Residual Network Extended 50, ResNeXt50) and a bidirectional feature pyramid network (Bidirectional Feature Pyramid Network, BiFPN), and is used for pixel-level filtering analysis and feature extraction on the original CTPA image, and outputs a CTPA multi-scale feature map; The pulmonary embolus potential position detection network is composed of a plurality of stand-alone attention mechanisms (Stand-Alone Attention), and is used for detecting the position area of the pulmonary embolus that may exist on the CTPA multi-scale feature map, and outputs the existence probability and possible position of the pulmonary embolus; The pulmonary embolism type recognition and positioning network includes a plurality of serial recognition modules, iteratively optimizes all possible positions of the pulmonary embolus, gradually improves the detection accuracy of the network for small target pulmonary embolus, and finally completes the discrimination and accurate positioning of the pulmonary embolism type.

[0025] CTPA data collection: The present application collects pulmonary CTPA sequence images of 42 patients from Jinan Hospital of Integrated Traditional Chinese and Western Medicine and the Cad-pe challenge public data set. There are about 200-300 CTPA images for each patient, in order to ensure the relevance and effectiveness of the data, the professional doctors are screened first. According to the clinical experience and medical image knowledge, the doctors exclude the images without pulmonary blood vessels, and retain the CTPA images containing only pulmonary blood vessels.

[0026] Data set labeling and division: This part labels the collected CTPA images with pulmonary embolism, and divides them into different data sets. The data labeling is performed by professional radiologists according to the clinical diagnosis standard, and the CTPA images after screening are labeled one by one to determine whether they contain pulmonary embolism, and the corresponding four types of labels are generated: occlusive pulmonary embolism, mural pulmonary embolism, eccentric pulmonary embolism, and central pulmonary embolism. The labeled data set is randomly divided into a training set (30 people), a validation set (8 people) and a test set (4 people) according to the ratio of 7:2:1, to ensure the balanced distribution of each subset.

[0027] The steps of CTPA image enhancement are: The original CTPA image contains a large number of invalid regions (such as background tissue, non-related anatomical structure), which interfere with the model's learning of key pulmonary embolism features. By cropping to remove invalid regions to the maximum extent, focusing on the core area of the pulmonary embolism, the learning efficiency of the model can be improved.

[0028] S111: Load the CT pulmonary angiography sequence image, and determine the center point coordinates of the pulmonary blood vessel region; S112: Based on the center point as the origin, the original 512x512 size image is cropped to 319x319 size based on the Open Computer Vision Library (OpenCV); S113: The cropped image is enlarged to the original size by bilinear interpolation, and image enhancement is completed.

[0029] For small targets such as pulmonary embolism, the enhanced image makes the relative size larger and the features more prominent, which helps to improve the detection accuracy.

[0030] The steps of data set augmentation are: The amount of medical image data is small, which is not conducive to the training of deep neural networks. Data augmentation is performed on each CTPA image in the training set and validation set.

[0031] S121: The CTPA image is horizontally mirrored, vertically mirrored, and rotated 90 degrees left and right to generate images with multiple angles and shapes; S122: The augmented image and the original image form a new training set and validation set.

[0032] This allows the model to learn the feature changes of pulmonary embolism at different angles and directions, enhancing the model's robustness to image rotation, flipping and other transformations.

[0033] When faced with image differences caused by patient position, device angle and other factors in actual scenarios, the model can detect more stably, effectively reducing false positives and false negatives, and improving overall performance.

[0034] The steps of data normalization are: The pixel value range of CTPA image is wide, and the pixel value difference of different images is large, which may cause unstable numerical calculation in the model training process, affecting learning efficiency and accuracy.

[0035] S131: Obtain the pixel matrix of the CTPA image; S132: Calculate the maximum and minimum values of the pixels; S133: Normalize the pixel values to the [0, 1] interval to optimize the input data distribution of the model.

[0036] Normalized data makes the pixel value range uniform, and the numerical calculation of the model is more stable during training, improving the model's adaptability to different CTPA images.

[0037] The residual transformation network introduces group convolution technology based on the residual network (ResNet), which improves the model's detection ability for small pulmonary emboli without significantly increasing the model's calculation and parameter amount. The residual transformation network is stacked by multiple residual blocks, each of which is composed of three convolutional layers, wherein the second convolutional layer adopts a grouped convolution technique to divide the input channels into multiple groups for convolution operation, the number of groups of the grouped convolution is set to 32, and the number of channels is set to 4, and then the outputs of the groups are spliced to increase the width and expression capacity of the model, and more rich feature representations such as the morphology of different blood vessel branches and the density changes of emboli in the CTPA image can be learned; The input of ResNeXt50 is a CTPA image with a dimension of 512x512x5. In pulmonary embolism detection, the spatial continuity of the lesion (whether it persists in adjacent images) is a key feature, and multi-image input can simultaneously learn single-frame spatial features and cross-frame temporal correlation. This method is efficient and practical in medical image sequence analysis. In clinical practice, the CTPA slice scanning interval is generally set to 1 mm, and the pulmonary embolus morphology is not less than 5 mm, therefore, 5 CTPA images are spliced in the channel dimension and input into the network for learning together to predict the intermediate image. The specific structure of ResNeXt50 is shown in Table 1, and the grouped convolution technique is shown in Figure 2 .

[0038] Table 1 ResNeXt50 network structure

[0039] The bidirectional feature pyramid network is constructed based on the output feature maps of the residual block group of the residual transformation network, and includes an upward path and a downward path; the upward path (P5-P2) gradually transmits the high-resolution feature maps of the high layer to the low layer through convolution and up-sampling operations, to extract deep semantic information. The downward path (P2-P5) gradually transmits the high-resolution feature maps of the low layer to the high layer through convolution and down-sampling operations, to extract shallow detail information.

[0040] S21: The output feature maps of the residual block groups 2-5 of the residual transformation network are processed respectively, the channel number is adjusted through 1x1 convolution to make the channel number uniform to 256, and then up-sampling operation is performed to gradually restore the size of the feature maps; S22: The output feature maps of each stage of the residual transformation network are adjusted in channel number through 1x1 convolution, and then down-sampling operation is performed to gradually reduce the size of the feature maps; S23: For each level of the bidirectional feature pyramid network, the feature maps of the bidirectional paths are adjusted, and a learnable weight parameter is assigned to the feature maps of each path, the weight parameter is normalized through a Softmax function, so that its value is distributed in [0, 1], and the sum of the weights of all paths is 1; S24: The feature maps of each path are multiplied by their corresponding weights, and then element-wise addition is performed to obtain the final fused feature map, and the specific formula is as follows: (1) ; wherein: represents the output feature map of the residual transformation network residual block group 2, as the input of the bidirectional feature pyramid network; represents the intermediate process output of the second layer from top to bottom of the bidirectional feature pyramid network; is a network learnable weight parameter, represents a very small constant to prevent the denominator from being 0; represents a down-sampling operation, represents a 1x1 convolution operation, is the output of the bidirectional feature pyramid network corresponding to the residual transformation network residual block group 2; (2) ; wherein: represents the output feature map of the residual transformation network residual block group 3, as the input of the bidirectional feature pyramid network; is a network learnable weight parameter; is the output of the bidirectional feature pyramid network corresponding to the residual transformation network residual block group 3; (3) ; wherein: represents the output feature map of the residual transformation network residual block group 4, as the input of the bidirectional feature pyramid network, represents the intermediate process output of the first layer from top to bottom of the bidirectional feature pyramid network, is a network learnable weight parameter, is the output of the bidirectional feature pyramid network corresponding to the residual transformation network residual block group 4; (4) ; wherein: represents the output feature map of the residual transformation network residual block group 5, as the input of the bidirectional feature pyramid network, is a network learnable weight parameter, is the output of the bidirectional feature pyramid network corresponding to the residual transformation network residual block group 5.

[0041] After the bidirectional feature fusion network of BiFPN, four different scale feature maps are output, the sizes are 128x128, 64x64, 32x32 and 16x16 respectively, and the channel numbers are all 256. These multi-scale feature maps can fully capture the features of different sizes and shapes of pulmonary embolism in CTPA images, and provide rich multi-scale information for subsequent detection tasks.

[0042] The shape of pulmonary embolism in CTPA images is diverse and the characteristics are not obvious. Traditional target detection methods have many limitations, and often have high false positives and small target detection problems. Self-attention mechanism has been proven to enhance the network's long-distance spatiotemporal modeling ability and effectively improve the target detection accuracy. The pulmonary embolus potential location detection network (PLN) is composed of multiple stand-alone attention (SSA) mechanism layers.

[0043] The calculation steps of the stand-alone self-attention mechanism are as follows: S311: Figure 4 The leftmost represents the input feature map, and a local neighborhood centered on the pixel point is extracted in the input feature map, wherein the input feature dimension is obtained by the attention kernel to obtain the query vector and the key matrix and value matrix of each pixel point on the local neighborhood; S312: After the weight matrix (query vector and key matrix matrix multiplication and normalized by softmax) and value matrix are pixel-by-pixel dot product and summed, the output center pixel of the center pixel is obtained, that is, one pixel of the output feature map; S313: Because of the lack of position information of each pixel point in the stand-alone self-attention mechanism, the full attention layer is not sensitive to the arrangement order of the input feature map, which limits the network's expression ability in the regression task. In order to solve the above problems, the relative distance information of each pixel point is added to the softmax matrix of the stand-alone self-attention mechanism, which is defined as the relative distance from the center pixel to other pixel points in the neighborhood; For a two-dimensional feature map, there are two dimensions of relative distance between two pixel points, row offset and column offset , and the formula for calculating the stand-alone self-attention mechanism is as follows: (5); (6); (7); (8); Wherein: represents the size of the square local neighborhood; are the query attention kernel, the key attention kernel, and the value attention kernel, respectively, which are used to map the pixels in the local neighborhood to the query vector , the key vector and the value vector , Indicates the output feature dimension; Indicates the relative distance of the row offset. Indicates the relative distance of column offset; It is a learnable line offset position embedding. It is a learnable column offset position embedding; By independently adding relative distance information in the row and column directions, the problem of insensitivity to pixel arrangement order is solved; By using value vectors After performing matrix multiplication and summing, the output of the local neighborhood is obtained. The same operation is performed sequentially on other local neighborhoods of the input feature map to obtain the final output feature map.

[0044] The pulmonary embolism latent location detection network uses feature maps of four scales output from the previous stage network as input. Each scale of feature map undergoes the same detection processing. The following description uses a 128×128×256 feature map as an example; the steps are as follows: S321: Each pixel in a CT pulmonary angiography image Four different sized square detection frames are set on each frame, with side lengths... This is used to initially predict the potential location of pulmonary emboli; S322: The feature map completes long-distance spatiotemporal structural modeling of CT pulmonary angiography through an independent self-attention mechanism layer with a local neighborhood size of K=3×3 and 512 channels. S323: Two parallel, independent self-attention mechanism layers (K=1×1) are used. The attention layer in the upper branch is used for target classification, i.e., whether there is a pulmonary embolism in the detection box. Since each detection point has 4 detection boxes, each detection box has two probability values. Therefore, the output for each detection point is an 8-dimensional vector, meaning the upper branch has 8 channels; the attention layer in the lower branch is used to obtain the offset of the detection box position relative to the ground truth position, and each detection box has 3 offset values. Therefore, the output corresponding to each monitoring point is a 12-dimensional vector, that is, the lower branch has 12 channels; S324: During the training phase, 512 bounding boxes are selected from all detection boxes for classification and location regression. The selection process is as follows: For each ground truth label, select the bounding box with the largest Intersection over Union (IoU) as the positive sample; Then, from the remaining detection boxes, samples with an intersection-union ratio (IU) > 0.7 with any real label are selected as positive samples, for a total of 256 positive samples. Finally, randomly select and any real label IoU value <0.3 as negative samples, the rest of the detection box (0.3<IoU<0.7) does not participate in training, the total number of negative samples and positive samples is 512; The objective function used by the network is as follows: (9); Wherein: is the entire objective function; is the binary cross-entropy loss; is the smooth loss; denotes the detection box label; denotes the size of the batch sample size of each iteration; denotes the number of anchor points with positive labels in each iteration; The weight parameter is used to balance the sample number; denotes the network predicted embolus existence probability; denotes the corresponding detection box true probability; denotes the network predicted detection box offset; denotes the corresponding positive sample detection box offset; Each has three values, defined as follows: (10); Wherein: and denote the center point coordinates of the predicted detection box; and denote the center point coordinates of the set detection box; and denote the center point coordinates of the real labeled pulmonary embolus detection box; denote the edge length of the predicted, set, and labeled detection box, respectively; are the network predicted detection box relative to the set detection box coordinate and edge length offset, respectively; are the real labeled detection box relative to the set detection box coordinate and edge length offset, respectively; Used for classifying detection boxes as positive or negative samples, it is a binary cross-entropy loss. Used for regression detection box relative to the real label offset, it is a smooth loss.

[0045] and are defined as follows: (11); (12); S325: In the reasoning stage, the generated probability of pulmonary embolus is sorted from large to small, the top 20% of the corresponding detection box is selected, and then the generated coordinate offset is used The selected detection box is position corrected, the detection box exceeding the image boundary is coordinate cropped, and the detection box with unreasonable size (edge length greater than 60 or less than 5) is discarded; S326: Finally, non-maximum suppression is performed on all detection boxes, and the remaining detection box coordinates are output, completing the detection of all potential positions of pulmonary embolus.

[0046] According to the existence form of pulmonary embolus, the pulmonary embolus can be divided into four types: occlusion type pulmonary embolism, mural type pulmonary embolism, eccentric type pulmonary embolism and central type pulmonary embolism. The pulmonary embolism type recognition and positioning network uses the obtained multi-scale feature map and detection box to gradually improve the IoU comparison threshold, and through two serial identification modules, the pulmonary embolus appearance probability and all possible positions are iteratively optimized, the detection accuracy of small target pulmonary embolus is gradually improved, and finally the type of pulmonary embolism is determined and accurately positioned.

[0047] The steps of the pulmonary embolism type recognition and positioning network are: S411: First, the different scale feature maps and the generated pulmonary embolus detection boxes are aligned and sampled by the region of interest (RoI) align module for accurate feature alignment and sampling. The regions of different sizes are all down-sampled to the same scale (7x7). RoI align retains more accurate spatial position information by avoiding coordinate quantization and introducing interpolation calculation, solves the feature position deviation problem caused by quantization operation in traditional RoI Pooling, and effectively improves the positioning accuracy of small targets by the model.

[0048] S412: After all the detection boxes of different scale feature maps are down-sampled to the same size feature map, all the feature maps are added element by element for fusion, and then the feature map is converted into a one-dimensional vector and sent to two fully connected layers (FC). Then it is divided into two branches and connected to the fully connected layer. The upper branch is used for classification to predict which category the pulmonary embolus belongs to, and the lower branch is used for regression to predict the position coordinates of the pulmonary embolus. Each category has 3 position parameters; S413: The pulmonary embolus detection box obtained by the lower branch is sent to the next identification module for the same operation to further iteratively improve the recognition and positioning accuracy of the pulmonary embolus; In the training stage, a large number of detection boxes generated by the pulmonary embolus potential position detection network are processed: S421: First, calculate the intersection over union value between each detection frame and the real sample, if the intersection over union value > 0.7, it is classified as a positive sample, otherwise it is a negative sample, and 256 samples are selected according to the ratio of 3:1 to send into the region of interest alignment module and the subsequent full connection layer for lung embolus classification and regression; S422: The second identification module increases the intersection over union value to 0.9, and performs the same classification and regression processing to generate the fine-tuned lung embolus classification and position positioning; S423: The loss function of the network also adopts binary cross-entropy loss and smooth loss.

[0049] In the inference stage, since the real sample label cannot be known, the network will perform lung embolus category classification and position positioning on the detection frame generated by the lung embolus potential position detection network.

[0050] The above only describes the embodiments of the present application, and does not limit the patent scope of the present application, any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A pulmonary embolism detection system combining CT pulmonary angiography and neural networks, characterized in that, The system comprises a CT pulmonary angiography sequence image cleaning and enhancement network, a CT pulmonary angiography multi-scale feature extraction network, a pulmonary embolus potential position detection network, and a pulmonary embolism type identification and positioning network. The CT pulmonary angiography sequence image cleaning and enhancement network specifically comprises five parts: CT pulmonary angiography data collection, data set labeling and division, CT pulmonary angiography image enhancement, data set expansion, and data normalization. The CT pulmonary angiography multi-scale feature extraction network is composed of a residual transformation network and a bidirectional feature pyramid network. The pulmonary embolus potential position detection network is composed of multiple independent self-attention mechanisms. The pulmonary embolism type identification and positioning network comprises multiple identification modules connected in series. The steps of the CT pulmonary angiography image enhancement are as follows:

2. The combined CT pulmonary angiography and neural network for pulmonary embolism detection system of claim 1, wherein: S111: Load the CT pulmonary angiography sequence image, and determine the center point coordinates of the pulmonary vascular region; S112: Take the center point as the origin, and perform cropping based on the open source computer vision library to crop the original 512x512 size image to 319x319 size; S113: The cropped image is enlarged to the original size by bilinear interpolation, and the image enhancement is completed. The steps of the data set expansion are as follows:

3. The combined CT pulmonary angiography and neural network for pulmonary embolism detection system of claim 2, wherein: S121: Perform horizontal mirroring, vertical mirroring, and left-right rotation of 90 degrees on the CT pulmonary angiography image to generate images of various angles and shapes; S122: The expanded images and the original images together form a new training set and a validation set. The steps of the data normalization are as follows:

4. The combined CT pulmonary angiography and neural network for pulmonary embolism detection system of claim 3, wherein: S131: Obtain the pixel matrix of the CT pulmonary angiography image; S132: Calculate the maximum and minimum values of the pixels; S133: Normalize the pixel values to map them to the [0, 1] interval, and optimize the input data distribution of the model. The residual transformation network introduces a grouping convolution technique based on the residual network, which improves the detection capability of the model for small pulmonary emboli without significantly increasing the model calculation and parameter amount.

5. The combined CT pulmonary angiography and neural network for pulmonary embolism detection system of claim 1, wherein: The residual transformation network is stacked by multiple residual blocks, and each residual block is composed of three convolution layers, wherein the second convolution layer adopts the grouping convolution technique to divide the input channels into multiple groups for convolution operation, and then splices the outputs of each group.

6. The pulmonary embolism detection system combining CT pulmonary angiography and neural network according to claim 5, wherein the bidirectional feature pyramid network constructs a bidirectional feature fusion network based on the output feature maps of the residual block groups of the residual transformation network, and comprises a top-down path and a bottom-up path. S21: Process the output feature maps of the residual block groups 2-5 of the residual transformation network respectively, adjust the channel number by 1x1 convolution to make the channel number uniform to 256, and then perform upsampling operation to gradually restore the feature map size; S22: Adjust the channel number of the output feature maps of each stage of the residual transformation network by 1x1 convolution, and then perform downsampling operation to gradually reduce the feature map size; ​ S23: For each level of the bidirectional feature pyramid network, the feature maps of the bidirectional paths are adjusted, and a learnable weight parameter is assigned to each path of the feature maps. The weight parameter is normalized by the Softmax function, so that its value is distributed between [0, 1], and the sum of the weights of all paths is 1; S24: Multiply the feature maps of each path by their corresponding weights, then add them element by element to obtain the final fused feature map, the specific formula is as follows: （1）； wherein: represents the output feature map of the residual block group 2 of the residual transformation network as the input of the bidirectional feature pyramid network; represents the intermediate process output of the second layer from top to bottom of the bidirectional feature pyramid network; is the weight parameter that the network can learn, represents a very small constant to prevent the denominator from being 0; represents a down-sampling operation, represents a 1x1 convolution operation, is the output of the residual block group 2 of the bidirectional feature pyramid network corresponding to the residual transformation network; （2）； wherein, represents the output feature map of the residual block group 3 of the residual transformation network, as the input of the bidirectional feature pyramid network; is a network learnable weight parameter; is the output of the bidirectional feature pyramid network corresponding to the residual block group 3 of the residual transformation network; （3）； wherein: represents the output feature map of the residual transformation network residual block group 4 as the input of the bidirectional feature pyramid network, represents the intermediate process output of the first layer from top to bottom of the bidirectional feature pyramid network, is the weight parameter learnable by the network, is the output of the bidirectional feature pyramid network corresponding to the residual transformation network residual block group 4; （4）； wherein: represents the output feature map of the residual block group 5 of the residual transformation network as an input of the bi-directional feature pyramid network, is a network learnable weight parameter, is the output of the bi-directional feature pyramid network corresponding to the residual block group 5 of the residual transformation network.

7. The combined CT pulmonary arteriography and neural network for pulmonary embolism detection system of claim 2, wherein: The steps of the independent self-attention mechanism are as follows: S311: extract a local neighborhood centered at a pixel point in the input feature map as the center, represents the input feature dimension, and is obtained by the attention kernel query vector and key matrix and value matrix of each pixel point on the local neighborhood S312: the output center pixel point of the center pixel point is obtained by performing pixel-by-pixel dot product and summation on the weight matrix and the value matrix , that is, one pixel of the output feature map​ S313: The relative distance information of each pixel point is added in the softmax matrix of the independent self-attention mechanism, which is defined as the relative distance of the center pixel point to other pixel points in the neighborhood . ​ For a two-dimensional feature map, there is a row offset between two pixel points and a column offset The relative distance of the two dimensions, and the formula for calculating the independent self-attention mechanism is as follows: （5）； （6）； （7）； （8）； wherein: represents the size of the local neighborhood of a square; are respectively a query attention kernel, a key attention kernel and a value attention kernel for mapping pixels in the local neighborhood into query vectors , key vectors and value vectors , denotes the output feature dimension; denotes the row offset relative distance, denotes the column offset relative distance; is a learnable row offset position embedding, is a learnable column offset position embedding; By multiplying with the value vector Summing up after matrix multiplication to get the output of the local neighborhood The same operation is sequentially performed on other local neighborhoods of the input feature map to obtain the final output feature map.

8. The combined CT pulmonary arteriography and neural network for pulmonary embolism detection system of claim 7, wherein: The pulmonary embolus potential position detection network uses the 4 scales of feature maps output by the upper level network as input, and each scale of feature map is processed in the same way. Taking the 128x128x256 feature map as an example, the steps are as follows: S321: In each pixel point of the CT pulmonary arteriography image Each of the above is provided with four square detection boxes of different sizes, with side length , to preliminarily predict the potential location of the pulmonary embolus; S322: The feature map passes through the independent self-attention mechanism layer with a local neighborhood size K=3x3 and a channel number of 512, to complete the CT pulmonary angiography long-distance spatio-temporal structure modeling; S323: through two parallel independent self-attention mechanism layers, the attention layer of the upper branch is used for target classification, i.e. whether there is a pulmonary embolism in the detection box, since there are 4 detection boxes for each detection point and each detection box has two probability values Therefore, the output corresponding to each detection point is an 8-dimensional vector, i.e. the upper branch has 8 channels; the attention layer of the lower branch is used to obtain the offset of the detection box position relative to the true position, and each detection box has 3 offset values Therefore, the output corresponding to each detection point is a 12-dimensional vector, i.e. the lower branch has 12 channels; S324: In the training stage, 512 detection boxes are selected from all detection boxes for classification and position regression, and the selection process is as follows: For each real label, select the detection box with the maximum intersection over union (IOU) value as the positive sample; Then select the detection box with an IOU value greater than 0.7 with any real label as a positive sample from the remaining detection boxes, and the total number of positive samples is 256; Finally, randomly select a detection box with an IOU value less than 0.3 with any real label as a negative sample, and the remaining detection boxes do not participate in training. The total number of positive samples and negative samples is 512; The objective function used is as follows: （9）； wherein: is the whole objective function; is the binary cross-entropy loss; is the smooth loss; represents the detection frame label; represents the size of the batch sample number in each iteration; represents the number of anchors with positive labels in each iteration; The weight parameter is used to balance the sample number; represents the network predicted probability of embolus existence; represents the corresponding detection frame true probability; represents the network predicted detection frame offset; represents the corresponding positive sample detection frame offset; Each has three values, defined as follows: （10）； wherein: and denote the center point coordinates of the predicted bounding box; and denote the center point coordinates of the set bounding box; and denote the center point coordinates of the real labeled pulmonary embolus bounding box; denote the side length of the predicted, set and labeled bounding box, respectively; are the offsets of the coordinates and side length of the network predicted bounding box relative to the set bounding box, respectively; are the offsets of the coordinates and side length of the real labeled bounding box relative to the set bounding box, respectively; and are defined as: （11）； （12）； S325: In the reasoning stage, the generated probability of pulmonary embolus is sorted from large to small, the top 20% corresponding detection boxes are selected, and then the generated coordinate offset is used The selected detection boxes are position corrected, the detection boxes exceeding the image boundary are coordinate cropped, and the detection boxes with unreasonable size are discarded; S326: Finally, non-maximum suppression is performed on all detection boxes to output the remaining detection box coordinates, and all potential positions of the pulmonary embolus are detected.

9. The pulmonary embolism detection system combined with CT pulmonary angiography and neural network according to claim 8, characterized in that: The steps of the pulmonary embolism type identification and positioning network are as follows: S411: First, the different scale feature maps and the generated pulmonary embolus detection boxes are precisely aligned and sampled using the region of interest alignment module, and all regions of different sizes are down-sampled to the same scale; S412: After all the detection boxes of different scale feature maps are down-sampled to the same size, all the feature maps are added element by element for fusion, and then the feature map is converted into a one-dimensional vector and sent to two fully connected layers. The two branches are connected to the fully connected layer, the upper branch is used for classification to predict which category the pulmonary embolus belongs to, and the lower branch is used for regression to predict the position coordinates of the pulmonary embolus. Each category has 3 position parameters; S413: The pulmonary embolus detection box obtained by the lower branch is sent to the next identification module for the same operation to further improve the recognition and positioning accuracy of the pulmonary embolus; In the training stage, a large number of detection boxes generated by the pulmonary embolus potential position detection network are processed as follows: S421: First, calculate the intersection over union (IOU) value between each detection box and the real sample. If the IOU value is greater than 0.7, it is a positive sample, otherwise it is a negative sample. Select 256 samples according to a ratio of 3:1 and send them to the region of interest alignment module and the subsequent fully connected layer for pulmonary embolus classification and regression. S422: The second identification module increases the intersection over union value to 0.9, and performs the same classification and regression processing to generate a fine-tuned pulmonary embolus classification and position positioning; S423: The loss function of the network also adopts binary cross-entropy loss and smoothing loss.

Images