Coronary artery stenosis detection method based on coronary angiography and space-time collaborative learning

Abstract

The invention discloses a coronary artery stenosis detection method based on coronary angiography and space-time collaborative learning, and belongs to the field of intelligent medical treatment. According to the method, through a deep feature extraction network based on a channel shuffling structure re-parameterization convolution single aggregation module, through channel segmentation and multi-branch training, feature multiplexing and information flowing are enhanced, a double-flow path is introduced into a feature fusion network to realize adaptive weighted fusion of features of different scales, and the feature fusion efficiency is improved. The representation of spatial features is improved; a space-time collaborative analysis module capable of fusing time information and local space information at the same time is adopted, and blood vessel shape deformation and hemodynamic information in a space-time sequence are effectively captured and effective feature information is enhanced by constructing time sequence connection and fusing a space-time feature graph, so that the detection accuracy and robustness of coronary artery stenosis lesions are improved. According to the method, the time and space information of the coronary angiography sequence image is comprehensively considered, the number of false positive detection is reduced, and the detection capability of the model coronary artery stenosis lesion is comprehensively improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent medical treatment assisted by artificial intelligence, and specifically relates to a method for detecting coronary artery stenosis based on serial coronary angiography images and spatiotemporal collaborative analysis. Background Art

[0002] At present, artificial intelligence driven by big data has become one of the research frontiers in the field of medical auxiliary diagnosis due to its high accuracy and high stability. Coronary artery disease (CAD) is a disease of insufficient blood supply to the heart caused by coronary atherosclerosis. It is one of the most common cardiovascular diseases in the world and the main cause of myocardial infarction, heart failure and cardiac death. According to statistics from the World Health Organization (WHO), CAD caused about 8.8 million deaths worldwide in 2019 alone, accounting for 15.8% of all deaths from diseases. The harmfulness of coronary artery stenosis is mainly reflected in the number, location and degree of stenosis. In current hospital practice, the complexity of the detection background (vasodilation and contraction, occlusion, imaging angle, etc.) and the subjectivity caused by the reliance on the experience and level of practicing doctors restrict the accuracy and objectivity of coronary artery stenosis detection (number and location) and the determination of stenosis degree.

[0003] Therefore, it is particularly important to conduct research on computer-aided coronary artery stenosis diagnosis technology. At present, the main coronary artery stenosis detection methods based on CAG include coronary artery stenosis detection based on vascular diameter and coronary artery stenosis detection based on vascular features. Coronary artery stenosis detection based on vascular diameter includes vascular segmentation, arterial centerline extraction, diameter calculation, stenosis analysis and other steps. Since these steps themselves have errors, the errors of multiple steps will reduce the robustness and robustness of stenosis assessment. Therefore, stenosis detection methods based on vascular features are more effective than those based on vascular diameter. At present, some methods have adopted the method of directly processing X-ray coronary angiography images to eliminate the influence of multiple steps such as vascular segmentation on the accuracy of coronary artery stenosis detection. However, most studies currently only use the spatial structure information in a single frame image for stenosis detection without considering the inter-frame contextual relationship of the image sequence. In some frames, there are problems such as insufficient robustness of single-frame stenosis detection caused by transient contrast agent inhomogeneity, which to a certain extent reduces the overall detection accuracy. Summary of the invention

[0004] The present invention provides a coronary artery stenosis detection method based on coronary angiography and spatiotemporal collaborative learning to improve the detection performance of coronary artery stenosis.

[0005] The technical solution adopted by the present invention is: a method for detecting coronary artery stenosis based on coronary angiography and spatiotemporal collaborative learning, the method comprising the following steps:

[0006] Step 1, obtaining original coronary angiography data and doctor's diagnosis information;

[0007] Step 2: preprocess the original coronary angiography data and the doctor's diagnosis information to obtain a sequence of coronary angiography images and a corresponding stenosis location information label;

[0008] Step 3, constructing a coronary angiography image spatial feature constraint network, which includes a deep feature extraction network and a dual-stream path multi-scale feature fusion network;

[0009] Among them, the deep feature extraction network is a network structure based on a channel shuffle structure re-parameterized convolution single aggregation module, which is used to extract spatial features of three different scales of each frame of the coronary angiography image of the sequence;

[0010] The multi-scale feature fusion network of the dual-stream path strengthens and fuses the multi-scale spatial features extracted by the deep feature extraction network through the bottom-up and top-down dual-stream paths to obtain enhanced spatial features of three different scales for each frame of coronary angiography image; the three scales of the enhanced spatial features are consistent with the scales of the spatial features extracted by the deep feature extraction network;

[0011] Step 4, construct a spatiotemporal information collaborative analysis module, input the three-scale enhanced spatial features of each frame of the extracted sequence coronary angiography images into the spatiotemporal information collaborative analysis module, use the single-frame spatial features as prior knowledge, build a temporal connection between the sequence frames to obtain the spatiotemporal features containing past frame information, and thus obtain the three-scale spatiotemporal features of the current frame containing past frame information to further enhance the effective feature information; then fuse the spatiotemporal features with the enhanced spatial features of the corresponding scales of the current frame to obtain the final three-scale spatiotemporal features of the current frame, that is, use the single-frame spatial features as prior knowledge, and simultaneously capture the temporal dependency between sequence frames and the spatial dependency within each frame to obtain the spatiotemporal features of three different scales of each frame, thereby realizing the enhancement of the effective stenosis lesion feature information of the current frame by the information of the past frames in the comprehensive sequence, weakening the redundant information in the spatiotemporal feature sequence, and realizing the spatiotemporal collaborative analysis of the coronary angiography sequence images;

[0012] Then, the multi-scale spatiotemporal features of each frame are input into the detection head to perform target detection of coronary artery stenosis, and the coronary artery stenosis detection result of the current frame is obtained.

[0013] Furthermore, the loss functions used for model parameter training of the deep feature extraction network, the dual-stream path multi-scale feature fusion network, and the spatiotemporal information collaborative analysis module include:

[0014] Distribution focus loss function and SIoU loss function. The distribution focus loss function optimizes the probabilities of the two positions closest to the label on the left and right in the form of cross entropy, so that the network can focus on the target position and the distribution of the adjacent area more quickly; SIoU enables the loss function to simultaneously consider the position, size and shape information of the bounding box during the optimization process, thereby improving the detection effect of coronary stenosis lesions.

[0015] Furthermore, a deep feature extraction network, a stacked network including a Focus layer, a convolutional block, a plurality of stacked units consisting of a convolutional block and a structured reparameterized convolutional single shot aggregation (RCS-OSA) module, and an SPPF module in sequence;

[0016] Among them, the convolution block includes a convolution layer, a batch normalization layer, and an activation function layer in sequence;

[0017] The SPPF module is set as follows: the input of the SPPF module first passes through a convolution block, and then its output is pooled through three parallel maximum pooling layers respectively, and then the three-way pooling results are spliced ​​through a splicing layer and sent to a convolution block, and the output of the SPPF module is obtained based on the output of the convolution block.

[0018] The number of further stacked units consisting of convolutional blocks and structured reparameterized convolutional single aggregation modules is set to 4.

[0019] Furthermore, the structural reparameterized convolution single aggregation module includes two RepVGG modules, two RCS modules and one channel reshaping module; wherein, the input of the RCS-OSA module is first sent to the RepVGG module 1 for feature extraction, the output features of the RepVGG module 1 are respectively sent to the channel reshaping module and the RCS module 1, the output features of the RCS module 1 are respectively sent to the channel reshaping module and the RCS module 2, and the output features of the RCS module 2 are also sent to the channel shuffling module; the channel shuffling module performs channel shuffling operations on the three inputs respectively, and then splices the channel dimensions through the splicing layer, and then sends them to the RepVGG module 2, and obtains the output of the RCS-OSA module based on the output of the RepVGG module 2;

[0020] The RCS module is set as follows: the input features of the RCS module are divided according to the channel dimension, and divided into two sub-features with different channel numbers and the same feature plane dimensions (width and height), wherein the first sub-feature is respectively passed through three parallel branches, namely, the identity branch, the convolution layer with a convolution kernel of 1×1, and the convolution layer with a convolution kernel of 3×3, and then the outputs of the three branches are added and mapped through the activation function to obtain the intermediate mapping feature of the first sub-feature; the second sub-feature is spliced ​​with the intermediate mapping feature of the first sub-feature through the splicing layer after the channel division (ChannelSplit) Channel Split operation layer, and the spliced ​​features are then passed through the channel shuffle module to obtain the output of the RCS module;

[0021] Furthermore, the multi-scale feature fusion network of the dual-stream path performs adaptive weighted fusion on the three spatial features of different scales extracted by the deep feature extraction network, specifically including:

[0022] The spatial features of three different scales are defined as large-scale spatial features, medium-scale spatial features, and small-scale spatial features;

[0023] The small-scale spatial features are upsampled to the dimension of the medium-scale spatial features, and then weighted concatenated with the medium-scale spatial features to obtain the first concatenated features; the first concatenated features are then upsampled to the dimension of the large-scale spatial features, and then weighted concatenated with the large-scale spatial features to obtain the second concatenated features;

[0024] Downsampling the second splicing feature to a medium-scale spatial feature, and then performing weighted splicing with the first splicing feature to obtain a third splicing feature, and downsampling the third splicing feature to a dimension of a small-scale spatial feature, and then splicing with the small-scale spatial feature to obtain a fourth splicing feature;

[0025] Enhanced spatial features at three scales are obtained based on the second stitching feature, the third stitching feature and the fourth stitching feature.

[0026] Furthermore, the spatiotemporal information collaborative analysis module is specifically configured as follows:

[0027] The enhanced spatial features of three scales of each frame of the sequential coronary angiography images are extracted through the long short-term memory network ST-LSTM unit based on spatiotemporal characteristics;

[0028] If the current frame is the first frame of the sequence, the enhanced spatial features of each scale are input into the corresponding ST-LSTM unit, and the output features of the ST-LSTM unit are added to the input features of the ST-LSTM unit to obtain the final spatiotemporal features of the current scale of the current frame;

[0029] Starting from the second frame of the sequence, the input of the ST-LSTM unit includes the enhanced spatial features of the current scale of the current frame, and also includes the output features of the enhanced spatial features of the same scale of the previous frame through the ST-LSTM unit; then the output features of the ST-LSTM unit corresponding to the current scale of the current frame are added to the enhanced spatial features of the current scale of the current frame to obtain the final spatiotemporal features of the current scale of the current frame.

[0030] In summary, the technical solution provided by the present invention brings at least the following beneficial effects:

[0031] The present invention rationally utilizes the deep feature extraction network to extract effective feature layers of different scales and performs weighted fusion, comprehensively considers the temporal and spatial information of the coronary angiography sequence image, and improves the loss function to further distinguish the coronary stenosis target and background interference, reduce the number of false positive detections, and comprehensively improve the detection ability of the model for coronary stenosis lesions. Specifically: a deep feature extraction network based on a channel shuffle structure reparameterized convolution single aggregation module is designed. Through channel segmentation and multi-branch training, the reuse of features and information flow are enhanced. A dual-stream path is introduced in the feature fusion network to achieve adaptive weighted fusion of feature maps of different scales, thereby improving the network's representation of spatial features. Then, a spatiotemporal collaborative analysis module that can simultaneously fuse temporal information and local spatial information is adopted. By constructing a temporal connection and fusing spatiotemporal feature maps, the vascular shape deformation and hemodynamic information in the spatiotemporal sequence are effectively captured, and the effective feature information is enhanced, thereby improving the detection accuracy and robustness of coronary stenosis lesions. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A flowchart of a method for detecting coronary artery stenosis based on coronary angiography and spatiotemporal collaborative learning in an embodiment;

[0034] Figure 2 This is a schematic diagram of the structure of the deep feature extraction network;

[0035] Figure 3 It is a schematic diagram of the structure of the RCS module;

[0036] Figure 4 It is a schematic diagram of the structure of the weighted dual-stream path multi-scale feature fusion network;

[0037] Figure 5This is a schematic diagram of the overall structure of the spatiotemporal collaborative analysis network;

[0038] Figure 6 This is a schematic diagram of the structure of the spatiotemporal collaborative analysis module;

[0039] Figure 7 Schematic diagram for calculating the IoU bounding box loss function;

[0040] Figure 8 Schematic diagram of SIoU bounding box loss function calculation;

[0041] Fig. 9 This is a comparison chart of the coronary artery stenosis detection results of the method proposed in the embodiment of the present invention. DETAILED DESCRIPTION

[0042] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0043] The embodiment of the present invention provides a method for detecting coronary artery stenosis based on coronary angiography and spatiotemporal collaborative learning. Figure 1 , which comprises the following steps:

[0044] S1: Obtaining original data information. Including coronary angiography data and corresponding diagnostic information (to obtain the detection tag of the stenosis location coordinates). In the embodiment of the present invention, the original coronary angiography data is obtained from the hospital equipment and the corresponding diagnostic information is given by a professional doctor;

[0045] S2: Data preprocessing. The original data format of coronary angiography is usually in DICOM format. It is necessary to convert the coronary angiography data in DICOM format into the serial image format required by the deep learning model, and make the coronary stenosis diagnosis information into XML format tags to facilitate the deep learning training of network model parameters;

[0046] Afterwards, the coronary angiography sequence images were randomly shuffled to form a five-fold cross-validation data set, and each subset was divided according to the ratio of training set: test set = 4: 1. Among them, the training set is used to train the model parameters, and the test set is used to evaluate the detection effect of the trained model.

[0047] S3: Spatial feature extraction. The deep feature extraction network architecture of the present invention is as follows: Figure 2As shown in the figure, it gradually compresses the size of the feature map in the spatial dimension while expanding the number of channels of the feature map to extract high-level semantic features of the image. At the same time, the RCS-OSA (RCS-One-Shot Aggregation) module is introduced to ensure feature reuse and strengthen information flow between different layers by stacking RCS.

[0048] In this embodiment, the deep feature extraction network used includes Focus layer (by rearranging the information of spatial dimension to channel dimension, the purpose of improving computational efficiency without losing information is achieved, that is, by slicing operation, the high-resolution feature map is split into multiple low-resolution feature maps, and by splicing and convolution operation, a low-resolution feature map with increased number of channels is finally obtained), convolution block, a stacked network of stacked units composed of convolution blocks and structural reparameterized convolution single aggregation (RCS-OSA) modules, and SPPF module (Spatial Pyramid PoolingFast); in order to realize the extraction of multi-scale features, this embodiment extracts feature maps of different scales at three different levels of the deep feature extraction network (the second to last RCS-OSA module, the third to last RCS-OSA module and the last layer, that is, the output of the SPPF module) to input the subsequent weighted dual-stream path multi-scale feature fusion network, and the feature maps of these three scales correspond to large, medium and small coronary stenosis lesion targets. Among them, the convolution block includes convolution layer, batch normalization layer and activation function layer in sequence. In this embodiment, the RCS-OSA module is as follows: Figure 2 As shown in FIG. 1 , it includes two RepVGG modules, two RCS modules and one channel reshaping module; wherein the input of the RCS-OSA module is first sent to the RepVGG module 1 for feature extraction, the output feature map of the RepVGG module 1 is sent to the channel reshaping module and the RCS module 1 respectively, the output feature map of the RCS module 1 is sent to the channel reshaping module and the RCS module 2 respectively, and the output feature map of the RCS module 2 is also sent to the channel shuffling module; after the channel shuffling module performs channel shuffling operations on the three inputs respectively, the channel shuffling feature maps are spliced ​​through the splicing layer to obtain feature maps with the same dimension as the input, and then sent to the RepVGG module 2, and the output of the RCS-OSA module is obtained based on its output. Among them, the channel shuffling operation is specifically as follows: first, the input four-dimensional tensor is reshaped into a five-dimensional tensor, and the number of channels is divided into multiple groups, each group containing an equal number of channels; then, the order of the channels is changed by the transposition operation to enhance the diversity of features. Finally, the tensor is resized to its original four-dimensional form. This process effectively shuffles the channels, thereby improving the feature expressiveness of the model.

[0049] See also Figure 2In this embodiment, the SPPF module is specifically configured as follows: the input of the SPPF module first passes through a convolution block, and then its output is pooled through three parallel maximum pooling layers respectively, and then the three-way pooling results are spliced ​​through a splicing layer and sent to a convolution block, and the output of the SPPF module is obtained based on the output of the convolution block.

[0050] See also Figure 3 In this embodiment, the structure of the RCS module is specifically set as follows: given an input tensor with a feature dimension of C×H×W, after the channel splitting operator, it is divided into two different channel tensors with equal dimensions, namely C / 2×H×W. For one of the tensors, identical branches, 1×1 convolutions, and 3×3 convolutions are used. After multi-branch training of one of the tensors, it is concat to the other tensor in a channel manner, and channel shuffling is applied to enhance the information fusion between the two tensors. Afterwards, the RCS module is repeatedly stacked in the RCS-OSA module to ensure feature reuse and enhance the information flow between different channels between features of adjacent layers. In order to reduce the degree of network fragmentation, only three feature cascades are retained on the one-time aggregation path, which can reduce the network computational burden and reduce memory usage.

[0051] S4: Weighted dual-stream path multi-scale feature fusion. The feature fusion network is responsible for fusing and optimizing the features of three different scales extracted by the deep feature extraction network. The dual-stream multi-scale feature fusion network established in the present invention is as follows: Figure 4 As shown in the figure, it captures the underlying detail features and high-level semantic features of the image through two parallel paths, the underlying feature extraction path and the high-level feature extraction path. The underlying feature extraction path mainly extracts detail information such as edges and textures through a shallow convolutional network, while the high-level feature extraction path extracts semantic information features through a deep convolutional network. These two features will be fused together at each layer of the feature pyramid to form a rich and comprehensive feature representation.

[0052] In this embodiment, the dual-stream multi-scale feature fusion network is specifically set as follows:

[0053] The three spatial features of different scales output by the deep feature extraction network are defined as large-scale spatial features, medium-scale spatial features, and small-scale spatial features.

[0054] The small-scale spatial features are upsampled to the dimension of the medium-scale spatial features, and then weighted concatenated with the medium-scale spatial features to obtain the first concatenated features; the first concatenated features are upsampled to the dimension of the large-scale spatial features, and then weighted concatenated with the large-scale spatial features to obtain the second concatenated features; then, the second concatenated features (P3) are downsampled to the medium-scale spatial features, and then weighted concatenated with the first concatenated features to obtain the third concatenated features (P4), and the third concatenated features are downsampled to the dimension of the small-scale spatial features, and then concatenated with the small-scale spatial features to obtain the fourth concatenated features (P5); finally, enhanced spatial features of three scales are obtained based on the second concatenated features, the third concatenated features and the fourth concatenated features.

[0055] S5: Spatiotemporal collaborative analysis. The spatiotemporal collaborative analysis network architecture of the present invention is as follows: Figure 5 As shown in the figure, after passing through the spatial feature constraint network, each frame of coronary angiography image obtains enhanced spatial features of three different scales. The enhanced spatial features of different scales are then input into the spatiotemporal collaborative analysis module respectively, and a temporal connection is constructed between the sequence frames to obtain a spatiotemporal feature map containing the past frame information. The spatiotemporal feature map is then fused with the spatial feature map of the current frame to further enhance the effective feature information and obtain the final multi-scale spatiotemporal feature.

[0056] Finally, the multi-scale spatiotemporal features of each frame are input into the detection head for target detection of coronary artery stenosis to obtain the coronary artery stenosis detection result of the current frame, including: the object category of the target frame, the foreground and background class information of the target frame, and the position information of the target frame.

[0057] That is, in the embodiment of the present invention, the final spatiotemporal features (spatiotemporal features at three scales) are used for judgment, and the stenosis target boundary box (i.e., positioning detection), category (whether it is a stenosis lesion, i.e., classification detection) and corresponding probability (confidence of the prediction box, i.e., regression detection) are output at different scales. The inspection results of coronary artery stenosis lesion detection are obtained by calculation and screening to improve the accuracy and robustness of coronary artery stenosis lesion detection.

[0058] In one embodiment, the loss function used in the model parameter training of the deep feature extraction network, dual-stream path multi-scale feature fusion network, and spatiotemporal information collaborative analysis module constructed in this application is designed as follows:

[0059] According to the characteristics of coronary angiography sequence images, this embodiment sets a new bounding box loss function, which consists of two parts:

[0060] In the first part, the loss of the "bounding box distribution probability" and the "distribution probability" of the label is calculated through DFL (Debiased Focal Loss), and the loss of the position around the label is learned to enhance the generalization of the model in complex situations (coronary blood vessels move with the heartbeat), thereby optimizing each edge;

[0061] In the second part, the "bounding box distribution probability" is restored to the prediction box, and the loss of the prediction box and the "actual box" of the label is calculated through SIOU (Simplified IoU) loss, so as to optimize the prediction box as a whole.

[0062] As a possible implementation method, in this example, in the two-stream multi-scale feature fusion network, for any two features to be weighted concatenated (also called feature maps), the two weights involved are learnable parameters, which are defined as ω a and ω b , first randomly initialize it, and then normalize the weights to the interval [0,1], the normalization method is as follows:

[0063]

[0064] Among them, ε is a preset learning rate used to avoid instability in weights. In this embodiment, it is set to 0.0001.

[0065] For the two feature maps F to be weighted concatenated a and F b Weighted separately, and then fused and spliced ​​to form the feature map F out , the expression is as follows:

[0066] F out =Concat(ω a F a ,ω b F b )

[0067] As a possible implementation method, in the present invention, the spatiotemporal collaborative analysis module adopts the ST-LSTM unit which can simultaneously integrate the temporal information and the local spatial information, and the expanded structure is as follows: Figure 6 The specific calculation formula is as follows:

[0068]

[0069] Among them, σ() is the sigmoid activation function, * and ⊙ represent the convolution operator and Hadamard product respectively, i t , f t , g t , o tRespectively represent the input gate, forget gate, input modulation gate and output gate. The input gate determines how much new information will be written into the cell state; the forget gate determines how much past information will be retained in the cell state; the output gate determines how much information of the cell state will be output to the hidden state of the current time step. xg , W hg , W xi , W hi , W xf , W hf etc. are the convolution kernel parameters, b g , b i , b f etc. are bias parameters, which are continuously updated by network learning. t , i′ t , f′ t are the auxiliary parameters involved in the calculation, is a standard time unit, which is passed from the previous node at t-1 to the current time step in each LSTM cell. is the spatiotemporal memory described in the current section, which is passed vertically from layer l-1 to the current node at the same time step. Finally, the final hidden state of the node is That is, the output of the current time step depends on the fused spatiotemporal memory. These memories from different directions are connected together, and then a 1×1 convolution layer is applied to reduce the dimension, so that the hidden state The same dimension as the memory unit. Different from simple memory concatenation, the use of shared output gates for the two memory types to achieve seamless memory fusion can effectively capture the vascular shape deformation and hemodynamic information in the spatiotemporal sequence, thus performing spatiotemporal synergistic analysis.

[0070] In this embodiment, see Figure 5, the enhanced spatial features of three scales (large, medium, and small) of each frame of the sequence are subjected to spatiotemporal collaborative analysis through the ST-LSTM unit. Starting from the second frame of the sequence, the input data of the ST-LSTM unit includes the enhanced spatial features of the current frame, and also includes the output features of the enhanced spatial features of the same scale of the previous frame through the ST-LSTM unit; finally, the output features of the ST-LSTM unit corresponding to the current scale of the current frame are added to the enhanced spatial features of the current scale of the current frame to obtain the final spatiotemporal features of the current scale of the current frame. For example, for the small-scale enhanced spatial features of the second frame, they are first sent to the ST-LSTM unit, and the input of the ST-LSTM unit also includes the output features of the ST-LSTM unit corresponding to the small scale of the first frame, and then the output features of the ST-LSTM unit corresponding to the small scale of the second frame are added to the small-scale enhanced spatial features of the second frame to obtain the final small-scale spatiotemporal features of the second frame, so that each frame will obtain the spatiotemporal features of three scales. Then, the spatiotemporal features of the three scales of each frame are input into the detection head ( Figure 5 The Dectect in the coronary artery was used for target detection of coronary artery stenosis.

[0071] In this embodiment, see Figure 7 , the IoU intersection and union ratio of the predicted box A and the real box B and the bounding box loss are defined as follows:

[0072]

[0073] L IoU =1-IoU

[0074] As a possible implementation method, the example of the present invention adopts the SIoU loss function for bounding box regression. The SIoU loss function is composed of four Cost cost functions: Angle cost, Distance cost, Shape cost and IoU cost.

[0075] The angle cost Λ is calculated as follows:

[0076] Λ=1-2*sin 2 (arcsin(x)-4π)

[0077]

[0078] See also Figure 8 , c h is the height difference between the center point of the real box and the predicted box, σ is the distance between the center point of the real box and the predicted box, Represent the x-coordinates of the real box and the predicted box, and the y-coordinates of the real box and the predicted box, respectively.

[0079] You can simplify the formula by substituting the values ​​of x and angle α into the formula:

[0080]

[0081]

[0082] The distance between the true bounding box and the predicted bounding box is calculated as Figure 8 As shown, the distance cost is redefined according to the angle cost defined above:

[0083]

[0084] in, γ=2-Λ.

[0085] The shape cost is defined as:

[0086]

[0087] in, The value of θ defines the cost of the shape and its value is unique for each dataset.

[0088] The final regression loss of SIOU is:

[0089]

[0090] As a possible implementation, the DFL loss in this example is defined as follows:

[0091] DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(yy i )log(S i+1 ))

[0092] In the formula, That is, the predicted value output by the network, That is, the near-prediction value output by the network, y represents the true target label, y i and i+1 Respectively represent the right boundary and the left boundary of the target (y8≤y≤y 8+1 ).

[0093] In order to further analyze the performance of the method proposed in this embodiment, the performance of the basic detection network and the spatiotemporal information collaborative network (spatiotemporal information collaborative analysis module) constructed in the embodiment of the present invention in the detection of sequential coronary angiography images is compared. Fig. 9As shown, the result of the basic detection network is shown on the left, and there are two frames of misdetection, which significantly affects the accuracy of the overall detection. The spatiotemporal information collaborative network on the right successfully detected the coronary artery stenosis lesions in all five frames of images, showing good stability and accuracy. This result shows that the spatiotemporal information collaborative network can more effectively integrate information in the time dimension when processing coronary angiography images, thereby improving the accuracy and reliability of detection. The advantage of the method proposed in the embodiment of the present invention is that it can capture the dynamic changes and correlations between different frames in the image sequence, thereby reducing missed detections and misdetections, and ensuring that the important information of each frame can be fully utilized.

[0094] In summary, the spatiotemporal information collaborative network constructed by the embodiment of the present invention has shown obvious advantages in the detection task of coronary angiography images, can provide more accurate auxiliary diagnosis for clinicians, and has important application value.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0096] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for detecting coronary artery stenosis based on coronary angiography and spatiotemporal collaborative learning, characterized in that: The following steps are involved: Step 1, obtaining original coronary angiography data and doctor's diagnosis information; Step 2: preprocess the original coronary angiography data and doctor's diagnosis information to obtain a sequence of coronary angiography images and corresponding stenosis location information labels; Step 3, constructing a coronary angiography image spatial feature constraint network, which includes a deep feature extraction network and a dual-stream path multi-scale feature fusion network; Among them, the deep feature extraction network is a network structure based on a channel shuffle structure re-parameterized convolution single aggregation module, which is used to extract spatial features of three different scales of each frame of the coronary angiography image of the sequence; The multi-scale feature fusion network of the dual-stream path strengthens and fuses the multi-scale spatial features extracted by the deep feature extraction network through the bottom-up and top-down dual-stream paths to obtain enhanced spatial features of three different scales for each frame of coronary angiography image; the three scales of the enhanced spatial features are consistent with the scales of the spatial features extracted by the deep feature extraction network; Step 4, constructing a spatiotemporal information collaborative analysis module, inputting the three-scale enhanced spatial features of each frame of the extracted sequential coronary angiography images into the spatiotemporal information collaborative analysis module, using the single-frame spatial features as prior knowledge, constructing a temporal connection between the sequence frames to obtain the spatiotemporal features containing past frame information, thereby obtaining the three-scale spatiotemporal features of the current frame containing past frame information, and then fusing the spatiotemporal features with the enhanced spatial features of the corresponding scale of the current frame to obtain the final spatiotemporal features of the current frame; Then, the multi-scale spatiotemporal features of each frame are input into the detection head to perform target detection of coronary artery stenosis, and the coronary artery stenosis detection result of the current frame is obtained.

2. The method according to claim 1, characterized in that The loss functions used for model parameter training of the deep feature extraction network, dual-stream path multi-scale feature fusion network, and spatiotemporal information collaborative analysis module include: Distribution focus loss function and SIoU loss function, where the distribution focus loss function optimizes the probabilities of the two positions closest to the label, one on the left and one on the right, in the form of cross entropy.

3. The method according to claim 1, characterized in that A deep feature extraction network, a stacked network including a Focus layer, a convolutional block, several stacked units consisting of convolutional blocks and structured reparameterized convolutional single aggregation modules, and an SPPF module; Among them, the convolution block includes a convolution layer, a batch normalization layer, and an activation function layer in sequence; The SPPF module is set as follows: the input of the SPPF module first passes through a convolution block, and then its output is pooled through three parallel maximum pooling layers respectively, and then the three-way pooling results are spliced ​​through a splicing layer and sent to a convolution block, and the output of the SPPF module is obtained based on the output of the convolution block.

4. The method according to claim 3, characterized in that The number of stacked units consisting of convolutional blocks and structured reparameterized convolutional single aggregation modules is set to 4.

5. The method according to claim 3, characterized in that The structural reparameterized convolution single aggregation module includes two RepVGG modules, two RCS modules and one channel reshaping module; among them, the input of the RCS-OSA module is first sent to the RepVGG module 1 for feature extraction, the output features of the RepVGG module 1 are respectively sent to the channel reshaping module and the RCS module 1, the output features of the RCS module 1 are respectively sent to the channel reshaping module and the RCS module 2, and the output features of the RCS module 2 are also sent to the channel shuffling module; the channel shuffling module performs channel shuffling operations on the three inputs respectively, and then splices the channel dimensions through the splicing layer, and then sends them to the RepVGG module 2, and obtains the output of the RCS-OSA module based on the output of the RepVGG module 2; The RCS module is set as follows: the input features of the RCS module are divided according to the channel dimension, and divided into two sub-features with different channel numbers and the same feature plane dimension, wherein the first sub-feature is respectively passed through three parallel branches: an identity branch, a convolution layer with a convolution kernel of 1×1, and a convolution layer with a convolution kernel of 3×3, and then the outputs of the three branches are added and mapped through an activation function to obtain the intermediate mapping feature of the first sub-feature; the second sub-feature is spliced ​​with the intermediate mapping feature of the first sub-feature through a splicing layer after the channel division operation layer, and the spliced ​​features are then passed through a channel shuffling module to obtain the output of the RCS module; 6. The method according to claim 1, characterized in that The multi-scale feature fusion network of the dual-stream path performs adaptive weighted fusion of the three spatial features of different scales extracted by the deep feature extraction network, including: The spatial features of three different scales are defined as large-scale spatial features, medium-scale spatial features, and small-scale spatial features; The small-scale spatial features are upsampled to the dimension of the medium-scale spatial features, and then weighted concatenated with the medium-scale spatial features to obtain the first concatenated features; the first concatenated features are then upsampled to the dimension of the large-scale spatial features, and then weighted concatenated with the large-scale spatial features to obtain the second concatenated features; Downsampling the second splicing feature to a medium-scale spatial feature, and then performing weighted splicing with the first splicing feature to obtain a third splicing feature, and downsampling the third splicing feature to a dimension of a small-scale spatial feature, and then splicing with the small-scale spatial feature to obtain a fourth splicing feature; Enhanced spatial features at three scales are obtained based on the second stitching feature, the third stitching feature and the fourth stitching feature.

7. The method according to claim 1, characterized in that The specific settings of the spatiotemporal information collaborative analysis module are: The enhanced spatial features of three scales of each frame of the sequential coronary angiography images are extracted through the long short-term memory network ST-LSTM unit based on spatiotemporal characteristics; If the current frame is the first frame of the sequence, the enhanced spatial features of each scale are input into the corresponding ST-LSTM unit, and the output features of the ST-LSTM unit are added to the input features of the ST-LSTM unit to obtain the final spatiotemporal features of the current scale of the current frame; Starting from the second frame of the sequence, the input of the ST-LSTM unit includes the enhanced spatial features of the current frame and the current scale, and also includes the output features of the enhanced spatial features of the same scale of the previous frame through the ST-LSTM unit; Then, the output features of the ST-LSTM unit corresponding to the current scale of the current frame are added to the enhanced spatial features of the current scale of the current frame to obtain the final spatiotemporal features of the current scale of the current frame.

Images