Blood vessel stenosis lesion position detection method and related equipment

Abstract

The invention provides a blood vessel stenosis lesion position detection method and related equipment, and the detection method comprises the following steps: obtaining a coronary angiography image sequence to be detected, the coronary angiography image sequence comprising multiple frames of images; and marking a lesion detection frame for each frame of obtained image through the trained detection neural network, and outputting a detected coronary angiography image sequence. Wherein the lesion detection frame marks at least one of the lesion position, the score representing the lesion probability and the lesion type. According to the invention, the position of the lesion can be found more conveniently and accurately, and reference is provided for a treatment scheme.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, specifically to a method and related equipment for detecting the location of vascular stenosis lesions. Background Technology

[0002] Coronary artery stenosis is the root cause of serious conditions such as insufficient blood supply to the heart and decreased blood oxygen levels. It is also a major factor in inducing myocardial infarction and even fatal risks. Therefore, early detection of this lesion is a key factor in treatment.

[0003] With the development of computer technology, automatic or semi-automatic image processing techniques are widely used in the auxiliary diagnosis of coronary artery stenosis. These techniques can automatically extract coronary artery trees, segment blood vessels, detect stenosis, and output quantitative indicators such as the degree of stenosis. For example, Chinese patent CN114266742A discloses a method for detecting stenosis regions in cerebral vascular CTA images. This method uses OpenCV and Libra RCNN convolutional neural networks in Python programming software to process cerebral vascular CTA images, automatically or semi-automatically detecting cerebral vascular stenosis regions. The method includes steps such as image input, feature extraction, global information integration, candidate region generation, and stenosis region determination, aiming to improve the accuracy and efficiency of detection.

[0004] However, this stenosis detection method may focus more on detection in a single frame and lack the ability to track the lesion location across consecutive frames, which could limit its ability to assess dynamic changes in lesions. Furthermore, this stenosis detection method focuses on stenosis detection in cerebral CTA images, and its algorithm and model may be more suitable for the cerebral vascular system. For stenosis detection in other locations such as coronary angiography, corresponding adjustments and optimizations may be necessary. Summary of the Invention

[0005] The purpose of this invention is to provide a method and related equipment for detecting the location of vascular stenosis lesions, which can more conveniently and accurately locate the lesion based on coronary angiography image sequences, providing a reference for treatment plans.

[0006] To achieve the above objectives, this invention proposes a method for detecting the location of vascular stenosis lesions, comprising:

[0007] Obtain a sequence of coronary angiography images to be detected, wherein the sequence of coronary angiography images includes multiple frames;

[0008] The trained detection neural network labels each frame of the acquired image with a lesion detection box, and outputs a sequence of detected coronary angiography images. The lesion detection box marks at least one of the following: lesion location, score representing the probability of lesion, and lesion type.

[0009] In one embodiment, the training method for the detection neural network includes:

[0010] Data preparation: Collect multiple frames of images, pre-label lesion detection boxes on each frame, preprocess the labeled images, and divide the preprocessed images into training set and test set;

[0011] Detection neural network training: Set the parameters of the detection neural network, input the images in the training set into the detection neural network, obtain the predicted lesion detection box on each frame of the image through the detection neural network, calculate the loss function based on the predicted lesion detection box and the pre-labeled lesion detection box, and iteratively update the parameters of the detection neural network through the loss function until the loss function no longer decreases in multiple iterations, and determine that the detection neural network training is complete.

[0012] Detection Neural Network Testing: Input the images in the test set into the trained detection neural network and evaluate the performance of the trained detection neural network.

[0013] In one embodiment, the step of annotating each frame of the acquired image with a lesion detection box using a trained detection neural network and outputting a sequence of detected coronary angiography images includes the following steps:

[0014] Features are extracted from each acquired frame of image, and the corresponding first feature map is generated;

[0015] Generate several candidate boxes at each location in the first feature map;

[0016] Extract the region corresponding to each candidate box from the first feature map, perform pooling operation on the extracted regions, and generate a second feature map;

[0017] Based on the second feature map, the pooled features are classified to obtain the score representing the lesion probability and lesion type of each candidate box. Candidate boxes with scores less than a preset score threshold are filtered out. At the same time, the position of each candidate box is corrected by coordinate point regression. All candidate boxes after classification and coordinate point regression are the lesion detection boxes.

[0018] Output the sequence of coronary angiography images after detection.

[0019] In one embodiment, the method for detecting the location of vascular stenosis lesions further includes: optimizing all lesion detection boxes based on the detected coronary angiography image sequence, wherein the optimization process includes removing redundant lesion detection boxes, merging the same lesion, and filling in one or more of the locations of lesion detection boxes that were not directly detected.

[0020] In one embodiment, the merging of the same lesion or the completion of the detection box for a lesion not directly detected is performed by combining a post-processing algorithm and a trained tracking neural network.

[0021] In one embodiment, the merging of the same lesion includes:

[0022] The lesion detection boxes with the highest scores in the coronary angiography image sequence after the detection are sequentially queried;

[0023] For each lesion detection box with the highest score, the trained tracking neural network is used to track the center of the lesion detection box in the current frame image and its corresponding position in other frames image. The other frames image refers to all other frames image in the coronary angiography image sequence after detection, except for the current frame image.

[0024] If the center of one or more lesion detection boxes is within a preset radius of the center of the lesion detection box in the current frame image, then these lesion detection boxes are merged.

[0025] In one embodiment, completing the position of the lesion detection box that was not directly detected includes:

[0026] In other frames, the positions of one or more lesion detection boxes tracked by the trained tracking neural network;

[0027] Regarding the position of one or more lesion detection boxes in other frame images, if there are no one or more lesion detection boxes within a preset radius of the corresponding position in a specific frame image, the trained tracking neural network is used to complete the corresponding position of one or more lesion detection boxes in the specific frame image.

[0028] The specific frame image refers to the image in which the position of the lesion detection box that was not directly detected needs to be filled in;

[0029] The other frame images refer to all other frame images in the detected coronary angiography image sequence, except for the specific frame image.

[0030] In one embodiment, the training method of the tracking neural network includes:

[0031] Data preparation: Collect multiple sets of images, each set containing several frames. Pre-label two feature locations on each frame. Preprocess the labeled images and divide the preprocessed sets of images into training and testing sets.

[0032] Training the tracking neural network: Set the parameters of the tracking neural network, input each set of images in the training set into the tracking neural network, use two feature location points on one frame of each set of images to predict the predicted location points of other frames, calculate the loss function based on the predicted location points and the pre-labeled feature location points, and iteratively update the parameters of the tracking neural network through the loss function until the loss function no longer decreases in multiple iterations, and then determine that the training of the tracking neural network is complete.

[0033] Tracking Neural Network Test: Input each set of images in the test set into the trained tracking neural network and evaluate the performance of the trained tracking neural network.

[0034] In one embodiment, the tracking neural network is used to: track corresponding coordinate points in other frames based on each labeled coordinate point in one or more frames of images, wherein the other frames are all other frames in the detected coronary angiography image sequence except for one or more frames of images.

[0035] In one embodiment, the step of tracking the corresponding coordinate points in other frames of images based on the coordinate points of each labeled point in one or more frames of images includes: extracting feature information from each frame of images and outputting the feature map corresponding to each frame of images.

[0036] Based on the feature map, for each labeled coordinate point, bilinear interpolation is used to extract the feature vector F corresponding to the current coordinate point from the feature map;

[0037] Based on the feature vector F, the feature map, and the current coordinate point, the neighborhood of the current coordinate point is cropped at multiple scales to generate a feature block. The feature block is then multiplied with the feature maps of other frames to construct the correlation matrix C.

[0038] Based on the feature vector F, the correlation matrix C, and the current coordinate point, the increments of coordinates and features are obtained through a multilayer perceptron (MLP). The coordinates and features are iterated multiple times using the increments to obtain the coordinate point after tracking the current coordinate point.

[0039] The present invention also proposes a device for detecting the location of vascular stenosis lesions, used to implement the method for detecting the location of vascular stenosis lesions as described above, wherein the device for detecting the location of vascular stenosis lesions includes:

[0040] The acquisition unit is used to acquire a sequence of coronary angiography images to be detected, wherein the sequence of coronary angiography images includes multiple frames of images;

[0041] The detection unit is used to annotate each frame of the acquired image with a lesion detection box through a trained detection neural network, and output the detected coronary angiography image sequence. The lesion detection box marks at least one of the following: lesion location, score representing the probability of lesion, and lesion type.

[0042] The present invention also proposes a device for detecting the location of vascular stenosis lesions, including a memory and a processor;

[0043] The memory is used to store computer programs;

[0044] The processor is configured to implement the method for detecting the location of vascular stenosis lesions as described above when executing the computer program.

[0045] The present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for detecting the location of vascular stenosis lesions as described above.

[0046] The beneficial effects of the method and related equipment for detecting the location of vascular stenosis lesions of the present invention are as follows:

[0047] 1. The method for detecting the location of vascular stenosis lesions of the present invention only requires coronary angiography sequence to detect the lesion location, which is low in cost.

[0048] 2. This invention uses a deep neural network, which integrates the features of the training data, making the results more robust.

[0049] 3. The present invention also employs a tracking network, which utilizes the continuity of time to avoid the situation where a few frames in the sequence fail to be detected and cannot be predicted, and also eliminates some false detections. Attached Figure Description

[0050] Figure 1 This is a schematic flowchart of a method for detecting the location of vascular stenosis lesions according to an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of a lesion detection frame according to an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram illustrating the prediction of prediction points in other frames of an embodiment of the present invention using two feature location points on one frame of each group of images.

[0053] Figure 4 This is a schematic diagram of the Faster RCNN network structure according to an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of a Region Candidate Module (RPN) according to an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of a pips++ network structure according to an embodiment of the present invention; and

[0056] Figure 7This is a schematic block diagram of a vascular stenosis lesion location detection device according to an embodiment of the present invention.

[0057] Figure Labels

[0058] 100. Device for detecting the location of vascular stenosis lesions; 10. Acquisition unit; 20. Detection unit; 21. First feature extraction module; 22. Region candidate module; 23. ROI pooling module; 24. Classification and regression module; 30. Tracking unit; 31. Second feature extraction module; 32. Similarity calculation module; 33. Iterative update module. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0060] like Figure 1 As shown, this invention proposes a method for detecting the location of vascular stenosis lesions, comprising the following steps:

[0061] Step S1: Obtain the coronary angiography image sequence to be detected. The coronary angiography image sequence includes multiple frames of images.

[0062] In this preferred embodiment, the frame rate is generally 15fps due to the changes in blood vessel dilation and contraction between frames.

[0063] Step S2: The trained detection neural network annotates each acquired image frame with a lesion detection box, outputting a sequence of detected coronary angiography images. The lesion detection box marks at least one of the following: lesion location, a score representing the probability of the lesion, and lesion type. The score on the lesion detection box is used to determine the likelihood that the region within the detection box contains a lesion.

[0064] In step S2, the training method for the detection neural network includes the following steps:

[0065] Data preparation: Collect multiple frames of images and pre-label lesion detection boxes on each frame. Preprocess the labeled images and divide them into training and test sets.

[0066] Detection Neural Network Training: The parameters of the detection neural network are set, and images from the training set are input into the network. The network then generates predicted lesion detection boxes for each frame. A loss function is calculated based on the predicted and pre-labeled lesion detection boxes. The parameters of the detection neural network are iteratively updated using this loss function until it no longer decreases in multiple iterations, at which point the training is considered complete.

[0067] The predicted lesion detection box includes a prediction score and prediction coordinates. In a specific embodiment, a foreground score is generated for each predicted lesion detection box, used for predicting the lesion detection box classification. The foreground score represents the confidence that the predicted lesion detection box contains the target lesion. For the classification of the predicted lesion detection boxes, a softmax loss function is used.

[0068] The predicted coordinate information typically includes the x and y coordinates of a vertex of the predicted lesion detection box, as well as the length and width of the lesion detection box. This predicted coordinate information is used to correct the position of the predicted lesion detection box. For the position of the predicted lesion detection box, a smooth L1 loss function is used.

[0069] Detection Neural Network Testing: Input the images in the test set into the trained detection neural network and evaluate the performance of the trained detection neural network.

[0070] Specifically, the detection neural network was trained using a single GTX 4090 GPU. The batch size was set to 2, meaning the network would process two images simultaneously in each training iteration. Random symmetric processing was applied to the input images in the training set. The output score threshold was set to 0.5. SGD was used as the optimizer, with a learning rate of 0.02, momentum of 0.9, and weight regularization of 0.0001. The training set consisted of 1600 contrast images, and the test set consisted of 400 contrast images.

[0071] Typically, the lesion detection box is a rectangle. For example... Figure 2 As shown, narrowing of a blood vessel generally indicates vascular lesions; therefore, the location marked by the lesion detection box represents the narrowing area. The data annotation method for the detection neural network is as follows: for each image in the coronary angiography sequence, the narrowing area is marked with a rectangle, and the positions of the four vertices of the rectangle are recorded.

[0072] Furthermore, in step S2, the trained detection neural network annotates each frame of the acquired image with a lesion detection box, and outputs the detected coronary angiography image sequence, including the following steps:

[0073] Step S21: Extract features from each acquired frame image and generate the corresponding first feature map.

[0074] Step S22: Generate several candidate boxes at each location in the first feature map.

[0075] Step S23: Extract the region corresponding to each candidate box from the first feature map, perform pooling operation on the extracted region, and generate the second feature map.

[0076] Step S24: Based on the second feature map, the pooled features are classified to obtain the score representing the lesion probability and lesion type for each candidate box, and candidate boxes with scores less than a preset score threshold are filtered out. In this embodiment, the score threshold is preferably 0.5. At the same time, the position of each candidate box is corrected by coordinate point regression. All candidate boxes after classification and coordinate point regression are the lesion detection boxes.

[0077] The classification process in this embodiment is the same as the classification performed when training the detection neural network above. The score is used to determine the possibility that the candidate box contains a lesion.

[0078] Step S25: Output the detected coronary angiography image sequence.

[0079] Step S3: Based on the detected coronary angiography image sequence, optimize all lesion detection boxes. The optimization process includes removing redundant lesion detection boxes, merging the same lesion, and filling in one or more positions of lesion detection boxes that were not directly detected.

[0080] Further, in step S3, redundant lesion detection boxes are removed by filtering out a preset number of lesion detection boxes using non-maximum suppression (NMS). To remove candidate boxes with high overlap and retain the best candidate boxes, NMS is applied. NMS filters out a preset number of candidate boxes by comparing the overlap (IoU) between candidate boxes and a preset NMS threshold. In this embodiment, the preset number is 2000, meaning there are a maximum of 2000 candidate boxes. Furthermore, the preset NMS threshold is 0.7.

[0081] Furthermore, the merging of the same lesion in step S3 includes the following steps:

[0082] The highest-scoring lesion detection boxes are sequentially retrieved from the sequence of coronary angiography images after detection (typically 1 to 2 boxes per sequence). For each highest-scoring lesion detection box, a trained tracking neural network is used to track the corresponding position of the center of the lesion detection box in the current frame in other frames. If the centers of one or more lesion detection boxes are within a preset radius of the center of the lesion detection box in the current frame, these lesion detection boxes are merged. Here, "other frames" refers to all other frames in the sequence of coronary angiography images besides the current frame.

[0083] It should be understood that the operation of merging the same lesion can be performed cyclically. For example, after performing the above operation on the lesion detection box with the highest score, the lesion detection box with the highest score and its corresponding lesion detection box can be excluded, and the next lesion detection box with the highest score can be queried. If the score of the next lesion detection box with the highest score is also greater than the score threshold mentioned above (i.e., it is not filtered out in step S24), then it is considered that there is a lesion in the lesion detection box, and the above operation of merging the same lesion continues.

[0084] In step S3, the positions of the lesion detection boxes that were not directly detected are filled in, including:

[0085] In other frames, the trained tracking neural network tracks the positions of one or more lesion detection boxes. If, for a given position in one or more lesion detection boxes within a preset radius in a specific frame, the trained tracking neural network is used to fill in the corresponding positions of those lesion detection boxes in that specific frame. Here, the specific frame refers to the image where the positions of the undetected lesion detection boxes need to be filled in. Other frames refer to all other frames in the post-detection coronary angiography image sequence, excluding the specific frame. This step uses the tracking network to obtain the corresponding positions to achieve temporal continuity.

[0086] In one specific embodiment, the training method for the tracking neural network includes:

[0087] Data preparation: Collect multiple sets of images, each set containing several frames (preferably three frames in this embodiment). Mark two feature location points on each frame beforehand. See [link to documentation]. Figure 3 The labeled images are preprocessed, and the preprocessed images are divided into training and testing sets.

[0088] Training the tracking neural network: The parameters of the tracking neural network are set, and each set of images in the training set is input into the network. Two feature points from one frame in each set are used to predict the predicted location points in other frames. A loss function is calculated based on the predicted location points and the pre-labeled feature points. The parameters of the tracking neural network are iteratively updated using this loss function until the loss function is less than a second preset threshold, at which point the training is considered complete. In this embodiment, the loss function is L1 smooth loss.

[0089] Tracking Neural Network Test: Input each set of images in the test set into the trained tracking neural network and evaluate the performance of the trained tracking neural network.

[0090] The tracking neural network was trained using a single GTX 4090 GPU. The batch size was set to 16, meaning the network would process 16 images simultaneously in each training iteration. The training input consisted of three images from different time points, each labeled with two feature points. The network used the two coordinates of one image to predict the corresponding coordinates of the other two images. The loss function was L1 smooth loss for the coordinates. The training used ADMW as the optimizer, with a learning rate of 5e-5 and a weight regularization coefficient of 0.001. The training set contained 2000 images and their corresponding points, while the test set contained 500 images and their corresponding points. The tracking neural network was labeled as follows: two points were marked on the main blood vessel, and then these two points were labeled again at intervals of 3-4 frames in the sequence. (See [link to documentation]). Figure 3 .

[0091] Furthermore, the tracking neural network in step S3 is used to: track the corresponding coordinate points in other frames based on the coordinate points of each labeled coordinate point in one or more frames of images. Here, "other frames" refers to all other frames in the detected coronary angiography image sequence, excluding the one or more labeled frames.

[0092] Specifically, based on the coordinates of each labeled point in one or more image frames, the corresponding coordinates in other image frames are traced, including:

[0093] Extract feature information from each frame of the image and output the feature map corresponding to each frame.

[0094] Based on the feature map output in the previous step, for each labeled coordinate point, bilinear interpolation is used to extract the feature vector F corresponding to the current coordinate point from the feature map.

[0095] Based on the feature vector F, the feature map, and the current coordinate point, multi-scale cropping is performed on the neighborhood of the current coordinate point to generate feature blocks. These multi-scale feature blocks are used to improve the robustness and accuracy of tracking. Subsequently, the feature blocks are multiplied with the feature maps of other frames to construct a correlation matrix C, which is used to calculate the correlation between different frames. The correlation matrix C contains information about the frame number T, the neighborhood size P, and the pyramid scale L. The size of the correlation matrix C is (T*P*P*L).

[0096] Based on the feature vector F, the correlation matrix C, and the current coordinate point, the coordinates dx and the feature increment df are obtained through a multilayer perceptron (MLP). The obtained increment is used to iterate the coordinates and features multiple times to obtain the tracked coordinate point x. k =x k-1 +dx.

[0097] This invention also proposes a device 100 for detecting the location of vascular stenosis lesions, used to implement the method for detecting the location of vascular stenosis lesions as described above. The device includes an acquisition unit 10 and a detection unit 20, see [link to relevant documentation]. Figure 7 The acquisition unit 10 is used to acquire a sequence of coronary angiography images to be detected, which includes multiple frames. The detection unit 20 is used to annotate each frame of the acquired image with a lesion detection box using a trained detection neural network, and output the detected coronary angiography image sequence. The lesion detection box marks at least one of the following: lesion location, a score representing the probability of the lesion, and lesion type. In a specific embodiment, the vascular stenosis lesion location detection device further includes a tracking unit 30, see [link to documentation]. Figure 7 The tracking unit 30 is used to merge the same lesion and complete the position of the detection box for lesions that were not directly detected.

[0098] Furthermore, the detection unit 20 includes: a first feature extraction module 21, a region candidate module 22, an ROI pooling module 23, and a classification and regression module 24. In this embodiment, the detection unit 2 is a detection neural network, preferably a Faster R-CNN network, see [link to documentation]. Figure 4 Based on this, in step S2 described above, the trained detection neural network annotates each acquired image frame with a lesion detection box, and outputs the detected coronary angiography image sequence, including the following steps:

[0099] The first feature extraction module 21 receives each acquired image frame, extracts features from each acquired image frame, and generates a corresponding first feature map. In this embodiment, the first feature extraction module 21 extracts image features through a convolutional layer encoder, specifically ResNet50.

[0100] The region candidate module 22 receives the first feature map output by the feature extraction module and generates several candidate boxes at each location in the first feature map. Each candidate box corresponds to a region on the feature map that may contain a lesion. Figure 5As shown, the region candidate module 22 described above includes a convolutional layer, a reshape layer, a softmax layer, and a fully connected layer. The workflow of this region candidate module is as follows: The feature map is processed by a convolutional layer, which first undergoes a 3x3 convolution. The main purpose of this convolutional layer is to further abstract and reduce the dimensionality of the feature map while preserving the spatial location information of the features. The output of the convolutional layer is reshaped, generating several candidate boxes at each location in the feature map. A specific embodiment of generating several candidate boxes at each location in the first feature map using the region candidate module 22 is as follows: Let the resolution of the feature map be n*n, and for each coordinate location, set m candidate boxes, that is, there are n*n*m candidate boxes set on this feature map. Preferably, in this embodiment, the resolution of the feature map is 14*14, and for each coordinate location, there are 9 candidate boxes, for a total of 14*14*9 boxes.

[0101] The Region of Interest (ROI) pooling module 23 receives candidate boxes output by the region candidate module 22 and a first feature map output by the feature extraction module. For each candidate box, the ROI pooling module 23 extracts the corresponding region (i.e., the mapped region of the candidate box on the first feature map) from the first feature map. Since the size of the first feature map is smaller than the original image, the coordinates of the candidate box need to be scaled proportionally to the size of the first feature map. Then, the ROI pooling operation is performed on these regions, pooling them into a second feature map of a fixed size. The second feature maps after pooling are output, and these feature maps have a uniform size, which is convenient for subsequent classification and regression processing. ROI pooling is a special pooling operation that divides the region in the first feature map corresponding to each candidate box into multiple fixed-size pixel blocks. For each pixel block, the ROI pooling operation selects the maximum value from all features in the corresponding region of the original feature map (i.e., the first feature map) as the output value of that pixel block. In this way, regardless of the size of the original candidate box, a fixed-size feature map (i.e., the second feature map) will be obtained after ROI pooling, which facilitates subsequent classification and regression tasks.

[0102] The classification and regression module 24 receives the second feature map output by the ROI pooling module 23. Based on the second feature map, the classification and regression module 24 uses a fully connected layer to classify the pooled features to obtain the score and category of each candidate box, and filters out candidate boxes with scores lower than a preset score threshold. At the same time, coordinate point regression is used to correct the position of each candidate box. All candidate boxes after classification and coordinate point regression are the lesion detection boxes.

[0103] Finally, the detected coronary angiography image sequence is output.

[0104] Furthermore, the tracking unit 30 includes: a second feature extraction module 31, a similarity calculation module 32, and an iterative update module 33. In this embodiment, the tracking neural unit 3 is a tracking neural network, preferably a PIPS++ network, see [link to documentation]. Figure 6 Based on this, the specific process of tracking the position using a neural network is as follows:

[0105] The second feature extraction module 31 extracts feature information from each frame of the image and outputs a feature map corresponding to each frame. The feature extraction module extracts image features through a convolutional layer encoder, here using ResNet50.

[0106] The similarity calculation module 32 receives the feature map output by the second feature extraction module 31. For each labeled coordinate point, it extracts the feature vector F corresponding to the current coordinate point from the feature map using bilinear interpolation. It then performs multi-scale cropping on the neighborhood of the current coordinate point to generate a feature block. The feature block is then multiplied by the feature maps of other frames to construct a correlation matrix C. The correlation matrix C is used to calculate the correlation between different frames. The correlation matrix C contains information about the frame number T, the neighborhood size P, and the pyramid scale L. The size of the correlation matrix C is (T*P*P*L).

[0107] The feature vector F, correlation matrix C, and current coordinates are input into the iterative update module. Through a multilayer perceptron (MLP), the coordinates dx and the feature increment df are obtained. The increment is then used to iterate the coordinates and features multiple times to obtain the tracked coordinates x. k =x k-1 +dx.

[0108] This invention also proposes a device for predicting the location of vascular stenosis lesions based on coronary angiography image sequences, comprising a memory and a processor. The memory stores a computer program, and the processor, when executing the computer program, implements the method described above for predicting the location of vascular stenosis lesions based on coronary angiography image sequences.

[0109] This invention also proposes a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method described above for predicting the location of vascular stenosis lesions based on coronary angiography image sequences.

[0110] The beneficial effects of the method and related equipment for predicting the location of vascular stenosis based on coronary angiography image sequences of the present invention are as follows:

[0111] 1. The method for detecting the location of vascular stenosis lesions of the present invention only requires coronary angiography sequence to detect the lesion location, which is low in cost.

[0112] 2. This invention uses a deep neural network, which integrates the features of the training data, making the results more robust.

[0113] 3. The present invention also employs a tracking network, which utilizes the continuity of time to avoid the situation where a few frames in the sequence fail to be detected and cannot be predicted, and also eliminates some false detections.

[0114] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0115] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0116] The embodiments described above will be further appreciated by those skilled in the art. The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in their functional form. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0117] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0118] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0119] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0120] The embodiments described above are merely further illustrations of the present invention and are not intended to limit the present invention in any other way. The present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding modifications and changes based on the present invention, but all such modifications and changes should fall within the protection scope of the present invention.

Claims

1. A method for detecting the location of vascular stenosis lesions, characterized in that, include: Obtain a sequence of coronary angiography images to be detected, wherein the sequence of coronary angiography images includes multiple frames; The trained detection neural network labels each frame of the acquired image with a lesion detection box and outputs a sequence of detected coronary angiography images. The lesion detection box marks at least one of the following: lesion location, score representing the probability of lesion, and lesion type.

2. The method for detecting the location of vascular stenosis lesions according to claim 1, characterized in that, The training method for the detection neural network includes: Data preparation: Collect multiple frames of images, pre-label lesion detection boxes on each frame, preprocess the labeled images, and divide the preprocessed images into training set and test set; Detection neural network training: Set the parameters of the detection neural network, input the images in the training set into the detection neural network, obtain the predicted lesion detection box on each frame of the image through the detection neural network, calculate the loss function based on the predicted lesion detection box and the pre-labeled lesion detection box, and iteratively update the parameters of the detection neural network through the loss function until the loss function no longer decreases in multiple iterations, and determine that the detection neural network training is complete. Detection Neural Network Testing: Input the images in the test set into the trained detection neural network and evaluate the performance of the trained detection neural network.

3. The method for detecting the location of vascular stenosis lesions according to claim 1, characterized in that, The process of annotating each frame of the acquired image with a lesion detection box using a trained detection neural network and outputting a sequence of detected coronary angiography images includes the following steps: Features are extracted from each acquired frame of image, and the corresponding first feature map is generated; Generate several candidate boxes at each location in the first feature map; Extract the region corresponding to each candidate box from the first feature map, perform pooling operation on the extracted regions, and generate a second feature map; Based on the second feature map, the pooled features are classified to obtain the score and lesion type of each candidate box representing the lesion probability. Candidate boxes with scores less than a preset score threshold are filtered out. At the same time, the position of each candidate box is corrected by coordinate point regression. All candidate boxes after classification and coordinate point regression are the lesion detection boxes. Output the sequence of coronary angiography images after detection.

4. The method for detecting the location of vascular stenosis lesions according to claim 1, characterized in that, Also includes: Based on the detected coronary angiography image sequence, all lesion detection boxes are optimized. The optimization process includes removing redundant lesion detection boxes, merging the same lesion, and filling in one or more positions of lesion detection boxes that were not directly detected.

5. The method for detecting the location of vascular stenosis lesions according to claim 4, characterized in that, The merging of the same lesion or the completion of the detection box for a lesion not directly detected is performed by combining a post-processing algorithm and a trained tracking neural network.

6. The method for detecting the location of vascular stenosis lesions according to claim 5, characterized in that, The term "combined lesions" includes: The lesion detection boxes with the highest scores in the coronary angiography image sequence after the detection are sequentially queried; For each lesion detection box with the highest score, the trained tracking neural network is used to track the center of the lesion detection box in the current frame image and its corresponding position in other frames image. The other frames image refers to all other frames image in the coronary angiography image sequence after detection, except for the current frame image. If the center of one or more lesion detection boxes is within a preset radius of the center of the lesion detection box in the current frame image, then these lesion detection boxes are merged.

7. The method for detecting the location of vascular stenosis lesions according to claim 5, characterized in that, The process of filling in the positions of lesion detection boxes that are not directly detected includes: In other frames, the positions of one or more lesion detection boxes tracked by the trained tracking neural network; Regarding the position of one or more lesion detection boxes in other frame images, if there are no one or more lesion detection boxes within a preset radius of the corresponding position in a specific frame image, the trained tracking neural network is used to complete the corresponding position of one or more lesion detection boxes in the specific frame image. The specific frame image refers to the image in which the position of the lesion detection box that was not directly detected needs to be filled in; The other frame images refer to all other frame images in the detected coronary angiography image sequence, except for the specific frame image.

8. The method for detecting the location of vascular stenosis lesions according to claim 5, characterized in that, The training method for the tracking neural network includes: Data preparation: Collect multiple sets of images, each set containing several frames. Pre-label two feature locations on each frame. Preprocess the labeled images and divide the preprocessed sets of images into training and testing sets. Training the tracking neural network: Set the parameters of the tracking neural network, input each set of images in the training set into the tracking neural network, use two feature location points on one frame of each set of images to predict the predicted location points of other frames, calculate the loss function based on the predicted location points and the pre-labeled feature location points, and iteratively update the parameters of the tracking neural network through the loss function until the loss function no longer decreases in multiple iterations, and then determine that the training of the tracking neural network is complete. Tracking Neural Network Test: Input each set of images in the test set into the trained tracking neural network and evaluate the performance of the trained tracking neural network.

9. The method for detecting the location of vascular stenosis lesions according to claim 5, characterized in that, The tracking neural network is used to: track the corresponding coordinate points in other frames of images based on the coordinate points of each labeled coordinate point in one or more frames of images, wherein the other frames of images are all other frames in the detected coronary angiography image sequence except for one or more frames of images.

10. The method for detecting the location of vascular stenosis lesions according to claim 9, characterized in that, The step of tracing the corresponding coordinate points in other frames of images based on the coordinate points of each labeled point in one or more frames of images includes: Extract the feature information from each frame of the image and output the feature map corresponding to each frame of the image. Based on the feature map, for each labeled coordinate point, bilinear interpolation is used to extract the feature vector F corresponding to the current coordinate point from the feature map; Based on the feature vector F, the feature map, and the current coordinate point, the neighborhood of the current coordinate point is cropped at multiple scales to generate a feature block. The feature block is then multiplied with the feature maps of other frames to construct the correlation matrix C. Based on the feature vector F, the correlation matrix C, and the current coordinate point, the increments of coordinates and features are obtained through a multilayer perceptron (MLP). The coordinates and features are iterated multiple times using the increments to obtain the coordinate point after tracking the current coordinate point.

11. A device for detecting the location of a vascular stenosis lesion, used to implement the method for detecting the location of a vascular stenosis lesion as described in any one of claims 1 to 10, characterized in that, The vascular stenosis lesion location detection device includes: The acquisition unit is used to acquire a sequence of coronary angiography images to be detected, wherein the sequence of coronary angiography images includes multiple frames of images; The detection unit is used to annotate each frame of the acquired image with a lesion detection box through a trained detection neural network, and output the detected coronary angiography image sequence. The lesion detection box marks at least one of the following: lesion location, score representing the probability of lesion, and lesion type.

12. A device for detecting the location of vascular stenosis lesions, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the method for detecting the location of vascular stenosis lesions as described in any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method for detecting the location of vascular stenosis lesions as described in any one of claims 1 to 10.

Images