Method and device for constructing coronary artery angiography key frame selection model and medium

Abstract

The invention belongs to the field of intelligent medical treatment, and particularly relates to a construction method of a coronary artery angiography key frame selection model, equipment and a medium, and the method comprises the steps: obtaining a coronary artery angiography sequence set; marking each frame of image in the coronary artery angiography sequence set to obtain an image score, wherein the dimensions of the image score comprise a clarity dimension and an integrity dimension; inputting any two frames of images in the coronary artery angiography sequence set into a comparison network to obtain a predicted frame with a higher image score, comparing the predicted frame with the higher image score with an actual frame with a higher image score, and optimizing the comparison network according to a comparison result, and a trained comparison network is obtained until a stop condition is reached. The clinical standard of key frame selection is put forward for the first time, the standard considers multiple dimensions of key frame selection, key frame selection based on the score of the standard is more objective, and therefore a more accurate result is obtained through training of the key frame selection model.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical treatment, and more specifically, to a method, device, medium and program product for constructing a coronary angiography key frame selection model. Background Art

[0002] Coronary angiography is a common and effective method for diagnosing coronary atherosclerotic heart disease (CHD). It is a relatively safe and reliable invasive diagnostic technique, which is now widely used in clinical practice and is considered the "gold standard" for diagnosing CHD. Coronary angiography is the basis of interventional treatment for CHD. Currently, more than 1,636,055 patients undergo coronary intervention in my country each year. Theoretically, the amount of coronary angiography may reach several times the number of coronary interventions.

[0003] Only by accurately interpreting coronary angiography can we obtain a clinical diagnosis and guide the next treatment strategy. Coronary angiography often collects angiography of multiple blood vessels in multiple positions. To interpret a sequence of images (one position) and judge key issues such as the condition of the coronary artery and the degree of stenosis, it is necessary to find a frame with complete contrast agent filling and clear blood vessels from all the frames of this sequence of images. This frame is the key frame. Therefore, the detection and extraction of the key frame number is an important step in the interpretation and judgment of coronary angiography.

[0004] The problems of the current coronary angiography key frame detection algorithm include:

[0005] 1. Some studies only focus on the key frame extraction of some coronary arteries in some body positions, which is not suitable for clinical application;

[0006] 2. Insufficient data volume. All algorithms use relatively small data volumes, and cannot accurately handle some special imaging cases such as poor angiography quality and abnormal blood flow velocity.

[0007] 3. Insufficient image accuracy. For example, training on images of 128x128 pixels cannot meet the needs of accurate judgment under clinical high-resolution images.

[0008] 4. Some algorithms are based only on the absolute threshold judgment rule, without considering the comparison of different frames in the video sequence, and lack the ability to select the best from the best;

[0009] 5. Unable to achieve full automation: Some algorithms require manual intervention, and frames that do not meet the candidate frame standards are manually removed;

[0010] 6. High computational cost and complexity: Some algorithms are relatively complex and computationally expensive, and real-time performance is difficult to guarantee, especially when processing large-scale data. Summary of the invention

[0011] In view of the above problems, the present invention provides a method for constructing a coronary angiography key frame selection model, which uses data processing and feature extraction to capture information in time series data, and proposes improvements to the parameters of the disease prediction model and performs effective optimization, thereby laying a foundation for subsequent automatic film reading, lesion analysis and diagnosis.

[0012] The present application (first aspect) discloses a method for constructing a coronary angiography key frame selection model, the method comprising:

[0013] Obtain a coronary angiography sequence set;

[0014] Annotating each frame of the coronary angiography sequence set to obtain an image score, wherein the dimensions of the image score include: a clarity dimension and a completeness dimension; the clarity dimension includes the clarity of the main target blood vessel segment, and the completeness dimension includes the filling of the region of interest and the completeness of the envelope;

[0015] Input any two frames of images in the coronary angiography sequence set into the comparison network to obtain a frame with a higher predicted image score, compare the predicted frame with a higher actual image score with the frame with a higher actual image score, and optimize the comparison network according to the comparison results until a stop condition is reached to obtain a trained comparison network.

[0016] Furthermore, the vascular contour clarity evaluates whether the main target vascular segment that should be observed in a specific body position is clear;

[0017] Optionally, the main target vascular segment includes various vascular segments and positional relationships of the vascular segments, and the various vascular segments include one or more of the following blood vessels: the proximal and middle segments of the anterior descending branch, the bifurcation of the anterior descending branch and the diagonal branch, the left main trunk, the anterior descending branch, the lesion of the circumflex branch opening, the body of the LCX, the opening and body of the obtuse marginal branch, the opening of the right coronary artery, and the starting part of the right coronary artery to the posterior descending branch;

[0018] Optionally, the filling and envelope integrity of the region of interest include: the vascular filling and envelope integrity of the main target blood vessel segment.

[0019] Furthermore, the dimensions of the image scoring also include: overlap shortening dimension, which includes vascular overlap and vascular shortening;

[0020] Optionally, the dimension of the image scoring further includes: an illumination condition dimension, wherein the illumination condition dimension includes exposure and illumination field size;

[0021] Optionally, the image scoring dimension also includes: an overall evaluation dimension, and the overall evaluation dimension includes whether the overall quality of the image meets clinical diagnosis requirements.

[0022] Furthermore, the vascular contour clarity evaluates whether the main target blood vessels to be observed in a specific body position are clear; optionally, the image detail display capability evaluates the clarity of the anatomical structure details in the image.

[0023] Optionally, the vascular tree filling completeness evaluates whether the main target vascular segments that should be observed in a specific body position are filled completely.

[0024] Optionally, the diagnostic region of interest envelope fully evaluates whether the main target vascular segment that should be observed in a specific body position is within the image span.

[0025] Optionally, the vascular overlap assessment determines whether the main target vascular segment or lesion that should be observed in a specific body position affects the interpretation due to overlap.

[0026] Optionally, the vascular shortening assessment determines whether the main target vascular segment or lesion that should be observed in a specific body position affects the interpretation due to shortening.

[0027] Optionally, the irradiation condition dimension evaluates whether there is overexposure or poor contrast during irradiation, wherein the overexposure is caused by poor masking of the overly bright area of ​​the lung field, which affects the contrast of the region of interest; the poor contrast is caused by too narrow dynamic range of contrast between different tissues and too low differentiation; the irradiation field size evaluates whether the irradiation field is too large or too small;

[0028] Optionally, the visible angiography guides the catheter assessment image to determine whether it contains the catheter segment;

[0029] Optionally, whether there is a device shadow in the blood vessel is used to evaluate whether there is an interventional diagnosis and treatment related device in the blood vessel; whether there is an abnormality outside the blood vessel is used to evaluate whether there is an implant and / or a foreign body in the catheter bed, and the implant includes one or more of the following: a pacemaker, an electrode wire, a CABG postoperative cerclage wire, an aortic stent, and an artificial valve;

[0030] Optionally, whether the overall image quality meets the clinical diagnosis requirements is divided into three levels: when the number of bad items in the clarity dimension, completeness dimension, overlap foreshortening dimension, irradiation condition dimension, and instrument shadow dimension is 0 and the overall image fully meets the clinical diagnosis requirements, it is evaluated as "satisfied"; when the number of bad items is greater than or equal to 1 and less than or equal to 2 and the overall image basically meets the clinical diagnosis requirements, it is evaluated as "acceptable"; when the number of bad items is greater than 2 and the overall image does not meet the minimum requirements for clinical diagnosis, it is evaluated as "unsatisfactory".

[0031] Furthermore, the comparison network is constructed based on the Siamese network, and the comparison process includes:

[0032] Step 1: The two frames of images I1 and I2 to be compared are respectively extracted through the two branches of the Siamese network through the neural network to obtain the feature maps F1 and F2 of the two frames of images;

[0033] Step 2: Based on the feature maps F1 and F2 of the two frames of images, the bidirectional feature differences V1 and V2 are calculated, and the calculation is expressed as: V1 = F1-F2, V2 = F2-F1;

[0034] Step 3: Concatenate the bidirectional feature differences to obtain bidirectional difference fusion features:

[0035] V bilateral =[V1;V2]

[0036] Step 4: The bidirectional fusion features are input into the fully connected layer to output a frame with a higher predicted image score.

[0037] Furthermore, the comparison process also includes:

[0038] The feature maps F1 and F2 of the two frames of images are respectively obtained by global average pooling and maximum pooling to obtain an average pooling feature map and a maximum pooling feature map, and the feature map of feature fusion is obtained by splicing the average pooling feature map and the maximum pooling feature map;

[0039] The feature graph of the feature fusion is subjected to the attention mechanism to obtain the attention weight graphs A1 and A2;

[0040] The attention weight maps A1, A2 are element-wise multiplied with the feature maps F1, F2 to obtain the weighted feature map F1 ′ and F2 ′ ;

[0041] Weighted feature map F1 based on two frames of images ′ and F2 ′ The bidirectional feature difference V1, V2 is calculated and expressed as: V1 = F1 ′ -F2 ′ ,V2=F2 ′ -F1 ′ ;

[0042] The bidirectional feature differences are concatenated to obtain bidirectional fusion features, which are then input into a fully connected layer to output a frame with a higher predicted image score.

[0043] The present application also discloses a method for selecting a key frame for coronary angiography, the method comprising:

[0044] Obtain the coronary angiography sequence of the subject,

[0045] Any two image frames in the coronary angiography sequence are input into a trained comparison network to obtain a winning frame. The winning frame and other frames in the coronary angiography sequence are input into a trained comparison network to obtain an updated winning frame, which is iterated in sequence until the iteration is completed to obtain the final winning frame as a key frame. The trained comparison network is constructed according to the construction method.

[0046] Further, the method includes: firstly inputting the coronary angiography sequence of the subject to be tested into a preliminary screening network to obtain candidate key frames, then inputting any two frames of the candidate key frames into a trained comparison network to obtain a winning frame, inputting the winning frame and other frames in the coronary angiography sequence into the trained comparison network to obtain an updated winning frame, iterating in sequence until the iteration is completed to obtain the final winning frame as the key frame, and the trained comparison network is constructed according to the construction method;

[0047] The method for constructing the initial screening network is as follows: labeling each frame of the image in the coronary angiography sequence set to obtain a label, wherein the label is divided into unqualified and qualified, the coronary angiography sequence set and the corresponding label are input into a classifier to obtain a predicted label, the predicted label is compared with the label corresponding to the image, and the initial screening classifier is optimized according to the comparison result until the stopping condition is reached to obtain the initial screening classifier.

[0048] Furthermore, the method includes: inputting the first and second frames of the coronary angiography sequence into a trained comparison network to obtain a winning frame, inputting the winning frame and the third frame of the coronary angiography sequence into a trained comparison network to obtain an updated winning frame, and iterating in sequence until the iteration is completed to obtain the final winning frame as the key frame.

[0049] The second aspect of the present application discloses a system for constructing a coronary angiography key frame selection model, comprising:

[0050] Acquisition unit: used for acquiring coronary angiography sequence set;

[0051] A labeling unit is used to label each frame of the coronary angiography sequence set to obtain an image score, wherein the dimensions of the image score include: clarity dimension and completeness dimension; the clarity dimension includes the clarity of the vascular contour and the image detail display capability, and the completeness dimension includes the filling completeness of the vascular bundle and the envelope completeness of the diagnostic interest area;

[0052] A comparison network training unit: inputting any two frames of images in the coronary angiography sequence set into the comparison network to obtain a frame with a higher predicted image score, comparing the predicted frame with a higher image score with the actual frame with a higher image score, and optimizing the comparison network according to the comparison results until a stop condition is reached to obtain a trained comparison network.

[0053] The third aspect of the present application discloses a computer device, which includes: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions, and when the program instructions are executed, it is used to execute the steps of the above method.

[0054] A fourth aspect of the present application discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0055] A fifth aspect of the present application discloses a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0056] This application has the following beneficial effects:

[0057] 1. This application proposes a clinical standard for key frame selection for the first time. This standard takes into account multiple dimensions of key frame selection. The selection of key frames based on this standard is more objective, so that the training of the key frame selection model can obtain more accurate results.

[0058] 2. The key frame selection model of this application adopts a two-step strategy. First, the candidate key frames are obtained through the preliminary screening classifier, and then the key frames with the highest scores are obtained through the comparison network. This is more conducive to the model to eliminate interference, and the comparison network can better capture the characteristics of the key frames;

[0059] 3. The contrast network of the key frame selection model of the present application introduces bidirectional difference feature fusion to extract high-level features for judging the quality of two frames, thereby making the selection of key frames more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 is a schematic diagram of a method flow chart provided by the first aspect of an embodiment of the present invention;

[0062] Figure 2 is a schematic diagram of a program product provided by the second aspect of an embodiment of the present invention;

[0063] Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present invention;

[0064] Figure 4 is a schematic diagram of the architecture of an exemplary computing device provided by an embodiment of the present invention;

[0065] Figure 5 is a schematic diagram of a storage medium provided by an embodiment of the present invention;

[0066] Figure 6 is a schematic diagram of data flow of a comparison network provided by an embodiment of the present invention;

[0067] Figure 7 is a key frame scoring schematic diagram provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0068] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0069] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The sequence numbers of the operations, such as S101, S102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0071] Figure 1 The present invention provides a flow chart of a method for constructing a key frame selection model based on coronary angiography evaluation, and specifically, the method comprises the following steps:

[0072] S101: Acquire coronary angiography sequence set;

[0073] S102: labeling each frame of the coronary angiography sequence set to obtain an image score, wherein the dimensions of the image score include: a clarity dimension and a completeness dimension; the clarity dimension includes the clarity of the vascular contour and the image detail display capability, and the completeness dimension includes the filling completeness of the vascular bundle and the envelope completeness of the diagnostic interest region;

[0074] S103: Input any two frames of images in the coronary angiography sequence set into a comparison network to obtain a frame with a higher predicted image score, compare the predicted frame with a higher actual image score with the frame with a higher actual image score, and optimize the comparison network according to the comparison result until a stop condition is reached to obtain a trained comparison network.

[0075] In some embodiments, each frame of the coronary angiography sequence set of each subject is scored based on the image standard, and the trained comparison network is obtained by pairwise comparison training.

[0076] In some embodiments, unlike other previous image judgment methods, this method uses two steps of first screening candidate frames of key frames and then comparing and selecting to achieve key frame selection.

[0077] 1. The first step of this method is to use the image classification algorithm to achieve the initial screening of candidate key frames

[0078] The initial screening classifier uses a deep learning-based image classification network to classify frames in the coronary angiography sequence. The network combines the efficient computing power of convolutional neural networks (CNNs) with the advantages of deep feature extraction and is optimized specifically for the characteristics of medical images.

[0079] The input of the network is each frame of the coronary angiography sequence, and the output is the classification label of whether the frame is a key frame (i.e., a candidate key frame or a non-key frame). The specific architecture is based on the characteristics of EfficientNet and is balanced in multiple dimensions of width, depth, and resolution, so as to minimize the number of model parameters and computational complexity while maintaining high classification accuracy.

[0080] Network architecture: The network consists of several convolutional layers, batch normalization layers, ReLU activation functions, and max pooling layers, which gradually deepen the understanding of the image through hierarchical feature extraction. In addition, in order to ensure the recognition of specific imaging details, the network also introduces multiple layers of skip connections to fuse local and global features at different levels. The final fully connected layer outputs a binary classification probability result, indicating that the frame is a candidate keyframe or a non-keyframe.

[0081] Loss function: In order to optimize the classification performance of the model, the cross-entropy loss function (Cross-EntropyLoss) was selected. Its formula is as follows:

[0082]

[0083] where y i is the true label, is the predicted probability, and N is the number of contrast image pairs. This loss function is well adapted to the binary classification problem and can still work effectively in the case of imbalanced categories.

[0084] Dataset preparation: In order to train and optimize the network, the imaging doctors first manually annotated the frames in the coronary angiography sequence to distinguish between candidate key frames and non-key frames. The frames marked as candidate key frames should meet the clinical diagnosis requirements after being judged by professional clinical staff. In the data preparation process, the reasonable division of the training set and the validation set was ensured to avoid data leakage.

[0085] Training process: The Adam optimizer was used during the training process, with an initial learning rate of 0.001. The classification accuracy and loss changes of the validation set were monitored, and overfitting was prevented by an early stopping strategy. In addition, the learning rate scheduler was used to dynamically adjust the learning rate according to the training process, thereby improving the convergence speed and performance of the model.

[0086] After the model training is completed, the number of candidate frames can be freely adjusted by adjusting the probability setting, which is usually set to 0.7 so as not to miss possible key frames.

[0087] 2. In the second step of this method, a method based on image comparison is used to select the final key frame from the alternative frames. The standard for key frame selection is formulated from the perspective of clinical diagnosis. Strictly in accordance with this standard, a large number of annotations are performed on the basis of the candidate key frames screened out in the first step to construct a data set, and a deep learning method is used to construct an image comparison network.

[0088] First, the criteria for key frame selection were developed from the perspective of clinical diagnosis.

[0089] The selection of key frames in previous implementation methods did not clearly define the clinical application standard, and what standard should be used to determine the best key frames. Before labeling, this method clearly proposes the standard for key frame labeling from the perspective of clinical diagnosis and interpretation. The key frames selected according to this standard are more in line with the requirements of clinical doctors for interpretation and diagnosis.

[0090] In some embodiments, the annotation criteria are as follows;

[0091] (1) Fully display the main observed vascular segments

[0092] Different angiographic positions are used to display different vascular segments. For example, the left coronary artery cephalic position is mainly used to observe the proximal and middle segments of the anterior descending artery (LAD) and the bifurcation between the LAD and the diagonal branch (Diagonal); the left coronary artery cephalic position is mainly used to observe the ostial lesions of the left main trunk (LM), LAD, and circumflex artery (LCX), the body of the LCX, and the ostium and body of the obtuse marginal branch (OM); the right coronary artery cephalic position is mainly used to observe the ostium and origin of the right coronary artery (RCA) to the posterior descending artery (PDA).

[0093] Among the many frames in the current angiography sequence, the frames that can best demonstrate the anatomical relationship of the main observed vascular segments should be selected to meet the clinical needs for interpreting important situations such as the presence or absence of lesions in the key observation areas, the degree of stenosis of the lesions, the length and morphological characteristics of the lesions, and the size of the blood vessels.

[0094] Since the blood vessels in an angiography sequence are always in motion as they contract and relax with the cardiac cycle, in principle, frames near the end of diastole are generally selected. However, it should be noted that in this angiography position, the main observation segment of the blood vessels at the end of diastole has overlap, shortening, or local unclearness. Therefore, the most satisfactory frame should be comprehensively selected within this approximate range. This process is selected by the doctor based on his experience in reading films.

[0095] (2) Clear blood vessel contours

[0096] The selected key frames should satisfy the requirement that the vascular contours of the main observed vascular segments are clear. Frames that have blurred vascular contours of the main observed vascular segments due to motion artifacts (blood vessel image blur caused by cardiac rhythmic movement, relative movement of the patient / catheter bed / detector, etc. during image acquisition) or equipment reasons, which affect the clinical interpretation of vascular and lesion morphology, should not be selected as key frames.

[0097] The clarity of vascular outlines was graded according to the above requirements.

[0098] (3) The region of interest is filled and the envelope is complete

[0099] The selected keyframes should satisfy the requirement that the main observed vascular segment is filled and the envelope is complete in this position. At the same time, the entire vascular tree (main branch and branches) should be filled and displayed completely as much as possible to facilitate clinical judgment of the overall condition of the blood vessels and the perfusion range.

[0100] The filling and envelope integrity of the region of interest were graded according to the above requirements.

[0101] (4) Avoid selecting frames where the main observed blood vessels overlap or are shortened.

[0102] Overlapping blood vessels will make it impossible to distinguish the true contours of blood vessels in the main parts, thus affecting clinical interpretation; vascular shortening is due to the fact that the direction of the blood vessels is relatively perpendicular to the screen direction under the angiographic projection position, resulting in the displayed blood vessel length being significantly shorter than the actual length of the blood vessel, affecting the judgment of the true condition of the blood vessel or lesion. When selecting key frames, try to avoid selecting frames with the above conditions.

[0103] The vascular overlap shortening dimension was graded according to the above requirements.

[0104] (5) Avoid selecting frames with overexposure, poor contrast, or unclear layers

[0105] Overexposure: The poor masking of overly bright areas such as the lung field significantly affects the contrast of areas of interest such as blood vessels, affecting clinical interpretation; Poor contrast and unclear layers: The dynamic range of contrast between different tissues is too narrow, and the discrimination is too low, affecting clinical interpretation. When selecting key frames, try to avoid selecting frames with the above conditions.

[0106] The irradiation conditions are graded according to the above requirements.

[0107] (6) Overall evaluation selection

[0108] When the frames in the angiography sequence all have more or less the aforementioned problems such as unclear vascular contours, incomplete filling / envelope of the region of interest, overlapping / shortened blood vessels, overexposure / poor contrast / unclear layers, etc., the goal should be to first select the best frame with the overall quality that best meets the clinical diagnosis requirements.

[0109] In some embodiments, the evaluation includes the following dimensions and criteria: Figure 7 shown):

[0110] 1. Clarity Dimension

[0111] 1.1 Clarity of vascular contours

[0112] Definition: Whether the main target blood vessels that should be observed in this position are clear and unblurred. Note: This includes results caused by various factors such as motion artifacts (blood vessel images are blurred due to cardiac rhythmic movement, relative movement of the patient / catheter bed / detector, etc. during image acquisition) or equipment reasons.

[0113] Options: ["clear", "slightly blurred", "significantly blurred"], scored 2, 1, 0 points respectively;

[0114] 1.2 Image detail display capability

[0115] Definition: The clarity of anatomical details such as small blood vessel branches and tiny lesions in the image

[0116] Options: ["clear", "medium", "poor"], 2, 1, 0 points respectively;

[0117] 2. Completeness Dimension

[0118] 2.1 The vascular tree is fully filled

[0119] Definition: Whether the main target vascular segment to be observed in this position is fully filled

[0120] Options: ["complete", "relatively complete", "incomplete"], 2, 1, 0 points respectively;

[0121] 2.2 Diagnostic interest area envelope is complete

[0122] Definition: Whether the main target vascular segment to be observed in this position is within the image span

[0123] Options: ["complete", "relatively complete", "incomplete"], 2, 1, 0 points respectively;

[0124] 3. Overlapping Shortened Dimensions

[0125] 3.1 Vascular overlap

[0126] Definition: The main target vascular segment or lesion that should be observed in this position affects the interpretation due to overlap.

[0127] Definition of main parts: trunk; main branch mouth / proximal end; main bifurcation site; important abnormal imaging sites (such as "tumor-like" dilatation / ulcer / calcification / dissection / thrombus / gas embolism / spasm / myocardial bridge / collateral circulation, etc.)

[0128] Options: ["None", "Mild overlap", "Severe overlap"], scored 2, 1, 0 points respectively;

[0129] 3.2 Vascular shortening

[0130] Definition: The main target vascular segment or lesion that should be observed in this position is shortened, which affects the interpretation (such as the position for observing the proximal LAD segment but the proximal LAD segment is shortened)

[0131] Options: ["None", "Mild shortening", "Severe shortening"], scored 2, 1, 0 points respectively;

[0132] 4. Irradiation conditions

[0133] 4.1 Overexposure / poor contrast, unclear layers

[0134] Definition: Overexposure: The poor masking of overly bright areas such as the lung field significantly affects the contrast of areas of interest such as blood vessels, affecting the interpretation; Poor contrast and unclear layers: The dynamic range of contrast between different tissues is too narrow, and the discrimination is too low, affecting the interpretation

[0135] ["none", "mild", "severe"], scored 2, 1, and 0 points respectively;

[0136] 4.2 Improper irradiation field control

[0137] Definition: The irradiation field is too large (the blood vessels and other objects are small) or too small (the blood vessels and other objects are too large)

[0138] Options: ["None", "Irradiation field is too large", "Irradiation field is too small"], scored 2, 0, 0 points respectively;

[0139] 5. Overall evaluation dimensions

[0140] 5.1 Does the overall image quality meet clinical diagnostic requirements?

[0141] definition:

[0142] Satisfies: No (1-4) adverse items, and the analyst's overall judgment that the image fully meets the clinical diagnosis requirements;

[0143] Acceptable: There are no more than 2 (1-4) undesirable items, and the analyst's overall judgment is that the image can basically meet the clinical diagnosis requirements (such as other body position compensation);

[0144] Not satisfied: There are more than 2 (1-4) adverse items, and the analyst's overall judgment is that the image does not meet the minimum requirements for clinical diagnosis;

[0145] Options: ["satisfied", "acceptable", "unsatisfied"], 2, 1, 0 points respectively

[0146] Network architecture: Based on the Siamese network infrastructure, innovative improvements are made to the key comparison selection of coronary angiography: by introducing the spatial attention mechanism and bidirectional feature fusion, the coronary angiography frames are compared pairwise to determine the final key frame. The specific improvement methods are as follows:

[0147] Implementation of spatial attention mechanism:

[0148] The spatial attention mechanism aims to enhance the model's attention to important areas in the image. By adaptively adjusting the weight of each pixel or region, the model can better focus on the key parts that affect the image quality.

[0149] (1) Feature map generation: First, the two branches of the Siamese network extract the feature maps of input frames I1 and I2 through a shared convolutional neural network (CNN), and obtain

[0150] F1∈R H×W×C ,F2∈R H×W×C

[0151] Where H, W, and C are the height, width, and number of channels of the feature map, respectively.

[0152] (2) Pooling operation: Each feature map F1 and F2 is subjected to global average pooling and maximum pooling operations respectively to generate the global information of the two feature maps:

[0153] F avg1 =AvgPool(F1),F max1 =MaxPool(F1)

[0154] F avg2 =AvgPool(F2),F max2 =MaxPool(F2)

[0155] (3) Attention map generation: The pooled results are concatenated to form a feature fusion representation, and then a 1X1 convolution operation is performed to generate the attention weight maps A1 and A2 for each image. These two weight maps represent the importance of each spatial position. The generated attention map is normalized by the sigmoid activation function:

[0156] A1=σ(Conv([F avg1 ; F max1 ])),A2=σ(Conv([F avg2 ; F max2 ]))

[0157] (4) Weighted feature map: Multiply the generated attention weight maps A1 and A2 by the original feature maps F1 and F2 element by element to generate the weighted feature map F1 ′ and F2 ′ :

[0158] F1 ′ =A1⊙F1,F2 ′ =A2⊙F2

[0159] Implementation of bidirectional feature fusion:

[0160] Bidirectional feature fusion is used to more fully capture the interactive information between features during pairwise comparison, by calculating the feature differences in two directions and concatenating them together to form a more comprehensive feature representation.

[0161] (1) Feature difference calculation: Calculate the bidirectional differences between weighted feature maps separately:

[0162] V1=F1 ′ -F2 ′ ,V2=F2 ′ -f1 ′

[0163] (2) Bidirectional feature fusion: The two feature difference vectors V1 and V2 are concatenated to form the final bidirectional fusion feature:

[0164] V bilateral =[V1;V2]

[0165] (3) Fully connected layer: The fused feature vector Vbilateral is input into the fully connected layer for processing to extract high-level features for judging the quality of the two frames.

[0166] (4) Output layer: The output of the fully connected layer is a scalar P, which indicates which of the two input frames is more suitable as the final key frame.

[0167] Loss function: The loss function of the model is a weighted binary cross entropy loss function, which aims to maximize the accuracy of frame selection output by the model. For each pair of input frames (I1, I2), the model outputs a probability P, which indicates the probability that the quality of frame I1 is better than that of frame I2. The loss function formula is:

[0168]

[0169] Among them, ω i is the sample weight, y i is the true label, is the probability predicted by the model.

[0170] Dataset preparation

[0171] The candidate frames obtained through step 1 screening in an angiography sequence are labeled, and the only key frame is selected according to the key frame selection criteria formulated by this method from the perspective of clinical diagnosis. The remaining candidate frames can be used to form annotated pairs of two pictures with it as training data, and the data set is prepared according to this method.

[0172] Training process

[0173] During the training process, the dataset is input in batches, and the model generates the quality difference probability of each pair of frames through forward propagation and calculates the weighted binary cross entropy loss. Then, the gradient is calculated using the backpropagation algorithm, and the network weights are updated through an optimizer such as Adam or SGD. As the training progresses, the validation set is used to monitor the generalization performance of the model and adjust hyperparameters (such as learning rate and batch size) in a timely manner to ensure that the model performs well in complex coronary angiography images.

[0174] Get the final keyframe

[0175] Using the comparison network, the candidate key frames in a sequence are compared one by one, and finally the unique key frame of the sequence is obtained.

[0176] In some embodiments, the input data of the initial screening classifier is annotated by two ordinary imaging doctors. The annotation only requires the imaging doctors to judge as good (which can be used as a candidate key frame) and poor based on their clinical experience. The images that the two doctors judge inconsistently are annotated as good and poor after discussion by a more experienced doctor. The imaging experts further perform image scoring in multiple dimensions on the images annotated as good according to the key frame selection criteria proposed in this application, and the image with the highest image score in each angiography sequence is the key frame finally selected.

[0177] A method for selecting a key frame for coronary angiography, the method comprising:

[0178] Obtain the coronary angiography sequence of the subject,

[0179] Any two image frames in the coronary angiography sequence are input into a trained comparison network to obtain a winning frame. The winning frame and other frames in the coronary angiography sequence are input into a trained comparison network to obtain an updated winning frame, which is iterated in sequence until the iteration is completed to obtain the final winning frame as a key frame. The trained comparison network is constructed according to the construction method.

[0180] Further, the method includes: firstly inputting the coronary angiography sequence of the subject to be tested into a preliminary screening network to obtain candidate key frames, then inputting any two frames of the candidate key frames into a trained comparison network to obtain a winning frame, inputting the winning frame and other frames in the coronary angiography sequence into the trained comparison network to obtain an updated winning frame, iterating in sequence until the iteration is completed to obtain the final winning frame as the key frame, and the trained comparison network is constructed according to the construction method;

[0181] The method for constructing the initial screening network is as follows: labeling each frame of the image in the coronary angiography sequence set to obtain a label, wherein the label is divided into unqualified and qualified, the coronary angiography sequence set and the corresponding label are input into a classifier to obtain a predicted label, the predicted label is compared with the label corresponding to the image, and the initial screening classifier is optimized according to the comparison result until the stopping condition is reached to obtain the initial screening classifier.

[0182] In some embodiments, the first and second frames of the coronary angiography sequence of the subject are input into a trained comparison network to obtain a winning frame, and the winning frame and the third frame are input into the trained comparison network to obtain an updated winning frame, and the process is iterated sequentially until the iteration is completed to obtain the final winning frame as the key frame.

[0183] Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present invention, such as Figure 3 As shown, the device may include: one or more processors, and one or more memories; wherein the memories store computer-readable codes, and when the computer-readable codes are run by the one or more processors, the method described above may be executed.

[0184] The processor in this embodiment can be an integrated circuit chip with signal processing capabilities. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, operations and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can be an X86 architecture or an ARM architecture.

[0185] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0186] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Figure 4 The architecture of the computing device 3000 shown in FIG. Figure 4 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as ROM 3030 or hard disk 3070, may store various data or files used for processing and / or communication of the method provided by the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 4 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 4 One or more components of a computing device are shown.

[0187] The embodiment of the present invention also provides a computer-readable storage medium, such as Figure 5As shown, it is a schematic diagram of a storage medium provided by an embodiment of the present invention, and a computer readable instruction 4010 is stored on the computer storage medium 4020. When the computer readable instruction 4010 is executed by the processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0188] The present disclosure also provides a computer program product or a computer program, which implements the steps of the above method when executed by a processor, such as Figure 2 As shown, the computer program product or computer program comprises:

[0189] Acquisition unit 201: used for acquiring a coronary angiography sequence set;

[0190] The labeling unit 202 is used to label each frame of the coronary angiography sequence set to obtain an image score, wherein the dimensions of the image score include: clarity dimension and completeness dimension; the clarity dimension includes the clarity of the vascular contour and the image detail display capability, and the completeness dimension includes the filling completeness of the vascular bundle and the envelope completeness of the diagnostic interest area;

[0191] The comparison network training unit 203 inputs any two frames of images in the coronary angiography sequence set into the comparison network to obtain a frame with a higher predicted image score, compares the predicted frame with a higher actual image score with the frame with a higher actual image score, and optimizes the comparison network according to the comparison result until a stop condition is reached to obtain a trained comparison network.

[0192] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0193] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0194] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0195] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0196] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0197] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0198] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for constructing a coronary angiography key frame selection model, characterized in that: The method comprises: acquiring a coronary angiography sequence set; Annotating each frame of the coronary angiography sequence set to obtain an image score, wherein the dimensions of the image score include: a clarity dimension and a completeness dimension; the clarity dimension includes the clarity of the main target blood vessel segment, and the completeness dimension includes the filling of the region of interest and the completeness of the envelope; Input any two frames of images in the coronary angiography sequence set into the comparison network to obtain a frame with a higher predicted image score, compare the predicted frame with a higher actual image score with the frame with a higher actual image score, and optimize the comparison network according to the comparison results until a stop condition is reached to obtain a trained comparison network.

2. The method for constructing a coronary angiography key frame selection model according to claim 1, characterized in that: The vascular contour clarity evaluates whether the main target vascular segment that should be observed in a specific body position is clear; Optionally, the main target vascular segment includes various vascular segments and positional relationships of the vascular segments, and the various vascular segments include one or more of the following blood vessels: the proximal and middle segments of the anterior descending branch, the bifurcation of the anterior descending branch and the diagonal branch, the left main trunk, the anterior descending branch, the lesion of the circumflex branch opening, the body of the LCX, the opening and body of the obtuse marginal branch, the opening of the right coronary artery, and the starting part of the right coronary artery to the posterior descending branch; Optionally, the filling and envelope integrity of the region of interest includes: the vascular filling and envelope integrity of the main target vascular segment; Optionally, the filling and envelope integrity of the region of interest are graded according to the degree of vascular filling and envelope integrity of the main target blood vessel segment.

3. The method for constructing a coronary angiography key frame selection model according to claim 1, characterized in that: The dimensions of the image scoring also include: overlap shortening dimension, which includes vascular overlap and vascular shortening; Optionally, the dimension of the image score further includes: an illumination condition dimension, wherein the illumination condition dimension includes exposure and illumination field size; Optionally, the dimensions of the image scoring further include: an overall evaluation dimension, wherein the overall evaluation dimension includes whether the overall quality of the image meets the clinical diagnosis requirements; Optionally, the vascular overlap assessment determines whether the main target vascular segment or lesion that should be observed in a specific body position affects the interpretation due to the overlap, and a grade is given according to the degree of influence on the judgment; Optionally, the vascular shortening assessment determines whether the main target vascular segment or lesion that should be observed in a specific body position affects the interpretation due to shortening, and a grade is assigned according to the degree of influence on the judgment; Optionally, the irradiation condition dimension evaluates whether there is overexposure or poor contrast during irradiation, wherein the overexposure is caused by poor masking of the overly bright area of ​​the lung field, which affects the contrast of the region of interest; the poor contrast is caused by too narrow dynamic range of contrast between different tissues and too low differentiation; the irradiation field size evaluates whether the irradiation field is too large or too small; Optionally, whether the overall image quality meets the clinical diagnosis requirements is divided into three levels. When the number of bad items in the clarity dimension, completeness dimension, overlap foreshortening dimension, and illumination condition dimension is 0 and the overall image fully meets the clinical diagnosis requirements, it is evaluated as "satisfied"; when the number of bad items is greater than or equal to 1 and less than or equal to 2 and the overall image can basically meet the clinical diagnosis requirements, it is evaluated as "acceptable"; when the number of bad items is greater than 2 and the overall image does not meet the minimum requirements for clinical diagnosis, it is evaluated as "unsatisfied", and the bad item refers to the dimension that does not meet the requirements.

4. The method for constructing a coronary angiography key frame selection model according to claim 1, characterized in that: The comparison network is based on Siames e The network construction,comparison process includes: Step 1: The two frames of images I1 and I2 to be compared are respectively extracted through two branches of the Siamese network through a neural network to obtain feature maps F1 and F2 of the two frames of images; Step 2: Based on the feature maps F1 and F2 of the two frames of images, bidirectional feature differences V1 and V2 are calculated, and the calculation is expressed as: V1 = F1-F2, V2 = F2-F1; Step 3: Concatenate the bidirectional feature differences to obtain bidirectional difference fusion features: V bilateral =[V1;V2] Step 4: The bidirectional fusion features are input into the fully connected layer to output a frame with a higher predicted image score.

5. The method for constructing a coronary angiography key frame selection model according to claim 4, characterized in that: The comparison process also includes: The feature maps F1 and F2 of the two frames of images are respectively obtained by global average pooling and maximum pooling to obtain an average pooling feature map and a maximum pooling feature map, and the feature map of feature fusion is obtained by splicing the average pooling feature map and the maximum pooling feature map; The feature graph of the feature fusion is subjected to the attention mechanism to obtain the attention weight graphs A1 and A2; The attention weight maps A1, A2 are element-wise multiplied with the feature maps F1, F2 to obtain weighted feature maps F′1 and F′2; Based on the weighted feature maps F′1 and F′2 of the two frames of images, the bidirectional feature differences V1 and V2 are calculated, and the calculation is expressed as: V1 = F′1-F′2, V2 = F′2-F′1; The bidirectional feature differences are concatenated to obtain bidirectional fusion features, which are then input into a fully connected layer to output a frame with a higher predicted image score.

6. A method for selecting key frames for coronary angiography, characterized in that: The method comprises: Obtain the coronary angiography sequence of the subject, Any two image frames in the coronary angiography sequence are input into the trained comparison network to obtain a winning frame, and the winning frame and other frames in the coronary angiography sequence are input into the trained comparison network to obtain an updated winning frame, which is iterated in sequence until the iteration is completed to obtain the final winning frame as the key frame. The trained comparison network is constructed according to the method of claim 1.

7. The method for selecting key frames for coronary angiography according to claim 6, characterized in that: The method comprises: firstly inputting the coronary angiography sequence of the subject to be tested into a preliminary screening network to obtain candidate key frames, then inputting any two frames of the candidate key frames into a trained comparison network to obtain a winning frame, inputting the winning frame and other frames in the coronary angiography sequence into the trained comparison network to obtain an updated winning frame, iterating in sequence until the iteration is completed to obtain the final winning frame as the key frame, and the trained comparison network is constructed according to the method of claim 1; The method for constructing the preliminary screening network is as follows: labeling each frame of the coronary angiography sequence set to obtain a label, wherein the label is divided into unqualified and qualified, the coronary angiography sequence set and the corresponding label are input into a classifier to obtain a predicted label, the predicted label is compared with the label corresponding to the image, and the classifier is optimized according to the comparison result until a stop condition is reached to obtain the preliminary screening network; Optionally, the first and second frames of the image in the coronary angiography sequence of the subject are input into the trained comparison network to obtain a winning frame, and the winning frame and the third frame of the image are input into the trained comparison network to obtain an updated winning frame, and the process is iterated sequentially until the iteration is completed to obtain the final winning frame as the key frame.

8. A computer device, characterized in that: The device comprises: a memory and a processor; the memory is used to store a computer program; the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images