Automatic Tracking Method, Device, Equipment and Storage Medium for Coronary Angiography Marking Points

Abstract

The present invention belongs to the technical field of target tracking, and discloses an automatic tracking method, device, equipment and storage medium for coronary angiography marker points. The method includes: obtaining a target coronary angiography video, and obtaining a marker frame image from the target coronary angiography video; determining target implicit information according to the position coordinates of the marker frame image; inputting the target implicit information and the marker frame image into a trained tracking model to obtain the predicted position coordinates of the marker points in the marker frame image and output layer information; determining whether the marker frame image is the last frame image in the target coronary angiography video; if not, using the next frame image of the marker frame image as a new marker frame image, and using the output layer information as new target implicit information to obtain the predicted position coordinates of the new marker frame image until the marker frame image is the last frame image in the target coronary angiography video. It can realize the tracking of the marker points in the target coronary angiography video.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and particularly to an automatic tracking method, device, equipment and storage medium for coronary angiography marker points. Background Art

[0002] Cardiovascular diseases have become one of the main diseases threatening human life safety. At present, coronary angiography is a very important medical means for patient diagnosis. During the real-time imaging of coronary angiography DSA, doctors need to observe the dynamic image information of lesions or positions of interest. However, as the heart keeps beating, the target position in the dynamic image also keeps moving, resulting in time-consuming and laborious observation by doctors. Summary of the Invention

[0003] The main purpose of the present invention is to provide an automatic tracking method, device, equipment and storage medium for coronary angiography marker points, aiming to solve the technical problem that doctors spend time and effort in observing the target position due to the continuous change of the target position in the dynamic image in the prior art.

[0004] To achieve the above purpose, the present invention provides an automatic tracking method for coronary angiography marker points, and the method includes the following steps:

[0005] S1. Obtain a target coronary angiography video, and obtain a marker frame image from the target coronary angiography video, wherein the position coordinates of the marker points are marked in the marker frame image;

[0006] S2. Determine target implicit information according to the position coordinates of the marker points in the marker frame image;

[0007] S3. Input the target implicit information and the marker frame image into a trained tracking model to obtain the predicted position coordinates of the marker points in the marker frame image and output layer information;

[0008] S4. Determine whether the marker frame image is the last frame image in the target coronary angiography video;

[0009] S5. If not, use the next frame image of the marker frame image as a new marker frame image, and use the output layer information as new target implicit information to repeat steps S3 - S5 until the marker frame image is the last frame image in the target coronary angiography video.

[0010] Optionally, before inputting the target implicit information and the marker frame image into a trained tracking model, it further includes:

[0011] S31. Obtain a coronary angiography video, wherein the position coordinates of the marker points are marked in each frame image of the coronary angiography video;

[0012] S32. Determine the first implicit information according to the position coordinates of the marked points in the t-frame image;

[0013] S33. Input the first implicit information and the t-frame image in the coronary angiography video into the tracking model to obtain the predicted position coordinates of the t-frame image and the second implicit information;

[0014] S34. Determine the prediction error based on the position coordinates and the predicted position coordinates of the t-frame image;

[0015] S35. Update the parameters in the tracking model through backpropagation based on the prediction error;

[0016] S36. Repeat steps S33 - S36 with the (t + 1)-frame image in the coronary angiography video as the new t-frame image and the second implicit information as the new first implicit information until the prediction error is less than the preset value to obtain the trained tracking model.

[0017] Optionally, the determining the first implicit information according to the t-frame image in the coronary angiography video includes:

[0018] Perform dimensionality elevation processing on the position coordinates of the t-frame image through an MLP network to obtain the first implicit information.

[0019] Optionally, the tracking model includes a CNN network and an LSTM network; wherein,

[0020] Inputting the first implicit information and the t-frame image in the coronary angiography video into the tracking model to obtain the predicted position coordinates of the t-frame image and the second implicit information includes:

[0021] Input the t-frame image in the coronary angiography video into the CNN network of the tracking model to obtain the high-dimensional features of the t-frame image, wherein the spatial structure information of the t-frame image is retained in the high-dimensional features;

[0022] After performing a flattening operation on the high-dimensional features to obtain a one-dimensional vector, input the one-dimensional vector and the first implicit information into the LSTM network of the tracking model to obtain the predicted position coordinates of the t-frame image and the second implicit information.

[0023] Optionally, the LSTM network has three gates, namely the input gate, the forget gate, and the output gate; wherein,

[0024] Inputting the one-dimensional vector and the first implicit information into the LSTM network of the tracking model to obtain the predicted position coordinates of the t-frame image and the second implicit information includes:

[0025] Determine the input data through the input gate in the LSTM network, where the input data is determined from the one-dimensional vector and the first hidden information;

[0026] Remove the cleared data in the input data through the forget gate in the LSTM network to obtain the retained data;

[0027] Based on the retained data, obtain the predicted position coordinates of the t-frame image and the second hidden information through the output gate in the LSTM.

[0028] Optionally, before inputting the one-dimensional vector and the first hidden information into the LSTM network of the tracking model, further include:

[0029] Input the one-dimensional vector into the FC layer to obtain a relevant vector;

[0030] Inputting the one-dimensional vector and the first hidden information into the LSTM network of the tracking model to obtain the predicted position coordinates of the t-frame image and the second hidden information includes:

[0031] Input the relevant vector and the first hidden information into the LSTM network of the tracking model to obtain the predicted position coordinates of the t-frame image and the second hidden information.

[0032] Optionally, both the real position coordinates and the predicted position coordinates are two-dimensional coordinates; where

[0033] Determining the prediction error based on the real position coordinates and the predicted position coordinates of the t-frame image includes:

[0034] Determine the real position abscissa and the real position ordinate according to the real position coordinates of the t-frame image;

[0035] Determine the predicted position abscissa and the predicted position ordinate according to the predicted position coordinates of the t-frame image;

[0036] Determine the abscissa difference between the real position abscissa and the predicted position abscissa, and determine the ordinate difference between the real position ordinate and the predicted position ordinate;

[0037] Based on the square of the abscissa difference and the square of the ordinate difference, determine the prediction error.

[0038] In addition, to achieve the above object, the present invention also proposes an automatic tracking device for coronary angiography marker points, and the automatic tracking device for coronary angiography marker points includes:

[0039] An acquisition module, configured to acquire a target coronary angiography video and obtain a marked frame image from the target coronary angiography video, wherein position coordinates of marked points are marked in the marked frame image;

[0040] A determination module, configured to determine target implicit information according to the position coordinates of the marked frame image;

[0041] A prediction module, configured to input the target implicit information and the marked frame image into a trained tracking model to obtain predicted position coordinates of the marked points in the marked frame image and output layer information;

[0042] A judgment module, configured to judge whether the marked frame image is the last frame image in the target coronary angiography video;

[0043] The prediction module is configured to, when it is determined that the marked frame image is not the last frame image in the target coronary angiography video, use the next frame image of the marked frame image as a new marked frame image, and use the output layer information as new target implicit information to determine predicted position coordinates of the new marked frame image.

[0044] In addition, to achieve the above object, the present invention further provides an automatic tracking device for coronary angiography marked points, where the automatic tracking device for coronary angiography marked points includes: a memory, a processor, and an automatic tracking program for coronary angiography marked points stored on the memory and executable on the processor, and the automatic tracking program for coronary angiography marked points is configured to implement the steps of the automatic tracking method for coronary angiography marked points as described above.

[0045] In addition, to achieve the above object, the present invention further provides a storage medium, where an automatic tracking program for coronary angiography marked points is stored on the storage medium, and when the automatic tracking program for coronary angiography marked points is executed by a processor, the steps of the automatic tracking method for coronary angiography marked points as described above are implemented.

[0046] The automatic tracking method for coronary angiography marking points proposed by the present invention obtains a target coronary angiography video and acquires a marked frame image from the target coronary angiography video, wherein the position coordinates of the marking points are marked in the marked frame image; determines target implicit information according to the position coordinates of the marking points in the marked frame image; inputs the target implicit information and the marked frame image into a trained tracking model to obtain the predicted position coordinates of the marking points in the marked frame image and output layer information; determines whether the marked frame image is the last frame image in the target coronary angiography video; if not, uses the next frame image of the marked frame image as a new marked frame image and the output layer information as new target implicit information to obtain the predicted position coordinates of the new marked frame image until the marked frame image is the last frame image in the target coronary angiography video, thereby predicting the predicted position coordinates of the marking points in each frame image of the target coronary angiography video, and thus being able to track the marking points in the target coronary angiography video to facilitate doctors to observe the marking points in the target coronary angiography video. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a schematic structural diagram of an automatic tracking device for coronary angiography marking points in the hardware operating environment related to the embodiment solution of the present invention;

[0048] Figure 2 is a schematic flowchart of the first embodiment of the automatic tracking method for coronary angiography marking points of the present invention;

[0049] Figure 3 is a partial schematic flowchart of the first embodiment of the automatic tracking method for coronary angiography marking points of the present invention;

[0050] Figure 4 is a schematic flowchart of the second embodiment of the automatic tracking method for coronary angiography marking points of the present invention;

[0051] Figure 5 is a schematic diagram of predicting the position coordinates of the marking points in a frame image based on a CNN network and an LSTM network in the second embodiment of the automatic tracking device for coronary angiography marking points of the present invention;

[0052] Figure 6 is a structural block diagram of the first embodiment of the automatic tracking device for coronary angiography marking points of the present invention.

[0053] The implementation, functional features and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0055] Reference Figure 1 , Figure 1 is a schematic structural diagram of an automatic tracking device for coronary angiography marker points in the hardware operating environment involved in the solution of the embodiment of the present invention.

[0056] As Figure 1 shown, the automatic tracking device for coronary angiography marker points may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to implement connection communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless-fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0057] Those skilled in the art can understand that Figure 1 the structure shown in

[0058] does not constitute a limitation on the automatic tracking device for coronary angiography marker points, and may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements. Figure 1 shown, in the memory 1005 as a storage medium, there may be included an operating system, a network communication module, a user interface module, and an automatic tracking program for coronary angiography marker points.

[0059] In Figure 1 the automatic tracking device for coronary angiography marker points shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the automatic tracking device for coronary angiography marker points of the present invention may be provided in the automatic tracking device for coronary angiography marker points. The automatic tracking device for coronary angiography marker points calls the automatic tracking program stored in the memory 1005 through the processor 1001 and executes the automatic tracking method for coronary angiography marker points provided by the embodiment of the present invention.

[0060] Based on the above hardware structure, an embodiment of the automatic tracking method for coronary angiography marker points of the present invention is proposed.

[0061] Refer to Figure 2 , Figure 2 which is a schematic flowchart of the first embodiment of an automatic tracking method for coronary angiography marker points of the present invention.

[0062] In this embodiment, the automatic tracking method for coronary angiography marker points includes the following steps:

[0063] Step S1: Obtain a target coronary angiography video, and obtain a marked frame image from the target coronary angiography video, wherein the position coordinates of the marker points are marked in the marked frame image.

[0064] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program running functions, such as a mobile phone, a tablet computer, a personal computer, etc., or an electronic device or an automatic tracking device for coronary angiography marker points that can implement the above functions. Hereinafter, the automatic tracking device for coronary angiography marker points will be taken as an example to illustrate this embodiment and the following embodiments.

[0065] It should be noted that the coronary angiography video can be a clear image obtained by injecting a contrast agent into the coronary artery to make the coronary artery visible and then taking an X-ray; the target coronary angiography video refers to the coronary angiography video that the doctor needs to observe the marker points; the position coordinates refer to the position coordinates manually marked.

[0066] It can be understood that each frame image in the coronary angiography video will present different shapes with the pulsation of the heart. Therefore, the position of the coronary marker points in the frame image will also change with the pulsation of the heart, which makes it time-consuming and laborious for doctors to observe the coronary marker points.

[0067] It should be noted that in order to ensure the confidence of manual marking, the marker points can be some special points, such as lesion points, stent points, and vascular bifurcation points, etc. The positions of these special points in the coronary artery are more obvious than other points, which can not only facilitate doctors to mark but also improve the accuracy of tracking the marker points.

[0068] It can be understood that in order to track the marker points in all frame images of the target coronary angiography video, the marked frame image with the position coordinates of the marker points manually marked can be the first frame image in the target coronary angiography image, and this embodiment does not make any limitations on this.

[0069] In a specific implementation, it is necessary to first obtain one frame of image from the target coronary angiography video, and the doctor annotates the positions of the marked points in this frame of image, so as to obtain the frame of image with the position information of the marked points annotated, that is, the marked frame of image.

[0070] Step S2: Determine the target implicit information according to the position coordinates of the marked points in the marked frame of image.

[0071] It should be noted that the initial target implicit information is determined according to the position coordinates of the marked points in the initial marked frame of image, and the position coordinates of the marked points in the initial marked frame of image are manually annotated.

[0072] Step S3: Input the target implicit information and the marked frame of image into the trained tracking model to obtain the predicted position coordinates of the marked points in the marked frame of image and the output layer information.

[0073] It should be noted that the marked points in different frames of image in the target coronary angiography video refer to the same anatomical structure points on the coronary artery.

[0074] It should be noted that the position coordinates of the marked points are annotated in the marked frame of image; the trained tracking model refers to the tracking model that can predict the position coordinates of the marked points in the frame of image close to the real position coordinates.

[0075] It should be noted that predicting the predicted position coordinates of the marked points in the target coronary angiography video frame of image through the trained tracking model is more accurate than predicting the predicted position coordinates of the marked points in the target coronary angiography video frame of image through the untrained tracking model.

[0076] It should be noted that the tracking model can be composed of a CNN network + an LSTM network, or can be composed of a CNN network + a GRN network, or can be composed of a CNN network + an RNN network, or can also be composed of a CNN network + a transformer network.

[0077] In one embodiment, as Figure 3 shown, before inputting the marked frame of image into the trained tracking model, it further includes:

[0078] S31. Obtain a coronary angiography video, wherein the position coordinates of the marked points are annotated in each frame of image of the coronary angiography video.

[0079] It should be noted that the doctor can pre-annotate the real position coordinates of the marked points in each frame of image of the coronary angiography video, and the real position coordinates refer to the position coordinates manually annotated.

[0080] S32. Determine the first implicit information according to the position coordinates of the marked points in the t-th frame of image.

[0081] It should be noted that the initial first implicit information is determined according to the position coordinates of the marked points in the initial t-frame image, and the position coordinates of the marked points in the initial t-frame image are manually marked.

[0082] S33. Input the first implicit information and the t-frame image in the coronary angiography video into the tracking model to obtain the predicted position coordinates of the t-frame image and the second implicit information.

[0083] It should be noted that the tracking model refers to an untrained tracking model; the position coordinates of the marked points are marked in the t-frame image.

[0084] It can be understood that it is not accurate enough to input the first implicit information and the t-frame image into the untrained tracking model to predict the predicted position coordinates of the marked points in the t-frame image; therefore, it is necessary to continuously train the tracking model so that the tracking model can be more accurate when predicting the position coordinates of the marked points in the frame image.

[0085] S34. Determine the prediction error based on the position coordinates and the predicted position coordinates of the t-frame image.

[0086] It should be noted that the prediction error can be the sum of the squares of the differences between the real position coordinates and the predicted position coordinates, or the sum of the squares of the distances between the real position coordinates and the predicted position coordinates. This embodiment does not make a limitation on this.

[0087] In one embodiment, both the real position coordinates and the predicted position coordinates are two-dimensional coordinates; where

[0088] The determining of the prediction error based on the real position coordinates and the predicted position coordinates of the t-frame image includes:

[0089] Determine the real position abscissa and the real position ordinate according to the real position coordinates of the t-frame image;

[0090] Determine the predicted position abscissa and the predicted position ordinate according to the predicted position coordinates of the t-frame image;

[0091] Determine the abscissa difference between the real position abscissa and the predicted position abscissa, and determine the ordinate difference between the real position ordinate and the predicted position ordinate;

[0092] Determine the prediction error based on the square of the abscissa difference and the square of the ordinate difference.

[0093] In a specific implementation, the true position coordinates of the marked points in the t-frame image are represented by (x_label, y_label), and the predicted position coordinates of the marked points in the t-frame image are represented by (x_pred, y_pred). Then the calculation formula for the prediction error is loss = (x_label - x_pred) 2 +(y_label - y_pred) 2 。

[0094] S35. Update the parameters in the tracking model through backpropagation based on the prediction error.

[0095] It should be noted that the parameters in the tracking model can be updated once according to each prediction error, or the parameters in the tracking model may be updated multiple times according to each prediction error.

[0096] In a specific implementation, the gradient descent method can be used to perform backpropagation on the prediction error, thereby updating the parameters in the tracking network.

[0097] S36. Take the (t + 1)-th frame image in the coronary angiography video as the new t-frame image and the second hidden information as the new first hidden information, and repeat steps S33 - S36 until the prediction error is less than a preset value, and obtain a trained tracking model.

[0098] It should be noted that the predicted position coordinates of the marked points in the frame image are continuously determined according to the temporal information between the front and back frame images, and then the prediction error is obtained. Then, the parameters in the tracking model are adjusted according to the prediction error. Among them, the front and back frame images refer to the t-frame image and the (t + 1)-th frame image, and the latter frame image is the next frame image. In order to make the tracking model more accurate in predicting the position coordinates of the marked points in the frame image, the tracking model can be trained by continuously obtaining multiple prediction errors. For example, multiple prediction errors can be obtained by continuously repeating steps S33 - S36, or multiple prediction errors can be obtained by continuously repeating steps S33 - S36 with a new coronary angiography video.

[0099] It should be noted that the preset value is set in advance. When the prediction error is less than the preset value, it can be determined that the tracking model is in a converged state at this time, that is, it can be determined that the tracking model at this time is a trained tracking model.

[0100] In this embodiment, multiple prediction errors are determined through a large number of front and back frame images marked with true position coordinates, and then the parameters in the tracking model are adjusted according to the multiple prediction errors, that is, the tracking model is trained, so that the finally trained tracking model can be more accurate when predicting the position coordinates of the marked points in the frame image.

[0101] Step S4: Determine whether the marked frame image is the last frame image in the target coronary angiography video.

[0102] It can be understood that when the marked frame image is not the last frame in the target coronary angiography video, it can be determined that the tracking of the marked points in each frame image of the target coronary angiography image has not been completed. Therefore, it is necessary to determine whether the marked frame image is the last frame image in the target coronary angiography image.

[0103] Step S5: If not, use the next frame image of the marked frame image as the new marked frame image, and use the output layer information as the new target hidden information to repeat steps S3 - S5 until the marked frame image is the last frame image in the target coronary angiography video.

[0104] It can be understood that by predicting the predicted position coordinates of the marked points in the marked frame image by the trained tracking model to be close to the position coordinates of the marked points in the marked frame image, and inputting the first hidden information and the marked frame image into the trained tracking model, the second hidden information of the next frame image of the marked frame image can also be output.

[0105] It should be noted that the second hidden information is determined according to the tracking model.

[0106] In this embodiment, by obtaining the target coronary angiography video and obtaining the marked frame image from the target coronary angiography video, where the position coordinates of the marked points are marked in the marked frame image; determining the target hidden information according to the position coordinates of the marked points in the marked frame image; inputting the target hidden information and the marked frame image into the trained tracking model to obtain the predicted position coordinates of the marked points in the marked frame image and the output layer information; determining whether the marked frame image is the last frame image in the target coronary angiography video; if not, using the next frame image of the marked frame image as the new marked frame image, and using the output layer information as the new target hidden information to obtain the predicted position coordinates of the new marked frame image until the marked frame image is the last frame image in the target coronary angiography video, the predicted position coordinates of the marked points in each frame image of the target coronary angiography video can be predicted, so as to track the marked points in the target coronary angiography video, facilitating doctors to observe the marked points in the target coronary angiography video.

[0107] Reference Figure 4 , Figure 4 is a schematic flowchart of the second embodiment of an automatic tracking method for coronary angiography marked points of the present invention.

[0108] Based on the above first embodiment, the tracking model in the automatic tracking method for coronary angiography marked points in this embodiment includes a CNN network and an LSTM network; where,

[0109] Input the first implicit information and the t-frame image in the coronary angiography video into the tracking model to obtain the predicted position coordinates of the t-frame image and the second implicit information, including:

[0110] Step S331: Input the t-frame image in the coronary angiography video into the CNN network of the tracking model to obtain the high-dimensional features of the t-frame image, where the spatial structure information of the t-frame image is retained in the high-dimensional features.

[0111] It should be noted that the network structure of the CNN network can be VGG or other neural convolutional networks such as ResNet. The high-dimensional features obtained through the CNN network can well retain the spatial structure information of the image.

[0112] Step S332: After flattening the high-dimensional features to obtain a one-dimensional vector, input the one-dimensional vector and the first implicit information into the LSTM network of the tracking model to obtain the predicted position coordinates of the t-frame image and the second implicit information.

[0113] It should be noted that the high-dimensional features can be flattened to obtain a one-dimensional vector through flat.

[0114] It should be noted that the LSTM network can well utilize the temporal information between the front and back frame images to predict the position coordinates of the marked points in the frame image.

[0115] In a specific implementation, as Figure 5 shown, first input the t-frame image in the coronary angiography video into the CNN network of the tracking model to obtain the high-dimensional features of the t-frame image. After flattening the high-dimensional features to obtain a one-dimensional vector, then input the one-dimensional vector and the first implicit information into the LSTM network, where h0 and h1 represent the first implicit information of the frame image, so as to obtain the predicted position coordinates of the t-frame image and the first implicit information of the t+1 frame image.

[0116] In this embodiment, the CNN network can retain the spatial structure information of the image, the LSTM network can utilize the temporal information between the front and back frame images to realize the prediction of the position of the marked points in the frame image, and the CNN network + LSTM network can effectively improve the tracking accuracy of the marked points.

[0117] In one embodiment, the determining the first implicit information according to the t-frame image in the coronary angiography video includes:

[0118] Dimensionality up the position coordinates of the t-frame image through the MLP network to obtain the first implicit information.

[0119] It should be noted that since the input information of the LSTM network needs to be a vector, the position coordinates of the marked points in the t-frame image need to be processed by the MLP network to obtain the first hidden information, and the first hidden information of the marked points in the t-frame image is a vector.

[0120] In one embodiment, the LSTM network has three gates, namely the input gate, the forget gate, and the output gate; among them,

[0121] Inputting the one-dimensional vector and the first hidden information into the LSTM network of the tracking model to obtain the predicted position coordinates and the second hidden information of the t-frame image includes:

[0122] Determine the input data through the input gate in the LSTM network, where the input data is determined from the one-dimensional vector and the first hidden information;

[0123] Remove the clearing data in the input data through the forget gate in the LSTM network to obtain the retained data;

[0124] Based on the retained data, obtain the predicted position coordinates and the second hidden information of the t-frame image through the output gate in the LSTM.

[0125] It should be noted that the input gate is used to receive input information and control the influence of data input on the value of the memory unit; the forget gate is used to forget state information, and some useless timing information can be ignored through the forget gate; the output gate is mainly used to output new information.

[0126] In one embodiment, before inputting the one-dimensional vector and the first hidden information into the LSTM network of the tracking model, it further includes:

[0127] Input the one-dimensional vector into the FC layer to obtain a relevant vector;

[0128] Inputting the one-dimensional vector and the first hidden information into the LSTM network of the tracking model to obtain the predicted position coordinates and the second hidden information of the t-frame image includes:

[0129] Input the relevant vector and the first hidden information into the LSTM network of the tracking model to obtain the predicted position coordinates and the second hidden information of the t-frame image.

[0130] It should be noted that the FC layer is the fully connected layer, and the number of FC layers can be n layers. This embodiment does not limit the number of FC layers.

[0131] It can be understood that the one-dimensional vector needs to be transformed into a relevant vector through the FC layer before being input into the LSTM network for learning.

[0132] In this embodiment, a coronary angiography video is obtained. Among them, the position coordinates of the marked points are marked in each frame image of the coronary angiography video. The first implicit information is determined according to the position coordinates of the marked points in the t-th frame image. The first implicit information and the t-th frame image in the coronary angiography video are input into the tracking model to obtain the predicted position coordinates of the t-th frame image and the second implicit information. Based on the position coordinates and the predicted position coordinates of the t-th frame image, the prediction error is determined. The parameters in the tracking model are updated by backpropagation based on the prediction error. The (t + 1)-th frame image in the coronary angiography video is used as the new t-th frame image and the second implicit information is used as the new first implicit information to repeat steps S33 - S36 until the prediction error is less than the preset value, and a trained tracking model is obtained. In the above manner, after combining the CNN network and the LSTM network, the predicted position coordinates of the marked points in the marked frame image are predicted. First, the CNN network retains the spatial structure information of the image, and then the LSTM network uses the temporal information between the front and rear frame images to predict the position coordinates of the marked points in the marked frame image, thereby effectively improving the accuracy of predicting the position coordinates of the marked points in the marked frame image.

[0133] In addition, an embodiment of the present invention further provides a storage medium, on which an automatic tracking program for coronary angiography marked points is stored. When the automatic tracking program for coronary angiography marked points is executed by a processor, the steps of the automatic tracking method for coronary angiography marked points as described above are implemented.

[0134] Refer to Figure 6 , Figure 6 which is the structural block diagram of the first embodiment of the automatic tracking device for coronary angiography marked points of the present invention.

[0135] As Figure 6 shown, the automatic tracking device for coronary angiography marked points proposed by the embodiment of the present invention includes:

[0136] An acquisition module 10, configured to acquire a target coronary angiography video and acquire a marked frame image from the target coronary angiography video, where the position coordinates of the marked points are marked in the marked frame image;

[0137] A determination module 20, configured to determine target implicit information according to the position coordinates of the marked points in the marked frame image;

[0138] A prediction module 30, configured to input the target implicit information and the marked frame image into the trained tracking model to obtain the predicted position coordinates of the marked points in the marked frame image and the output layer information;

[0139] A judgment module 40, configured to judge whether the marked frame image is the last frame image in the target coronary angiography video;

[0140] The prediction module 30 is configured to, when it is determined that the marked frame image is not the last frame image in the target coronary angiography video, use the next frame image of the marked frame image as a new marked frame image, and use the output layer information as new target hidden information to determine the predicted position coordinates of the new marked frame image.

[0141] It should be understood that the above is only an example and does not constitute any limitation to the technical solution of the present invention. In specific applications, those skilled in the art can set according to needs, and the present invention does not limit this.

[0142] In this embodiment, by obtaining a target coronary angiography video and obtaining a marked frame image from the target coronary angiography video, wherein the position coordinates of marked points are marked in the marked frame image; determining target hidden information according to the position coordinates of the marked points in the marked frame image; inputting the target hidden information and the marked frame image into a trained tracking model to obtain the predicted position coordinates of the marked points in the marked frame image and output layer information; judging whether the marked frame image is the last frame image in the target coronary angiography video; if not, using the next frame image of the marked frame image as a new marked frame image, and using the output layer information as new target hidden information to obtain the predicted position coordinates of the new marked frame image until the marked frame image is the last frame image in the target coronary angiography video, so as to predict the predicted position coordinates of the marked points in each frame image of the target coronary angiography video, thereby being able to track the marked points in the target coronary angiography video to facilitate doctors to observe the marked points in the target coronary angiography video.

[0143] In one embodiment, the prediction module 30 is further configured to:

[0144] Obtain a coronary angiography video, wherein the position coordinates of marked points are marked in each frame image of the coronary angiography video;

[0145] Determine first hidden information according to the position coordinates of the marked points in the t-th frame image;

[0146] Input the first hidden information and the t-th frame image in the coronary angiography video into the tracking model to obtain the predicted position coordinates of the t-th frame image and second hidden information;

[0147] Determine a prediction error based on the position coordinates and predicted position coordinates of the t-th frame image;

[0148] Update the parameters in the tracking model by backpropagation based on the prediction error;

[0149] Take the t+1 frame image in the coronary angiography video as the new t frame image and take the second hidden information as the new first hidden information to obtain the predicted position coordinates of the new t frame image, until the prediction error is less than the preset value, and obtain a trained tracking model.

[0150] In one embodiment, the prediction module 30 is further configured to:

[0151] Perform dimensionality increase processing on the position coordinates of the t frame image through an MLP network to obtain the first hidden information.

[0152] In one embodiment, the tracking model includes a CNN network and an LSTM network; wherein,

[0153] The prediction module 30 is further configured to:

[0154] Input the t frame image in the coronary angiography video into the CNN network of the tracking model to obtain the high-dimensional features of the t frame image, wherein the spatial structure information of the t frame image is retained in the high-dimensional features;

[0155] After performing a flattening operation on the high-dimensional features to obtain a one-dimensional vector, input the one-dimensional vector and the first hidden information into the LSTM network of the tracking model to obtain the predicted position coordinates of the t frame image and the second hidden information.

[0156] In one embodiment, the LSTM network has three gates, and the three gates are an input gate, a forget gate, and an output gate; wherein,

[0157] The prediction module 30 is further configured to:

[0158] Determine the input data through the input gate in the LSTM network, wherein the input data is determined from the one-dimensional vector and the first hidden information;

[0159] Remove the cleared data in the input data through the forget gate in the LSTM network to obtain the retained data;

[0160] Based on the retained data, obtain the predicted position coordinates of the t frame image and the second hidden information through the output gate in the LSTM.

[0161] In one embodiment, the prediction module 30 is further configured to:

[0162] Input the one-dimensional vector into the FC layer to obtain a correlation vector;

[0163] Input the one-dimensional vector and the first hidden information into the LSTM network of the tracking model to obtain the predicted position coordinates of the t-frame image and the second hidden information, including:

[0164] Input the correlation vector and the first hidden information into the LSTM network of the tracking model to obtain the predicted position coordinates of the t-frame image and the second hidden information.

[0165] In one embodiment, both the true position coordinates and the predicted position coordinates are two-dimensional coordinates; where

[0166] The prediction module 30 is further configured to:

[0167] Determine the true position abscissa and the true position ordinate according to the true position coordinates of the t-frame image;

[0168] Determine the predicted position abscissa and the predicted position ordinate according to the predicted position coordinates of the (t + 1)-frame image;

[0169] Determine the abscissa difference between the true position abscissa and the predicted position abscissa, and determine the ordinate difference between the true position ordinate and the predicted position ordinate;

[0170] Based on the square of the abscissa difference and the square of the ordinate difference, determine the prediction error.

[0171] It should be noted that the above-described workflow is merely illustrative and does not limit the protection scope of the present invention. In actual applications, those skilled in the art can select some or all of them according to actual needs to achieve the purpose of the solution of this embodiment, and no limitation is imposed here.

[0172] In addition, for the technical details not described in detail in this embodiment, reference can be made to the automatic tracking method of coronary angiography marker points provided in any embodiment of the present invention, which will not be elaborated here.

[0173] In addition, it should be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or system including that element.

[0174] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0175] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as a read-only memory (ROM) / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0176] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. An automatic tracking method for coronary angiography marking points, characterized in that The automatic tracking method for the coronary angiography marking points includes: S1. Obtain a target coronary angiography video, and obtain a marked frame image from the target coronary angiography video, wherein the position coordinates of the marking points are marked in the marked frame image; S2. Determine target implicit information according to the position coordinates of the marking points in the marked frame image; S3. Input the target implicit information and the marked frame image into a trained tracking model to obtain the predicted position coordinates of the marking points in the marked frame image and output layer information; S4. Determine whether the marked frame image is the last frame image in the target coronary angiography video; S5. If not, take the next frame image of the marked frame image as the new marked frame image, and take the output layer information as the new target implicit information to repeat steps S3 - S5 until the marked frame image is the last frame image in the target coronary angiography video; Before inputting the target implicit information and the marked frame image into the trained tracking model, it further includes: S31. Obtain a coronary angiography video, wherein the position coordinates of the marking points are marked in each frame image of the coronary angiography video; S32. Determine first implicit information according to the position coordinates of the t-th frame image; S33. Input the first implicit information and the t-th frame image in the coronary angiography video into the tracking model to obtain the predicted position coordinates of the t-th frame image and second implicit information; S34. Determine a prediction error based on the position coordinates and the predicted position coordinates of the t-th frame image; S35. Update the parameters in the tracking model through backpropagation based on the prediction error; S36. Take the (t + 1)-th frame image in the coronary angiography video as the new t-th frame image and take the second implicit information as the new first implicit information to repeat steps S33 - S36 until the prediction error is less than a preset value to obtain a trained tracking model; The determining the first implicit information according to the t-th frame image in the coronary angiography video includes: Performing dimensionality increase processing on the position coordinates of the t-th frame image through an MLP network to obtain the first implicit information; The tracking model includes a CNN network and an LSTM network; wherein, Inputting the first implicit information and the t-th frame image in the coronary angiography video into the tracking model to obtain the predicted position coordinates of the t-th frame image and second implicit information includes: Input the t-th frame image in the coronary angiography video into the CNN network of the tracking model to obtain the high-dimensional features of the t-th frame image, wherein the spatial structure information of the t-th frame image is retained in the high-dimensional features; After performing a flattening operation on the high-dimensional features to obtain a one-dimensional vector, input the one-dimensional vector and the first implicit information into the LSTM network of the tracking model to obtain the predicted position coordinates of the t-th frame image and second implicit information.

2. The method according to claim 1, characterized in that, The LSTM network has three gates, and the three gates are respectively an input gate, a forget gate, and an output gate; wherein, Inputting the one-dimensional vector and the first hidden information into the LSTM network of the tracking model to obtain the predicted position coordinates of the t-frame image and the second hidden information, including: Determining input data through the input gate in the LSTM network, where the input data is determined from the one-dimensional vector and the first hidden information; Removing the clearing data in the input data through the forget gate in the LSTM network to obtain the retained data; Based on the retained data, obtaining the predicted position coordinates of the t-frame image and the second hidden information through the output gate in the LSTM.

3. The method according to claim 1, characterized in that, Before inputting the one-dimensional vector and the first hidden information into the LSTM network of the tracking model, it further includes: Inputting the one-dimensional vector into the FC layer to obtain a correlation vector; Inputting the one-dimensional vector and the first hidden information into the LSTM network of the tracking model to obtain the predicted position coordinates of the t-frame image and the second hidden information, including: Inputting the correlation vector and the first hidden information into the LSTM network of the tracking model to obtain the predicted position coordinates of the t-frame image and the second hidden information.

4. The method according to claim 1, characterized in that Both the real position coordinates and the predicted position coordinates are two-dimensional coordinates; where Based on the real position coordinates and the predicted position coordinates of the t-frame image, determining the prediction error, including: Determining the real position abscissa and the real position ordinate according to the real position coordinates of the t-frame image; Determining the predicted position abscissa and the predicted position ordinate according to the predicted position coordinates of the t-frame image; Determining the abscissa difference between the real position abscissa and the predicted position abscissa, and determining the ordinate difference between the real position ordinate and the predicted position ordinate; Based on the square of the abscissa difference and the square of the ordinate difference, determining the prediction error.

5. An automatic tracking device for coronary angiography marking points, characterized in that The automatic tracking device for coronary angiography marker points includes: An acquisition module, configured to acquire a target coronary angiography video and acquire a marker frame image from the target coronary angiography video, where the position coordinates of the marker points are marked in the marker frame image; A determination module, configured to determine target hidden information according to the position coordinates of the marker points in the marker frame image; A prediction module, configured to input the target hidden information and the marker frame image into a trained tracking model to obtain the predicted position coordinates of the marker points in the marker frame image and the output layer information; A judgment module, configured to judge whether the marker frame image is the last frame image in the target coronary angiography video; The prediction module, when determining that the marker frame image is not the last frame image in the target coronary angiography video, takes the next frame image of the marker frame image as a new marker frame image, and takes the output layer information as new target hidden information to determine the predicted position coordinates of the new marker frame image; The prediction module is further configured to: Obtain a coronary angiography video, wherein the position coordinates of the marker points are marked in each frame image of the coronary angiography video; determine the first hidden information according to the position coordinates of the t-th frame image; input the first hidden information and the t-th frame image in the coronary angiography video into the tracking model to obtain the predicted position coordinates of the t-th frame image and the second hidden information; determine the prediction error based on the position coordinates and the predicted position coordinates of the t-th frame image; update the parameters in the tracking model by backpropagation based on the prediction error; use the (t + 1)-th frame image in the coronary angiography video as the new t-th frame image and the second hidden information as the new first hidden information to determine the trained tracking model; The prediction module is further configured to: Perform dimensionality elevation processing on the position coordinates of the t-th frame image through an MLP network to obtain the first hidden information; The prediction module is further configured to: The tracking model includes a CNN network and an LSTM network; Input the t-th frame image in the coronary angiography video into the CNN network of the tracking model to obtain the high-dimensional features of the t-th frame image, wherein the spatial structure information of the t-th frame image is retained in the high-dimensional features; after performing a flattening operation on the high-dimensional features to obtain a one-dimensional vector, input the one-dimensional vector and the first hidden information into the LSTM network of the tracking model to obtain the predicted position coordinates of the t-th frame image and the second hidden information.

6. An automatic tracking device for coronary angiography marker points, characterized in that, The device includes: a memory, a processor, and an automatic tracking program for coronary angiography marker points stored on the memory and executable on the processor, and the automatic tracking program for coronary angiography marker points is configured to implement the steps of the automatic tracking method for coronary angiography marker points as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, An automatic tracking program for coronary angiography marker points is stored on the storage medium, and when the automatic tracking program for coronary angiography marker points is executed by a processor, the steps of the automatic tracking method for coronary angiography marker points as described in any one of claims 1 to 4 are implemented.

Images