Automatic identification and general tracking method and system of support system

Abstract

The invention discloses an automatic identification and general tracking method and system for a stent system. The method comprises the following steps: S1, collecting stent operation video data from an operating room; s2, personnel with professional medical knowledge carry out key point marking on videos of the straight-tube-type stent, the forked-type stent and the stent conveying system; s3, extracting and connecting key points to generate an interventional instrument skeleton; s4, constructing a deep learning network model to perform training of support feature extraction; and S5, evaluating the network prediction accuracy by using a mean square error, and correcting the model. According to the invention, accurate real-time identification of the stent in the interventional image is realized through the deep learning algorithm, and the technical effects of real-time, accurate, efficient and adaptive are realized.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular, to an automatic recognition and general tracking method and system for a stent system. Background Art

[0002] In aortic repair surgery, physicians rely on fluoroscopic imaging equipment to determine whether the stent graft has reached the lesion site and is successfully released. The visualization technology for endovascular interventional surgery has become a future development trend. This technology uses computer vision tracking technology to obtain a large amount of instrument information from DSA to assist physicians in aortic treatment. In addition, this technology also provides a technical basis for vascular interventional assisted robotic surgery navigation and intelligent surgical training. Developing a detection and tracking technology for endovascular interventional surgery with visual assistance can improve surgical accuracy and autonomy. However, the current stent segmentation is severely affected in the case of image angle deviation and defocus.

[0003] Stent recognition based on X-ray images faces four main challenges: (1) The quality of X-ray images with low contrast is poor, making it difficult to distinguish between stent grafts and physiological structure backgrounds. (2) The reticular metal support position of the stent graft is one of the important pieces of information for judging the graft state, but it is very rare in X-ray images. The number of pixels of the stent graft is much smaller than that of the background, so it is difficult to judge the stent state. (3) In a real surgical scenario, physicians need to obtain timely feedback on the position of instruments from X-ray images and make decisions, so the visualization processing speed needs to keep up with the doctor's reaction speed, which has not been fully considered in current other studies. (4) Due to the inevitability of the movement of the DSA lens, there is controversy in judging the edge and distal positions, which are important information for the stent release state, and it is difficult to accurately determine the release position.

[0004] The invention patent with the publication number CN200510011546.8 relates to a method and system for vascular stent positioning and preoperative selection. Through steps such as a selection step, a stenosis data extraction step, a reference data extraction step, a comparison calculation step, and a vascular stent positioning step, the stent selection and positioning are intelligently realized. Although the invention has certain advantages in stent positioning and selection, there are still some disadvantages: The system complexity is high, including multiple steps and complex calculation processes, which require high professional knowledge of operators and computing power of equipment. For medical institutions with limited resources, it may be difficult to popularize and apply. In addition, the system relies on a large amount of stenosis and normal blood vessel data for comparison and calculation. If the data collection is incomplete or the quality is not high, it may affect the accuracy of the final positioning and selection. The implementation of the system depends on high-quality medical imaging equipment and data processing equipment, increasing medical costs and posing higher requirements for the maintenance and management of the equipment. Summary of the Invention

[0005] The object of the present invention is to provide a method and system for automatic recognition and general tracking of a stent system to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object of the invention, an aspect of the present invention provides a method for automatic recognition and general tracking of a stent system, including the following steps:

[0007] Step S1, collect real-time video data of actual stent implantation surgeries in the operating room environment and establish an original video data set;

[0008] Step S2, manually annotate the characteristic key points of straight stents, bifurcated stents, and stent delivery systems in the video data set by annotators with professional medical knowledge to obtain accurate key point annotation data;

[0009] Step S3, based on the annotated key point information, extract the stent structure feature points and connect them to generate the skeleton model of the interventional instrument to form an effective feature skeleton data set;

[0010] Step S4, construct a deep learning network model, use the feature skeleton data set as the training sample, perform feature learning and optimization of the model, and obtain a network model with the ability to automatically recognize and track stents;

[0011] Step S5, use the mean square error to evaluate the error between the predicted key points output by the network model and the actual annotated key points, and adjust and optimize the model parameters based on the evaluation results to improve the accuracy and robustness of the model.

[0012] Further, in step S1, collect surgical videos from different operating rooms, different devices, and different operators, ensure an appropriate data ratio for different types of surgeries, and use data augmentation techniques to generate diverse training samples. The data augmentation techniques include rotation, flipping, and scaling.

[0013] Further, for straight stents, form a skeleton by extracting key points. The key points are distributed along the length of the stent to form the basic stent morphology, and the stent structure is identified through these key points. The stent structure includes the inner diameter of the stent, the stent node structure, and the connection of the stent segments.

[0014] Further, for bifurcated stents, form a skeleton by extracting key points. The key points are distributed along the main body and the bifurcation of the stent to form a bifurcated skeleton structure. The stent structure includes the inner diameter of the stent, the stent nodes, and the branch points.

[0015] Further, for the stent delivery system, form a skeleton by extracting key points. The stent structure includes the introducer dilator tip and the stent hooks.

[0016] Furthermore, the processing process of the deep learning network model includes the following steps:

[0017] Step S401: Receive and preprocess the image data, and standardize the input image to ensure that the input data meets the requirements of the network.

[0018] Step S402: The image passes through an encoder (Encode) and a decoder (Decode) respectively for feature extraction.

[0019] Step S403: Generate the final segmentation result through the Softmax function.

[0020] Furthermore, in step S402, the encoder enhances features through the CSA (Channel-wise Spatial Attention) module, and the decoder ensures the consistency and accuracy of features through the CSA module.

[0021] Furthermore, step S5 uses the mean squared error to evaluate the gap between the predicted stent key point positions by the network and the actual labeled positions. During the training process, the model continuously adjusts the parameters through the backpropagation algorithm, and then uses the optimization algorithm to adjust the network weights according to the gradient information of the loss function, so that the prediction error gradually decreases.

[0022] Another aspect of the present invention provides an automatic recognition and general tracking system for a stent system, including an acquisition module, a labeling module, a skeleton generation module, a network model module, and a correction module, wherein:

[0023] The acquisition module is used to acquire real-time video data of the actual stent implantation surgery in the operating room environment and establish an original video data set.

[0024] The labeling module is used by professional medical annotators to manually label the key feature points of straight stents, bifurcated stents, and stent delivery systems respectively to obtain accurate key point labeling data.

[0025] The skeleton generation module is used to extract the stent structure feature points based on the labeled key point information and connect them to generate the skeleton model of the interventional instrument, forming an effective feature skeleton data set.

[0026] The network model module is used to construct a deep learning network model, use the feature skeleton data set as the training sample, perform feature learning and optimization of the model, and obtain a network model with the ability of automatic stent recognition and tracking.

[0027] The correction module is used to use the mean squared error to evaluate the error between the predicted key points output by the network model and the actual labeled key points, and adjust and optimize the model parameters based on the evaluation results to improve the accuracy and robustness of the model.

[0028] Due to the adoption of this system and method, compared with the prior art, it has the following advantages:

[0029] 1. In low-contrast X-ray images, the present invention can accurately distinguish stent grafts from the physiological structure background. By using a deep learning network to extract features and combining with a self-attention mechanism (CSA module), the model can effectively identify the position and morphology of stent grafts in complex backgrounds. Compared with traditional methods, the recognition accuracy of the present invention in low-quality images is greatly improved, solving the problem of difficult recognition in low-contrast images in the prior art.

[0030] 2. The present invention can accurately identify and track the positions of the reticulated metal supports of stent grafts. These positions are one of the important information for judging the state of the graft, but the number of pixels in X-ray images is much smaller than the background, making it difficult to identify. The present invention ensures the accurate representation of the geometric shape and key structures of stent grafts through fine key point annotation and skeleton generation, thereby achieving an accurate judgment of the stent state.

[0031] 3. The present invention takes into account the real-time requirements in actual surgical scenarios. By optimizing the structure and calculation process of the deep learning network, it ensures that the model can still maintain real-time performance in high-resolution and high-frame-rate image processing. The fast visualization processing speed enables physicians to obtain feedback on the instrument position from X-ray images in a timely manner and make surgical decisions quickly, significantly improving the efficiency and safety of surgery.

[0032] 4. The present invention also solves the problem of judging the stent release state caused by the movement of the DSA lens. Through accurate key point positioning and skeleton generation, the present invention can accurately determine the edges and distal positions of the stent, ensuring the correct positioning and deployment of the stent during the release process. This is of great significance for ensuring the success of the surgery and reducing postoperative complications.

[0033] 5. Through the collection and annotation of diverse and high-quality surgical video data, the model of the present invention has strong generalization ability and can adapt to different types of interventional surgeries and complex surgical environments. The application of data augmentation technology further improves the robustness of the model under different operating conditions, ensuring stable operation in a variety of surgical scenarios.

[0034] In summary, the present invention realizes accurate real-time recognition of stents in interventional images through deep learning algorithms, achieving the technical effects of real-time, accurate, efficient, and adaptive. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flowchart of an automatic recognition and general tracking method for a stent system.

[0036] Figure 2 It is a schematic diagram of a key point annotation method.

[0037] Figure 3 It is an automatic recognition and skeleton tracking diagram of a straight tube stent under interventional imaging.

[0038] Figure 4 It is an automatic recognition and skeleton tracking diagram of a branched stent under interventional imaging.

[0039] Figure 5 It is an automatic recognition and skeleton tracking diagram of a stent delivery system under interventional imaging. Specific implementation manners

[0040] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0041] As Figure 1 shown in the method flow chart of the present invention, the embodiments of the present invention provide a method for recognizing and tracking stent grafts and their delivery systems in X-ray images through deep learning algorithms. The specific steps are as follows:

[0042] Step S1, collect real-time video data of actual stent implantation surgeries in the operating room environment and establish an original video data set.

[0043] In the data collection stage, a large amount of real surgical video data needs to be obtained to support the training and optimization of the model. These videos should cover interventional instrument images from multiple angles to ensure that the model has good generalization ability. To ensure data diversity, it is recommended to collect surgical videos from different operating rooms, different devices, and surgeries performed by different operators. In addition, the collected video data needs to be standardized and accurately annotated to provide high-quality training samples for the model. During the data collection process, attention should also be paid to data diversity and balance to ensure that the data distribution of various surgeries is balanced and avoid model bias caused by a too high proportion of a certain type of surgical data. To further enhance the recognition effect, data augmentation techniques such as rotation, flipping, and scaling can be used to generate more diverse training samples, thereby improving the robustness and adaptability of the model in different scenarios. It is recommended to collect surgical data from at least 50 cases to construct a representative and diverse training data set to provide sufficient guarantee for subsequent model training.

[0044] Step S2, personnel with professional medical knowledge respectively perform key point annotation on videos of straight tube stents, bifurcated stents, and stent delivery systems.

[0045] These videos need to be marked with key points, and the marking should be carried out by personnel with professional medical knowledge to ensure the accuracy of the marking. The marking method is as Figure 2 shown. First, for the straight tube stent, key points are extracted to form a skeleton, and the structure of the stent is further identified. In the left figure, the key points are distributed along the length of the stent, forming the basic shape of the stent. These key points are connected to form a skeleton, reflecting the basic geometric shape of the stent. The right figure shows the stent structure identified through these key points, including the inner diameter of the stent, the stent node structure, and the connection of the stent segments. Second, for the bifurcated stent, the extraction of key points and the formation of the skeleton are similar to those of the straight tube stent, but the structure is more complex. In the left figure, the key points are distributed not only along the main stem of the stent but also at the bifurcation, forming a bifurcated skeleton structure. The middle figure shows the formed skeleton, accurately reflecting the shape of the bifurcated stent. The right figure further identifies the important features of the stent through these skeletons, such as the inner diameter of the stent, the stent nodes, and the branch points. Finally, for the stent delivery system, the skeleton marking method is also achieved by extracting key points. In the left figure, the key point marking of the delivery system is shown, including parts such as the introducer dilator and the stent hooks.

[0046] Step S3: Based on the marked key point information, extract the stent structure feature points and connect them to generate the skeleton model of the interventional device, forming an effective feature skeleton data set.

[0047] Due to the particularity of the stent graft, the positions of its key points are shown in Table 1:

[0048] Table 1

[0049] The connection order of the key points is shown in Table 2:

[0050] Table 2

[0051]

[0052] Step S4: Build a deep learning network model, use the feature skeleton data set as the training sample, perform feature learning and optimization of the model, and obtain a network model with the ability to automatically identify and track the stent.

[0053] Step S401: The input is the starting point of the whole system, mainly responsible for receiving and preprocessing the image data to ensure that the input data meets the requirements of the network. The input image is usually in RGB format and needs to be normalized to improve the stability of training and inference. The normalization steps may include scaling the image to 512x512 and normalizing it to the feature extraction network.

[0054] Step S402: The image passes through the encoder and decoder respectively for feature extraction.

[0055] The structure of the feature extraction network includes two main parts: an encoder and a decoder. In the encoder part, a sequence of X-ray fluoroscopy images is input. These images are subjected to feature extraction through a series of convolutional layers (conv 3x3), depthwise convolutional layers (depthwise conv 3x3), and pointwise convolutional layers (pointwise conv 1x1). The input images gradually reduce their spatial dimensions (1 / 2, 1 / 4, 1 / 8) through multiple convolutional layers, and each convolutional layer is processed by the corresponding CSA module to enhance the feature representation.

[0056] In the decoder part, the spatial dimensions are gradually restored through the upsampling layer (upsample), and the feature maps from the corresponding layers of the encoder are combined. Each layer in the decoder is also processed by the CSA module to ensure the consistency and accuracy of the features.

[0057] Among them, the CSA module transforms the number of input feature channels into c' through a 1x1 convolutional layer. Then, the input feature map is divided into two parts, which are processed through two 3x3 convolutional layers respectively, and after merging, it passes through a 3x3 convolutional layer again. These feature maps are then subjected to global pooling, and then processed through a 1x1 convolutional layer, Batch Normalization (BN), and the ReLU activation function.

[0058] In step S403, the processed feature map is further normalized through the Softmax function. Finally, the obtained attention map is subjected to element-wise multiplication and addition operations with the input feature map to generate the final output feature.

[0059] In step S5, the mean square error is used to evaluate the error between the predicted key points output by the network model and the actual annotated key points. Based on the evaluation results, the model parameters are adjusted and optimized to improve the accuracy and robustness of the model.

[0060] The mean square error is a commonly used loss function for measuring the difference between the model's predicted values and the true values, and is particularly suitable for regression tasks. In the present invention, the MSE can be used to evaluate the gap between the positions of the stent key points predicted by the network and the actual annotated positions. Its basic idea is to calculate the squares of the differences between all predicted values and the true values, and then take the average of these squared differences. During the training process, the model continuously adjusts the parameters through the backpropagation algorithm to minimize the mean square error loss function. Finally, an optimization algorithm (such as Adam or SGD) is used to adjust the network weights according to the gradient information of the loss function, so that the prediction error gradually decreases. Through repeated iterative training, the model parameters gradually converge, enabling the network to accurately predict the positions of the key points.

[0061] Such as Figure 3The schematic diagram shown is the automatic identification and skeleton tracking diagram of straight tube stent under interventional imaging. The accurate identification and real-time tracking of stent structure under complex background are directly related to the quality and success rate of interventional surgery. Traditional methods usually rely on the subjective judgment of doctors, which is easily affected by image clarity, stent complexity and interference from other interventional devices, and there is a large risk of error. Therefore, in a low-contrast, low signal-to-noise ratio imaging environment, how to achieve rapid and accurate positioning of the stent structure position and morphology has become a clinical technical problem that needs to be solved urgently. As shown in the figure, the key points of the stent automatically generated by the deep learning method are clearly marked as B1 or B2 serial numbers. From B1-1 to B1-4, the important positions of the top and bottom of the stent are marked respectively, and B2-1 to B2-14 mark the key positions on the main skeleton of the stent. The coordinates in the image are clearly marked next to each node. Among them, the yellow nodes clearly show the important positioning points on the stent skeleton, and the skeleton lines formed by connecting the key points in sequence intuitively and accurately show the basic morphology, inner diameter size, node position and connection relationship of each segment of the stent.

[0062] like Figure 4 The schematic diagram shown is the automatic identification and skeleton tracking diagram of branched stents under interventional imaging. The schematic diagram shown in the figure shows the results of automatic identification and skeleton tracking of branched stents under interventional imaging. In actual clinical surgery, branched stents have complex structures and multiple branch structures. Traditional manual identification and positioning methods are easily affected by image quality, background interference and instrument overlap, and accuracy is difficult to guarantee, making clinical application difficult. The figure shows the results of automatic identification and real-time tracking based on deep learning models. The key nodes of automatic identification are clearly marked in blue, from B1-1 to B1-6 represent the end positions of the stent trunk, from B2-1 to B2-21 represent the key positions of the intermediate structure of the stent trunk; B3-1 to B3-5 are the key nodes of the stent branch structure, which clearly reflect the unique spatial structural characteristics of the branch stent. The skeleton model formed by connecting the key nodes accurately and intuitively reflects the spatial layout, branch position and segment connection relationship of the stent, significantly improving the visual identification accuracy and real-time tracking ability of the branched stent under interventional imaging.

[0063] like Figure 5The schematic diagram shown is an automatic recognition and skeleton tracking diagram of a stent delivery system under interventional imaging. During clinical interventional procedures, the accurate determination of the position of the stent delivery system is directly related to the implantation effect and patient safety. However, limited by the ambiguity of the image, the differences in human anatomical structures, and the mutual interference of various interventional instruments within the surgical field of view, doctors often face problems of difficult positioning and recognition deviation when judging the path of the delivery system. Therefore, how to effectively improve the accuracy and stability of the path positioning of the stent delivery system and reduce the impact of human error on the surgical quality has become a technical problem that urgently needs to be broken through in clinical surgery. In the figure, the key nodes of the stent delivery system are automatically generated through a deep learning algorithm, and the nodes are marked with the serial number A. The important positions of the stent delivery system along the path are clearly marked from A1 to A10 in sequence. Among them, the red nodes and connecting lines clearly form the skeleton structure of the delivery system, intuitively reflecting the actual path and spatial orientation of the delivery instrument, and providing an effective basis for real-time positioning.

[0064] The present invention realizes the accurate recognition of the real-time recognition of the aortic stent in the interventional image through a deep learning algorithm, achieving the technical effects of real-time, accurate, efficient, and adaptive. Specifically as follows:

[0065] First of all, in the low-contrast X-ray image, the present invention can accurately distinguish the stent graft from the physiological structure background. By using a deep learning network to extract features and combining with a self-attention mechanism, the model can effectively identify the position and shape of the stent graft in a complex background. Compared with traditional methods, the recognition accuracy of the present invention in low-quality images is greatly improved, solving the problem of difficult recognition in low-contrast images in the prior art.

[0066] Secondly, the present invention can accurately identify and track the positions of the reticulated metal supports of the stent graft. These positions are one of the important information for judging the state of the graft, but the number of pixels in the X-ray image is much smaller than the background, making it difficult to recognize. The present invention ensures the accurate representation of the geometric shape and key structure of the stent graft through fine key point annotation and skeleton generation, thereby realizing the precise judgment of the stent state.

[0067] At the same time, the present invention also takes into account the real-time requirements in the actual surgical scenario. By optimizing the structure and calculation process of the deep learning network, it is ensured that the model can still maintain real-time performance in high-resolution and high-frame-rate image processing. The fast visualization processing speed enables the physician to obtain the feedback of the instrument position from the X-ray image in a timely manner and make surgical decisions quickly, significantly improving the efficiency and safety of the surgery.

[0068] The present invention also solves the problem of judging the release state of the stent caused by the movement of the DSA lens. Through accurate key point positioning and skeleton generation, the present invention can accurately determine the edge and distal position of the stent, ensuring the correct positioning and deployment of the stent during the release process. This is of great significance for ensuring the success of the surgery and reducing postoperative complications.

[0069] Through the collection and annotation of diverse and high-quality surgical video data, the model has strong generalization ability and can adapt to different types of interventional surgeries and complex surgical environments. The application of data augmentation technology further improves the robustness of the model under different operating conditions, ensuring stable operation in a variety of surgical scenarios.

[0070] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An automatic recognition and general tracking method for a stent system, characterized in that, It includes the following steps: Step S1, collect the real-time video data of the actual stent implantation surgery in the operating room environment, and establish an original video data set; Step S2, the key feature points of the straight stent, bifurcated stent and stent delivery system are manually labeled by annotators with professional medical knowledge to obtain accurate key point annotation data; Step S3, based on the annotated key point information, extract the stent structure feature points and connect them to generate the skeleton model of the interventional instrument, forming an effective feature skeleton data set; Step S4, construct a deep learning network model, use the feature skeleton data set as the training sample, conduct feature learning and optimization of the model, and obtain a network model with the ability of automatic stent recognition and tracking; Step S5, use the mean square error to evaluate the error between the predicted key points output by the network model and the actual annotated key points, and adjust and optimize the model parameters based on the evaluation results to improve the accuracy and robustness of the model.

2. The automatic recognition and general tracking method of a bracket system according to claim 1, characterized in that In step S1, collect the surgical videos from different operating rooms, different devices and different operators, ensure that the data ratio of different types of surgeries is appropriate, and use data augmentation techniques to generate diverse training samples. The data augmentation techniques include rotation, flipping, and scaling.

3. The automatic recognition and general tracking method of a bracket system according to claim 1, characterized in that, For the straight stent, the skeleton is formed by extracting key points, and the key points are distributed along the length of the stent, forming the basic stent shape, and the stent structure is identified through these key points. The stent structure includes the inner diameter of the stent, the stent node structure, and the connection of the stent segments.

4. The automatic recognition and general tracking method of a bracket system according to claim 1, characterized in that, For the bifurcated stent, the skeleton is formed by extracting key points, and the key points are distributed along the main body and the bifurcation of the stent, forming a bifurcated skeleton structure. The stent structure includes the inner diameter of the stent, the stent nodes, and the branch points.

5. The automatic recognition and general tracking method of a bracket system according to claim 1, characterized in that, For the stent delivery system, the skeleton is formed by extracting key points, and the stent structure includes the introducer dilator and the stent claws.

6. The automatic recognition and general tracking method of a bracket system according to claim 1, characterized in that The processing process of the deep learning network model includes the following steps: Step S401, receive and preprocess the image data, and perform standardization processing on the input image to ensure that the input data meets the requirements of the network; Step S402, the image passes through the encoder and decoder respectively for feature extraction; Step S403, generate the final segmentation result through the Softmax function.

7. The automatic identification and general tracking method of a bracket system according to claim 1, characterized in that, In step S402, the encoder enhances the features through the CSA module, and the decoder ensures the consistency and accuracy of the features through the CSA module.

8. The automatic recognition and general tracking method of a bracket system according to claim 1, characterized in that, Step S5 uses the mean square error to evaluate the gap between the predicted stent key point positions by the network and the actual annotated positions. During the training process, the model continuously adjusts the parameters through the backpropagation algorithm, and then uses the optimization algorithm to adjust the network weights according to the gradient information of the loss function, so that the prediction error gradually decreases.

9. An automatic recognition and general tracking system for a bracket system, characterized in that, It includes an acquisition module, an annotation module, a skeleton generation module, a network model module, and a correction module, where: The acquisition module is used to collect the real-time video data of the actual stent implantation surgery in the operating room environment and establish an original video data set; The annotation module is used by annotators with professional medical knowledge to manually annotate the key feature points of the straight stent, bifurcated stent and stent delivery system respectively to obtain accurate key point annotation data; The skeleton generation module is used to extract the feature points of the stent structure based on the labeled key point information, connect them to generate the skeleton model of the interventional device, and form an effective feature skeleton data set; The network model module is used to construct a deep learning network model, use the feature skeleton data set as the training sample, perform feature learning and optimization of the model, and obtain a network model with the ability to automatically identify and track stents; The correction module is used to evaluate the error between the predicted key points output by the network model and the actual labeled key points using the mean square error, and adjust and optimize the model parameters based on the evaluation results to improve the accuracy and robustness of the model.

Images