Stent implantation position identification method and device, electronic equipment and medium

Abstract

The invention provides a stent implantation position identification method and device, electronic equipment and a medium, and the method comprises the steps: obtaining a plurality of medical images during stent implantation operation, and marking the medical images to construct a data set; constructing a support position identification model, wherein the support position identification model comprises a double-branch backbone network, a Neck network and a detection identification module; training the support position identification model based on the data set to obtain a trained support position identification model; and acquiring a medical image during the current stent implantation operation in real time, inputting the medical image acquired in real time into the trained stent position identification model, and outputting an identification result. According to the invention, whether the stent of the patient is accurately implanted to the correct position or not in the stent implantation operation process can be detected in real time in the stent implantation operation process.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device, electronic equipment and medium for identifying a stent implantation position. Background Art

[0002] In interventional cardiovascular surgery, it is crucial for doctors to accurately implant stents. Inaccurate implantation of stents may lead to adverse events during surgery. There may be many reasons for adverse events during surgery, such as errors in visual perception, lack of surgical experience, occasional misjudgment, etc. With the continuous development of medical image intelligent recognition technology, the use of new technical means to avoid the occurrence of adverse events in surgery has been increasingly applied in clinical medicine.

[0003] Traditional image target detection methods use a sliding window framework, including steps such as selecting candidate regions, extracting visual features, and using classifiers for recognition. Classic algorithms, such as the multi-scale deformable component model, are all expanded on the basis of "gradient histogram + support vector machine", but their high computational complexity and slow detection speed cannot meet the requirements of fast detection applications with real-time requirements. New intelligent image processing technology based on deep learning improves the accuracy of target detection algorithms.

[0004] Object detection algorithms based on deep learning can be divided into two categories: two-stage detection and one-stage detection. The two-stage detection algorithm is a "coarse-to-fine" process. First, the region candidate box is generated, then the features of each candidate box are extracted, and finally the position box is generated and the corresponding category is predicted. It is characterized by high accuracy but slow speed. The one-stage detection algorithm is a "one-step" process. The one-stage detection network performs classification and bounding box regression while generating candidate boxes. It is characterized by fast speed but slightly lower accuracy. The problems faced by these two types of algorithms are: either the detection accuracy is high, but the real-time detection is limited; or the real-time detection is good, but the detection accuracy is limited. At present, the application research of machine learning technology in the field of aortic stent implantation surgery mainly involves the following aspects:

[0005] Image segmentation and recognition: Machine learning is widely used in static image segmentation and recognition tasks of aortic stents. By training deep learning models, automatic recognition and segmentation of aortic stents in medical images can be achieved, helping doctors determine the implantation location and placement of the stent.

[0006] Lesion detection and diagnostic assistance: Machine learning technology can be used to detect and analyze vascular lesions around aortic stents in combination with medical examination images (CT and MRI), assisting doctors in making diagnosis and treatment decisions. These lesions may include aortic aneurysms, vascular stenosis, etc. Machine learning can help improve diagnostic accuracy and efficiency.

[0007] Surgery planning and assistance: Before aortic stent surgery, machine learning can be used to assist in surgical planning. By analyzing the patient's preoperative medical examination image data and condition information, machine learning algorithms can provide doctors with personalized surgical plans and optimize surgical results.

[0008] Prognosis prediction and risk assessment: Machine learning can also be used to predict patient prognosis monitoring after aortic stent surgery and assess surgical risks. By analyzing the patient's clinical data and image features of medical examinations, machine learning models can help doctors better understand the patient's disease progression and treatment effects.

[0009] Although artificial intelligence has been used in the field of vascular surgery for several purposes, such as image segmentation, or the development of predictive risk stratification models, its application in real-time detection of aortic stent implantation position during surgery remains rare. Summary of the invention

[0010] The present invention provides a method, device, electronic equipment and medium for identifying a stent implantation position, which are used for real-time detection of whether a patient's stent is accurately implanted in a correct position during a stent implantation operation.

[0011] According to one aspect of the present invention, a method for identifying a stent implantation position is provided, comprising:

[0012] Obtain multiple medical images during stent implantation operations and annotate the medical images to construct a data set;

[0013] Constructing a stent position recognition model, wherein the stent position recognition model includes a dual-branch backbone network, a Neck network, and a detection and recognition module;

[0014] Training the stent position recognition model based on the data set to obtain a trained stent position recognition model;

[0015] The medical image during the current stent implantation operation is acquired in real time, the medical image acquired in real time is input into the trained stent position recognition model, and the recognition result is output.

[0016] Optionally, the stent position recognition model is a YOLOSDB model; the dual-branch backbone network includes an edge branch and a main branch;

[0017] The edge branch is used to extract edge features of the medical image, and the main branch is used to extract main features of the medical image.

[0018] Optionally, the main branch includes a first downsampling module, a first C2 module and a plurality of first block modules;

[0019] The first downsampling module is used to perform downsampling processing on the input medical image and output a sampling feature map;

[0020] The first C2 module is used to extract the features of the sampling feature map to obtain a first main feature map, and transmit the first main feature map to the first block module;

[0021] The multiple first block modules are used to extract features of the first main feature map layer by layer, compress the extracted feature map, and output the main feature map of the corresponding number of layers; the multiple first block modules respectively output the main feature map of the corresponding number of layers.

[0022] Optionally, the edge branch includes an operator extraction module, a second downsampling module, a second C2 module and a plurality of third C2 modules;

[0023] The operator extraction module is used to extract edge information of medical images;

[0024] The second downsampling module is used to perform downsampling processing on the medical image after edge information is extracted to obtain a downsampled edge feature map;

[0025] The second C2 module is used to extract features from the input edge feature map and the first main feature map, and fuse the extracted edge feature map and the main feature map to obtain a first fused feature;

[0026] The multiple third C2 modules extract features from the input first fused feature map and the main feature map input from the first block module at the same layer as the third C2 module, and further fuse the first fused feature map after feature extraction with the main feature to obtain a second fused feature map.

[0027] Optionally, the Neck network includes a second block module, a plurality of third block modules, a fourth C2 module, a fifth C2 module, an spp module and a feature fusion module;

[0028] The spp module is used to receive the main feature map output by the first block module of the last layer, and output a feature map of a fixed size to the second block module;

[0029] The second block module is used to receive the main feature map output by the first block module at the same layer as the first block module and the feature map output by the spp module, and compress the input feature map to output a feature map of the target number of channels;

[0030] The third block module is used to receive the main feature map output by the first block module at the same layer as the third block module and the feature map of the second block module, and compress the input feature map to output a feature map of the target number of channels;

[0031] The fourth C2 module is used to receive the second fused feature map output by the third C2 module of the last layer and the second fused feature map output by the third C2 module of the same layer as the fourth C2 module to perform feature extraction, and further fuse the two second fused feature maps after feature extraction to obtain a third fused feature map;

[0032] The fifth C2 module is used to receive the third fused feature map output by the fourth C2 module of the last layer and the second fused feature map output by the third C2 module of the same layer as the fifth C2 module to perform feature extraction, and further fuse the second fused feature map after feature extraction and the third fused feature map to obtain a fourth fused feature map;

[0033] The feature fusion module is used to perform feature fusion on the feature map output by the spp module and the second fused feature map output by the third C2 module of the last layer, to perform feature fusion on the third fused feature map output by the fourth C2 module and the feature map output by the second block module, to perform feature fusion on the fourth fused feature map output by the fifth C2 module and the feature map output by the third block module, and to output fused feature maps of corresponding layers respectively.

[0034] Optionally, the C2 module includes a 1X1 convolution layer, a split layer, a 3X3 convolution layer and a connection layer;

[0035] The 1×1 convolution layer is used to perform convolution processing on the input feature map;

[0036] The split layer is used to split the feature map after convolution processing into multiple groups according to channels;

[0037] Each group is sequentially entered into the 1×1 convolution layer and the 3×3 convolution layer for convolution processing, and the feature map after the convolution processing of the 3×3 convolution layer is fused with the feature map after segmentation to obtain a fused feature map;

[0038] The connection layer is used to further fuse the feature map of each group output by the split layer and the fused feature map.

[0039] Optionally, the block module includes a first convBNSiLU module, a second convBNSiLU module, a Split module, a plurality of BottleNeck modules and a connection layer;

[0040] The first convBNSiLU module is used to perform a convolution on the input feature map to double the number of channels of the input;

[0041] The Split module is used to split the feature map after convolution processing into multiple groups according to channels;

[0042] The feature maps of the multiple groups are sequentially passed through the multiple BottleNeck modules to extract features layer by layer; the BottleNeck module includes multiple convolutional layers;

[0043] The connection layer is used to splice the feature map output by each BottleNeck module with the original feature map;

[0044] The second convBNSiLU module is used to compress the concatenated feature map and output a feature map with a target number of channels.

[0045] According to another aspect of the present invention, there is provided a stent implantation position identification device, comprising:

[0046] A data set acquisition unit, used for acquiring medical images during multiple stent implantation operations and annotating the medical images to construct a data set;

[0047] A model building unit, used to build a stent position recognition model, wherein the stent position recognition model includes a dual-branch backbone network, a Neck network, and a detection and recognition module;

[0048] A model training unit, used for training the stent position recognition model based on the data set to obtain the trained stent position recognition model;

[0049] The recognition unit is used to obtain the medical image during the current stent implantation operation in real time, input the medical image obtained in real time into the trained stent position recognition model, and output the recognition result.

[0050] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0051] at least one processor; and

[0052] a memory communicatively connected to the at least one processor; wherein,

[0053] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the stent implantation position identification method described in any embodiment of the present invention.

[0054] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the stent implantation position identification method described in any embodiment of the present invention when executed.

[0055] The technical solution of the embodiment of the present invention uses medical images during stent implantation surgery to train a machine learning model to learn the image features when the stent is correctly implanted, thereby quickly and accurately identifying whether the stent in the medical image is correctly implanted; since a dual-branch backbone network and a Neck network are used in the neural network model, the model can obtain richer and more accurate feature expressions, thereby increasing the accuracy of model recognition. The model is also more suitable for complex medical image detection, so that the machine learning model after training can be applied in real-time surgical processes to quickly and accurately help doctors determine whether the stent is accurately implanted in the patient's aorta.

[0056] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 is a flow chart of a method for identifying a stent implantation position provided according to Embodiment 1 of the present invention;

[0059] Figure 2 The overall architecture diagram of the deep network used in the bracket position recognition model in the embodiment of the present invention;

[0060] Figure 3 is an architecture diagram of a dual-branch backbone network used in an embodiment of the present invention;

[0061] Figure 4 is an architecture diagram of the Neck network used in the embodiment of the present invention;

[0062] Figure 5is an architecture diagram of the C2 module used in the embodiment of the present invention;

[0063] Figure 6 is an architectural diagram of a block module used in an embodiment of the present invention;

[0064] Figure 7 Schematic diagram of edge extraction effect in an embodiment of the present invention;

[0065] Figure 8 It is a schematic diagram of the actual detection effect of the bracket position recognition model in the embodiment of the present invention;

[0066] Fig. 9 A schematic diagram of the structure of a stent implantation position identification device provided in Embodiment 2 of the present invention;

[0067] Fig.10 It is a structural schematic diagram of an electronic device for implementing a method for identifying a stent implantation position according to an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0069] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0070] In interventional cardiovascular surgery, it is crucial for doctors to accurately implant stents. Inaccurate implantation of stents may lead to adverse events during surgery. There may be many reasons for adverse events during surgery, such as errors in visual perception, lack of surgical experience, occasional misjudgment, etc. With the continuous development of medical image intelligent recognition technology, the use of new technical means to avoid the occurrence of adverse events in surgery has been increasingly applied in clinical medicine.

[0071] Embodiment 1

[0072] Figure 1 A flowchart of a method for identifying a stent implantation position is provided for the first embodiment of the present invention. Figure 1 As shown, the method includes:

[0073] S101. Acquire multiple medical images during stent implantation operations, and annotate the medical images to construct a data set.

[0074] Medical images are images that reflect the internal structure of the human body. In this embodiment, they are specifically medical images of the aorta near the heart collected during the stent implantation surgery. This type of image is used to train the machine learning model to determine whether the stent is accurately implanted.

[0075] In this embodiment, medical images can be annotated, for example, they can be annotated as images of accurate implantation or images of inaccurate implantation to guide the training of machine learning models. A sample can be formed by the medical image and the annotated result, thereby constructing a data set including multiple medical images.

[0076] S102, constructing a stent position recognition model, wherein the stent position recognition model includes a dual-branch backbone network, a Neck network, and a detection and recognition module.

[0077] In this embodiment, a deep network based on machine learning can be used to construct a stent position recognition model. Specifically, the deep network can use a dual-branch backbone network, a Neck network, and a detection and recognition module. The dual-branch backbone network can collect at least two feature information of the medical image, the Neck network is used to splice and fuse at least two feature information at multiple scales to fully obtain the features of the medical image, and the detection and recognition module is used to output the detection results.

[0078] S103: training the stent position recognition model based on the data set to obtain the trained stent position recognition model.

[0079] The medical images in the data set are input into the constructed stent position recognition model for training. After the training is completed, a trained stent position recognition model is obtained. In this embodiment, the data set can be divided into a training set and a test set according to a certain ratio for training the stent position recognition model.

[0080] S104, acquiring in real time the medical image during the current stent implantation operation, inputting the medical image acquired in real time into the trained stent position recognition model, and outputting the recognition result.

[0081] In actual application, the patient's medical images can be collected in real time during the stent implantation operation and input into the stent position recognition model to output the patient's current stent implantation status in real time, thereby assisting doctors in determining whether the current stent is accurately implanted in the patient's aorta.

[0082] The technical solution of the first embodiment of the present invention uses medical images during stent implantation surgery to train a machine learning model to learn the image features when the stent is correctly implanted, so as to quickly and accurately identify whether the stent in the medical image is correctly implanted; because a double-branch backbone network and a Neck network are used in the neural network model, the model can obtain richer and more accurate feature expressions, thereby increasing the accuracy of model recognition. The model is also more suitable for complex medical image detection, so that the machine learning model after training can be applied in real-time surgical processes to quickly and accurately help doctors determine whether the stent is accurately implanted in the patient's aorta.

[0083] In one embodiment, the stent position recognition model is a YOLOSDB model; the dual-branch backbone network includes an edge branch and a main branch; the edge branch is used to extract edge features of the medical image, and the main branch is used to extract main features of the medical image.

[0084] like Figure 2 The overall architecture diagram of the deep network used in the present invention is shown in FIG. Figure 2 In the figure, the dual-branch backbone network, including the main branch and the edge branch, finally fuses the multi-scale feature maps and outputs them for detection in the detection head.

[0085] Specifically, the main branch of the dual-branch backbone network is used to extract the main features of the medical image, and the edge branch is used to extract the edge feature information of the medical image. For example, the edge branch can process the edge information extraction part of the input image through the sobel operator to obtain the edge information features of batch*l*w*h. After that, the edge feature information is downsampled to 1 / 4 using stem, and parallel calculation with the main branch begins. The main branch and the edge branch can learn different levels of feature representation respectively. By fusing or jointly processing the features of the two branches, richer and more accurate feature expressions can be obtained. This structure helps to improve the generalization and processing capabilities of the model.

[0086] In one embodiment, the main branch includes a first downsampling module, a first C2 module and multiple first block modules; the first downsampling module is used to downsample the input medical image and output a sampling feature map; the first C2 module is used to extract the features of the sampling feature map to obtain a first main feature map, and transmit the first main feature map to the first block module; multiple first block modules are used to extract the features of the first main feature map layer by layer, compress the extracted feature map, and output the main feature map of the corresponding number of layers; multiple first block modules respectively output the main feature maps of the corresponding number of layers.

[0087] like Figure 3 As shown, Figure 3 The stem module in the main branch shown on the right is the first downsampling module, the C2 module in the main branch is the first C2 module, and the main branch also includes multiple first block modules. When a medical image sample is input, the medical image entering the main branch first enters the stem module for 1 / 4 downsampling to obtain a sampling feature map, and then the sampling feature map enters the first C2 module. The first C2 module is used to extract the features of the sampling feature map. Multiple first block modules are used to extract the features of the first main feature map layer by layer, and compress the extracted feature map to output the main feature map of the corresponding number of layers. For example, the feature map output by the first C2 module outputs a 1 / 8 feature map after passing through the first first block module, and the 1 / 8 feature map outputs a 1 / 16 feature map after passing through the second first block module, and the 1 / 16 feature map outputs a 1 / 32 feature map after passing through the third first block module, and so on. In this embodiment, the number of first block modules can be set as needed, for example Figure 3 A main branch is provided which includes three first block module layers connected in sequence.

[0088] In one embodiment, the edge branch includes an operator extraction module, a second downsampling module, a second C2 module and multiple third C2 modules; the operator extraction module is used to extract edge information of the medical image; the second downsampling module is used to downsample the medical image after edge information extraction to obtain a downsampled edge feature map; the second C2 module is used to extract features from the input edge feature map and the first main feature map, and fuse the extracted edge feature map and the main feature map to obtain a first fused feature; the multiple third C2 modules extract features from the input first fused feature map and the main feature map input by the first block module at the same layer as the third C2 module, and further fuse the first fused feature map after feature extraction with the main feature to obtain a second fused feature map.

[0089] like Figure 3 As shown, Figure 3 The sobel module in the edge branch shown on the left is an operator extraction module, the stem module in the edge branch is the second downsampling module, the second C2 module is connected below the stem module, and the feature map output by the second C2 module enters multiple third C2 modules in sequence.

[0090] Specifically, when a medical image sample is input, the medical image entering the edge branch first enters the Sobel module, that is, the operator extraction module uses the Sobel operator to extract features from the edge information of the medical image, and the feature map after edge information extraction will enter the stem module for 1 / 4 downsampling to obtain a sampled feature map, and then the sampled feature map enters the second C2 module, and the second C2 module is used to extract features from the input edge feature map and the first main feature map output by the first C2 module in the main branch, and fuse the extracted edge feature map and the main feature map to obtain a first fused feature; multiple third C2 modules extract features from the input first fused feature map and the main feature map input by the first block module at the same layer as the third C2 module, and further fuse the first fused feature map after feature extraction with the main feature to obtain a second fused feature map. It should be noted that the first fused feature and the main feature map input from the first block module at the same layer as the third C2 module enter the first third C2 module for feature extraction, and the first fused feature map after feature extraction is further fused with the main feature to obtain a fused feature map; the fused feature map output by the first third C2 module and the main feature map input from the first block module at the same layer as the second third C2 module are Figure 1The second third C2 module extracts the features of the two input feature maps and further fuses the fused feature map after the extracted features with the main feature to obtain a fused feature map; the fused feature map output by the second third C2 module and the main feature map input by the first block module at the same layer as the third third C2 module are Figure 1 The two feature maps are input into the third C2 module, and the third C2 module extracts features from the two input feature maps, and further fuses the fused feature map after the extracted features with the main features to obtain a fused feature map.

[0091] In this embodiment, at the same scale, each time before passing through the third C2 module, feature fusion is performed with the feature information processed by the main branch, which is equivalent to adding additional edge information to the original feature information. The network combines the original feature information with the edge information to learn features at the same time, which is more suitable for medical image detection. C2 is used to reduce the amount of calculation. Simply adding edge information does not require too many parameters of the model to learn features.

[0092] In one embodiment, the Neck network includes a second block module, a plurality of third block modules, a fourth C2 module, a fifth C2 module, an spp module, and a feature fusion module;

[0093] The spp module is used to receive the main feature map output by the first block module of the last layer, and output a feature map of a fixed size to the second block module;

[0094] The second block module is used to receive the main feature map output by the first block module at the same layer as the first block module and the feature map output by the spp module, and compress the input feature map to output a feature map of the target number of channels;

[0095] The third block module is used to receive the main feature map output by the first block module at the same layer as the third block module and the feature map of the second block module, and compress the input feature map to output a feature map of the target number of channels;

[0096] The fourth C2 module is used to receive the second fused feature map output by the third C2 module of the last layer and the second fused feature map output by the third C2 module of the same layer as the fourth C2 module to perform feature extraction, and further fuse the two second fused feature maps after feature extraction to obtain a third fused feature map;

[0097] The fifth C2 module is used to receive the third fused feature map output by the fourth C2 module of the last layer and the second fused feature map output by the third C2 module of the same layer as the fifth C2 module to perform feature extraction, and further fuse the second fused feature map after feature extraction and the third fused feature map to obtain a fourth fused feature map;

[0098] The feature fusion module is used to perform feature fusion on the feature map output by the spp module and the second fused feature map output by the third C2 module of the last layer, to perform feature fusion on the third fused feature map output by the fourth C2 module and the feature map output by the second block module, to perform feature fusion on the fourth fused feature map output by the fifth C2 module and the feature map output by the third block module, and to output fused feature maps of corresponding layers respectively.

[0099] It should be noted that the Neck network is a bidirectional feedback feature fusion network. By enhancing the fusion effect of multi-scale features, this deep network improves the model's detection ability for objects of different sizes. The bidirectional feedback feature fusion network supports bidirectional flow of feature information between multiple scales (top-down and bottom-up). The bidirectional feedback feature fusion network adaptively adjusts the importance of features of different scales in the fusion process by introducing learnable weights, so as to flexibly capture key features. In the bidirectional feedback feature fusion network, high-level feature maps contain rich semantic information but lack details. The top-down path gradually transfers these high-level features to low-level feature maps to enhance the semantic information of low-level feature maps. Low-level feature maps contain rich detail information but lack semantics. The bottom-up path transfers detail information to high-level feature maps to enrich their details. At each intersection of top-down and bottom-up, the bidirectional feedback feature fusion network fuses features from different paths by weighted summation, where the weights are obtained through learning, thereby achieving adaptive feature fusion.

[0100] Specifically, Figure 4 The Neck network architecture shown in FIG. 1 includes a second block module, multiple third block modules, a fourth C2 module, a fifth C2 module, an spp module, and a feature fusion module. Figure 4The embodiment of the invention shows an output network of three scales, namely, it includes three first block modules, and the three first block modules correspond to a third block module, a second block module and a spp module respectively. The spp module is used to receive the 1 / 32 main feature map output by the first block module of the last layer, and output a feature map of a fixed size to the second block module; the second block module is used to receive the 1 / 16 main feature map output by the first block module of the same layer as the first block module and the feature map output by the spp module, and compress the input feature map to output a feature map of the target number of channels; the third block module is used to receive the 1 / 8 main feature map output by the first block module of the same layer as the third block module and the feature map of the second block module, and compress the input feature map to output a feature map of the target number of channels.

[0101] In addition, the fourth C2 module is used to receive the second fused feature map output by the third C2 module of the last layer and the second fused feature map output by the third C2 module of the same layer as the fourth C2 module for feature extraction, and further fuse the two second fused feature maps after feature extraction to obtain a third fused feature map; the fifth C2 module is used to receive the third fused feature map output by the fourth C2 module of the last layer and the second fused feature map output by the third C2 module of the same layer as the fifth C2 module for feature extraction, and further fuse the second fused feature map after feature extraction and the third fused feature map to obtain a fourth fused feature map; the feature fusion module is used to perform feature fusion on the feature map output by the spp module and the second fused feature map output by the third C2 module of the last layer, perform feature fusion on the third fused feature map output by the fourth C2 module and the feature map output by the second block module, perform feature fusion on the fourth fused feature map output by the fifth C2 module and the feature map output by the third block module, and output fused feature maps of corresponding layers respectively. Among them, the spp (spatial pyramid pooling layer, Spatial Pyramid Pooling) module is used in the convolutional neural network (CNN) to process input images of different sizes and proportions. It aims to solve the problem that traditional CNN requires a fixed input image size, while improving the robustness and generalization ability of the model.

[0102] In one embodiment, the C2 module includes a 1X1 convolution layer, a split layer, a 3X3 convolution layer and a connection layer;

[0103] The 1×1 convolution layer is used to perform convolution processing on the input feature map;

[0104] The split layer is used to split the feature map after convolution processing into multiple groups according to channels;

[0105] Each group is sequentially entered into the 1×1 convolution layer and the 3×3 convolution layer for convolution processing, and the feature map after the convolution processing of the 3×3 convolution layer is fused with the feature map after segmentation to obtain a fused feature map;

[0106] The connection layer is used to further fuse the feature map of each group output by the split layer and the fused feature map.

[0107] The structure of the Neck network in the YOLOSDB model used in this embodiment is mainly composed of the C2 module and the C2f (block) module. After the dual-branch network, the edge branch and the three-scale output of the main branch are added for the final fusion, and then input into the detection head through the fourth block.

[0108] like Figure 5 The structural diagram of the C2 module shown in the figure, the C2 module includes a 1X1 convolution layer, a split layer, a 3X3 convolution layer and a connection layer. The input feature map is used to convolve the input feature map through the 1X1 convolution layer; then the split layer is used to divide the convolution-processed feature map into multiple groups according to the channel; each group enters the 1X1 convolution layer and the 3X3 convolution layer in turn for convolution processing, and the feature map after the convolution processing of the 3X3 convolution layer is fused with the segmented feature map to obtain a fused feature map; the connection layer is used to further fuse the feature map of each group output by the split layer and the fused feature map. In this embodiment, since the 1x1 convolution can be used to reduce or increase the number of channels of the feature map, by using the 1X1 convolution before the large convolution kernel operation, the number of parameters and the amount of calculation can be effectively reduced without sacrificing too much performance. Since the 1x1 convolution operation itself is linear, this embodiment can add a nonlinear activation function (such as ReLU) after the 1x1 convolution layer to introduce nonlinear characteristics, thereby increasing the expression ability of the network and enabling the network to learn more complex feature representations.

[0109] In addition, in this embodiment, the C2 module uses a split operation to divide the input feature map into multiple smaller groups according to the channel, and then perform convolution processing on each group independently. By grouping the channels of the input feature map and performing convolution operations on each group independently, the number of network parameters and computational costs can be significantly reduced. Grouped convolution can enable the network to learn a more sparse feature representation across channels, which can enhance the generalization ability of the model.

[0110] In one embodiment, the block module includes a first convBNSiLU module, a second convBNSiLU module, a Split module, a plurality of BottleNeck modules and a connection layer;

[0111] The first convBNSiLU module is used to perform a convolution on the input feature map to double the number of channels of the input;

[0112] The Split module is used to split the feature map after convolution processing into multiple groups according to channels;

[0113] The feature maps of the multiple groups are sequentially passed through the multiple BottleNeck modules to extract features layer by layer; the BottleNeck module includes multiple convolutional layers;

[0114] The connection layer is used to splice the feature map output by each BottleNeck module with the original feature map;

[0115] The second convBNSiLU module is used to compress the concatenated feature map and output a feature map with a target number of channels.

[0116] like Figure 6 As shown in the block module architecture diagram, the block module includes a first convBNSiLU module, a second convBNSiLU module, a Split module, multiple BottleNeck modules and a connection layer; the Split module is used to divide the feature map after convolution processing into multiple groups according to channels; the feature maps of multiple groups pass through multiple BottleNeck modules in turn to extract features layer by layer; the connection layer then uses the feature map output by each BottleNeck module to splice the original feature map; and the spliced ​​feature map is compressed in the second convBNSiLU module to output a feature map of the target number of channels.

[0117] In this embodiment, the method for evaluating the indicators of the bracket position recognition model includes: mAP, Recall and FPS indicators. Among them, mAP (mean Average Precision) is a commonly used evaluation indicator used to measure the accuracy of the model when detecting objects of different categories. AP is the average of the average precision (AP) of all categories. In order to understand the calculation of mAP, you first need to understand the concepts of precision (Precision) and recall (Recall), as well as how to calculate the AP of a single category.

[0118] Precision and Recall Precision: The proportion of samples that actually belong to a certain class among all samples identified by the model. The formula is:

[0119] Precision=TP / (TP+EP)

[0120] Among them, TP (True Positives) is the number of positive samples correctly identified, and FP (False Positives) is the number of positive samples incorrectly identified.

[0121] Recall rate (Reca11): The proportion of samples correctly identified by the model among all samples that actually belong to a certain class. The formula is:

[0122] Recall=TP / (TP+FN)

[0123] Among them, FN (False Negatives) is the number of positive samples not recognized by the model. For a single category, the detection results are first sorted according to the confidence of the model output. The precision and recall of each detection result are calculated, and the precision-recall curve (PR curve) is drawn. AP is the area under the curve (AUC), which is usually calculated using the numerical integration method.

[0124] The calculation of mAP involves the AP values ​​of all categories. The specific steps are as follows:

[0125] (1) Calculate the AP for each category: Follow the above steps to calculate the AP for each category separately.

[0126] (2) Calculate mAP: Calculate the average AP value of all categories. If there are N categories, the common value of mAP is

[0127] The formula can be expressed as:

[0128]

[0129] In some cases, mAP may be calculated for different intersection over union thresholds. For example, the C0C0 dataset usually reports the average value of mAP in the range of IoU = 0.50:0.05:0.95 (i.e., IoU from 0.50 to 0.95, with a step size of 0.05), referred to as mAP@.50:.05:.95.

[0130] IoU is used to measure the overlap between the predicted bounding box and the true bounding box. The calculation formula of IoU is the area of ​​the intersection of the predicted box and the true box divided by the area of ​​their union. In practical applications, there may be different evaluation indicators and calculation methods depending on the requirements of specific tasks and the characteristics of the dataset. For example, some tasks or competitions may require mAP at a specific IoU threshold, or use a weighted average method to calculate mAP.

[0131] During the experiment, the Sobel operator was finally selected for edge extraction in the bracket position recognition model. The effect of edge extraction is shown in the attached figure. Figure 7 shown.

[0132] We conducted comparative experiments on different edge detection algorithms, comparative experiments on the effects of convolution kernels of different sizes, and ablation experiments. The experimental results are shown in Tables 1, 2, and 3. The actual detection effect diagram is shown in the attached figure. Figure 8 shown.

[0133] Table 1 Evaluation indicators of different edge detection algorithms

[0134]

[0135] Table 2 Effects of convolution kernels of different sizes

[0136]

[0137] Table 3 Results of ablation experiments

[0138]

[0139] Experimental data show that the proposed method achieves an overall detection accuracy of 79.00% on the thoracic and abdominal aorta surgery image dataset, and the FPS reaches 77.8.

[0140] The technical solution of the embodiment of the present invention is based on the on-site images during the thoracic aortic and abdominal aortic stent implantation surgery, and aims at the real-time detection problem of the stent implantation position. An aortic stent detection method based on the YOLOSDB (You Only Look Once Sobel Double Branch) model is proposed. The feature extraction method of the improved double-branch parallel backbone network is adopted, and the network features are extracted using the double-branch edge detection network; the feature fusion method of the bidirectional feedback network is used, and the improved Neck network stacking is used to extract the fused multi-scale spatial features; finally, the installation restriction detection of the thoracic and abdominal aortic stent implantation position is performed.

[0141] Embodiment 2

[0142] Fig. 9 This is a schematic diagram of the structure of a stent implantation position identification device provided in Embodiment 3 of the present invention. Fig. 9 As shown, the device comprises:

[0143] The data set acquisition unit 901 is used to acquire multiple medical images during stent implantation operations and annotate the medical images to construct a data set;

[0144] A model building unit 902 is used to build a stent position recognition model, wherein the stent position recognition model includes a dual-branch backbone network, a Neck network, and a detection and recognition module;

[0145] A model training unit 903, configured to train the stent position recognition model based on the data set to obtain a trained stent position recognition model;

[0146] The recognition unit 904 is used to obtain the medical image during the current stent implantation operation in real time, input the medical image obtained in real time into the trained stent position recognition model, and output the recognition result.

[0147] The message data processing resource scheduling device provided in the embodiment of the present invention can execute the message data processing resource scheduling device method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0148] Embodiment 4

[0149] Fig.10 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0150] like Fig.10 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0151] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0152] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a method for identifying a stent implantation position.

[0153] In some embodiments, a stent implantation location identification method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the stent implantation location identification method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform a stent implantation location identification method in any other appropriate manner (e.g., by means of firmware).

[0154] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0156] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0157] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0158] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0159] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0160] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0161] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying a stent implantation position, characterized in that: include: Obtain multiple medical images during stent implantation operations and annotate the medical images to construct a data set; Constructing a stent position recognition model, wherein the stent position recognition model includes a dual-branch backbone network, a Neck network, and a detection and recognition module; Training the stent position recognition model based on the data set to obtain a trained stent position recognition model; The medical image during the current stent implantation operation is acquired in real time, the medical image acquired in real time is input into the trained stent position recognition model, and the recognition result is output.

2. The method for identifying the position of stent implantation according to claim 1, characterized in that: The stent position recognition model is a YOLOSDB model; the dual-branch backbone network includes an edge branch and a main branch; The edge branch is used to extract edge features of the medical image, and the main branch is used to extract main features of the medical image.

3. The method for identifying the position of stent implantation according to claim 2, characterized in that: The main branch includes a first downsampling module, a first C2 module and a plurality of first block modules; The first downsampling module is used to perform downsampling processing on the input medical image and output a sampling feature map; The first C2 module is used to extract the features of the sampling feature map to obtain a first main feature map, and transmit the first main feature map to the first block module; The multiple first block modules are used to extract features of the first main feature map layer by layer, compress the extracted feature map, and output the main feature map of the corresponding number of layers; the multiple first block modules respectively output the main feature map of the corresponding number of layers.

4. The method for identifying the stent implantation position according to claim 3, characterized in that: The edge branch includes an operator extraction module, a second downsampling module, a second C2 module and a plurality of third C2 modules; The operator extraction module is used to extract edge information of medical images; The second downsampling module is used to perform downsampling processing on the medical image after edge information is extracted to obtain a downsampled edge feature map; The second C2 module is used to extract features from the input edge feature map and the first main feature map, and fuse the extracted edge feature map and the main feature map to obtain a first fused feature; The multiple third C2 modules extract features from the input first fused feature map and the main feature map input from the first block module at the same layer as the third C2 module, and further fuse the first fused feature map after feature extraction with the main feature to obtain a second fused feature map.

5. The method for identifying the position of stent implantation according to claim 4, characterized in that: The Neck network includes a second block module, a plurality of third block modules, a fourth C2 module, a fifth C2 module, an spp module and a feature fusion module; The spp module is used to receive the main feature map output by the first block module of the last layer, and output a feature map of a fixed size to the second block module; The second block module is used to receive the main feature map output by the first block module at the same layer as the first block module and the feature map output by the spp module, and compress the input feature map to output a feature map of the target number of channels; The third block module is used to receive the main feature map output by the first block module at the same layer as the third block module and the feature map of the second block module, and compress the input feature map to output a feature map of the target number of channels; The fourth C2 module is used to receive the second fused feature map output by the third C2 module of the last layer and the second fused feature map output by the third C2 module of the same layer as the fourth C2 module to perform feature extraction, and further fuse the two second fused feature maps after feature extraction to obtain a third fused feature map; The fifth C2 module is used to receive the third fused feature map output by the fourth C2 module of the last layer and the second fused feature map output by the third C2 module of the same layer as the fifth C2 module to perform feature extraction, and further fuse the second fused feature map after feature extraction and the third fused feature map to obtain a fourth fused feature map; The feature fusion module is used to perform feature fusion on the feature map output by the spp module and the second fused feature map output by the third C2 module of the last layer, to perform feature fusion on the third fused feature map output by the fourth C2 module and the feature map output by the second block module, to perform feature fusion on the fourth fused feature map output by the fifth C2 module and the feature map output by the third block module, and to output fused feature maps of corresponding layers respectively.

6. The method for identifying the position of stent implantation according to any one of claims 2 to 5, characterized in that: The C2 module includes a 1X1 convolution layer, a split layer, a 3X3 convolution layer and a connection layer; The 1×1 convolution layer is used to perform convolution processing on the input feature map; The split layer is used to split the feature map after convolution processing into multiple groups according to channels; Each group is sequentially entered into the 1×1 convolution layer and the 3×3 convolution layer for convolution processing, and the feature map after the convolution processing of the 3×3 convolution layer is fused with the feature map after segmentation to obtain a fused feature map; The connection layer is used to further fuse the feature map of each group output by the split layer and the fused feature map.

7. The method for identifying the position of stent implantation according to any one of claims 2 to 5, characterized in that: The block module includes a first convBNSiLU module, a second convBNSiLU module, a Split module, a plurality of BottleNeck modules and a connection layer; The first convBNSiLU module is used to perform a convolution on the input feature map to double the number of channels of the input; The Split module is used to split the feature map after convolution processing into multiple groups according to channels; The feature maps of the multiple groups are sequentially passed through the multiple BottleNeck modules to extract features layer by layer; the BottleNeck module includes multiple convolutional layers; The connection layer is used to splice the feature map output by each BottleNeck module with the original feature map; The second convBNSiLU module is used to compress the concatenated feature map and output a feature map with a target number of channels.

8. A stent implantation position identification device, characterized in that: include: A data set acquisition unit, used for acquiring medical images during multiple stent implantation operations and annotating the medical images to construct a data set; A model building unit, used to build a stent position recognition model, wherein the stent position recognition model includes a dual-branch backbone network, a Neck network, and a detection and recognition module; A model training unit, used for training the stent position recognition model based on the data set to obtain the trained stent position recognition model; The recognition unit is used to obtain the medical image during the current stent implantation operation in real time, input the medical image obtained in real time into the trained stent position recognition model, and output the recognition result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the stent implantation position identification method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the stent implantation position identification method according to any one of claims 1 to 7 when executed.

Images