Blood vessel identification method and related device

Abstract

The present application provides a blood vessel recognition method and related devices. The method at least includes: obtaining an image to be processed; the image to be processed should at least include a surgical image; performing target detection and tracking and positioning based on the image to be processed to obtain tracking and positioning information of each blood vessel structure contained in the image to be processed; performing real-time compression encoding on the image to be processed, and performing mask decoding on the compressed encoded image according to the tracking and positioning information to obtain a first mask image; the first mask image is used to indicate the pixel-level segmentation result corresponding to the blood vessel structure in the real-time surgical field of view; performing mask optimization processing based on the first mask image to obtain a second mask image; the second mask image is used to indicate the pixel-level post-processing result corresponding to the target blood vessel structure in the real-time surgical field of view. The method realizes real-time tracking of surgical images during surgery, strengthens the visual distinction of blood vessel structures, and improves the visual effect of blood vessel structures in surgical images.

Description

Technical Field

[0001] The present application belongs to the field of data processing, and in particular, relates to a blood vessel identification method and related devices. Background Art

[0002] The surgery can be performed with a laparoscope or a robot. In the prior art, image processing and analysis of minimally invasive surgery is mostly static analysis before or after the operation, so it is impossible to provide doctors with real-time auxiliary analysis feedback during the operation, which makes doctors lack reference for intraoperative decision-making. However, in minimally invasive surgery, especially in complex surgery, doctors may need real-time visual guidance and support due to the limited surgical field of view caused by poor image quality and other reasons.

[0003] Therefore, in view of the deficiencies in the prior art, it is urgent to propose a new technical solution to solve at least one of the above technical problems. Summary of the invention

[0004] The embodiments of the present application provide a blood vessel recognition method and related devices for realizing blood vessel recognition and tracking of surgical images, enhancing the visual distinction of blood vessel structures, and improving the visual effects of blood vessel structures in surgical images to provide real-time visual guidance and support.

[0005] In a first aspect, an embodiment of the present application provides a blood vessel identification method, the method comprising:

[0006] Acquire images to be processed; the images to be processed should at least include surgical images;

[0007] Performing target detection and tracking and positioning based on the image to be processed to obtain tracking and positioning information of each blood vessel structure contained in the image to be processed;

[0008] The image to be processed is compressed and encoded in real time, and the compressed and encoded image is mask-decoded according to the tracking and positioning information to obtain a first mask image; wherein the first mask image is used to indicate the pixel-level segmentation result corresponding to the vascular structure in the real-time surgical field of view, and the first mask image is obtained by optimizing a multiple attention mechanism, and the multiple attention mechanism at least includes a self-attention mechanism and a cross-attention mechanism; the multiple attention mechanism is used to process multi-scale image feature information of each vascular structure in different spatial axes;

[0009] Mask optimization processing is performed based on the first mask image to obtain a second mask image; the second mask image is used to indicate a pixel-level post-processing result corresponding to the target vascular structure in the real-time surgical field of view.

[0010] In a second aspect, the present application provides a blood vessel identification device, the device comprising:

[0011] An acquisition module, used for acquiring images to be processed; the images to be processed should at least include surgical images;

[0012] A tracking module, used for performing target detection and tracking positioning based on the image to be processed, and obtaining tracking positioning information of each blood vessel structure contained in the image to be processed;

[0013] A segmentation module, used for performing real-time compression encoding on the image to be processed, and performing mask decoding on the compressed encoded image according to the tracking positioning information to obtain a first mask image; wherein the first mask image is used to indicate the pixel-level segmentation result corresponding to the vascular structure in the real-time surgical field, and the first mask image is obtained by optimizing a multiple attention mechanism, and the multiple attention mechanism at least includes a self-attention mechanism and a cross-attention mechanism; the multiple attention mechanism is used to process multi-scale image feature information of each vascular structure on different spatial axes;

[0014] The optimization module is used to perform mask optimization processing based on the first mask image to obtain a second mask image; the second mask image is used to indicate the pixel-level post-processing result corresponding to the target vascular structure in the real-time surgical field of view.

[0015] In a third aspect, the present application provides an electronic device, comprising: a memory for storing a computer software program; a processor for reading and executing the computer software program, the device being used to implement the steps of the blood vessel identification method provided in any embodiment of the present application.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a storage controller, implements the steps of the blood vessel identification method provided in any embodiment of the present application.

[0017] Compared with the prior art, this application has at least the following beneficial effects:

[0018] In the blood vessel identification method provided by the present application, first, an image to be processed is obtained. For example, the image to be processed should at least include a surgical image. Then, target detection and tracking and positioning are performed based on the image to be processed to obtain tracking and positioning information of each blood vessel structure in the image to be processed. After obtaining the tracking and positioning information, the image to be processed is compressed and encoded in real time, and the compressed and encoded image is mask-decoded according to the tracking and positioning information to obtain a first mask image. Among them, the first mask image is used to indicate the pixel-level segmentation result corresponding to the blood vessel structure in the real-time surgical field of view, and the first mask image is optimized by a multiple attention mechanism, and the multiple attention mechanism at least includes a self-attention mechanism and a cross-attention mechanism. The multiple attention mechanism is used to process the multi-scale image feature information of each blood vessel structure in different spatial axes. Finally, mask optimization processing is performed based on the first mask image to obtain a second mask image; the second mask image is used to indicate the pixel-level post-processing result corresponding to the target blood vessel structure in the real-time surgical field of view. The vascular identification method provided in the embodiment of the present application provides real-time decision support and visual guidance by combining real-time image stream acquisition, multi-scale feature processing, target tracking and positioning, real-time compression coding, mask decoding and other technologies, significantly improving the visual effect of vascular structure in surgery, especially in complex surgeries such as minimally invasive surgery. The visual distinction of vascular structure is enhanced in real time, providing doctors with powerful technical tools and support. This method helps to achieve real-time auxiliary analysis feedback during surgery, helps to strengthen the visual distinction of vascular structure, and improves the visual effect of vascular structure in surgical images, thereby providing doctors with intraoperative decision-making references, and assists in reducing the difficulty of surgery through real-time visual guidance and support. In addition, the vascular identification method provided in the embodiment of the present application is not limited to specific organs or types of surgery, and has the potential to be promoted and applied in other minimally invasive surgeries.

[0019] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a blood vessel identification method according to an embodiment of the present application;

[0021] Figure 2 A schematic diagram showing the principle of a blood vessel identification method according to an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the structure of a channel attention module shown in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the structure of a spatial attention module shown in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the principle of a blood vessel boundary segmentation model shown in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the principle of a lightweight encoder pre-training process shown in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the structure of a blood vessel identification system shown in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of a tracking and positioning module shown in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of the structure of a blood vessel identification device shown in an embodiment of the present application;

[0029] Fig.10 This is a schematic diagram of the structure of a blood vessel identification device shown in an embodiment of the present application.

[0030] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0033] The surgery can be performed with a laparoscope or a robot. In the prior art, image processing and analysis of minimally invasive surgery is mostly static analysis before or after the operation, so it is impossible to provide doctors with real-time auxiliary analysis feedback during the operation, which makes doctors lack reference for intraoperative decision-making. However, in minimally invasive surgery, especially in complex surgery, doctors may need real-time visual guidance and support due to the limited surgical field of view caused by poor image quality and other reasons.

[0034] Taking gastric cancer as an example, due to the complex anatomical structure of the stomach, numerous blood vessels, and rich lymph node metastasis sites, how to efficiently, safely, and accurately perform minimally invasive gastric cancer surgery to reduce the risk of intraoperative injury and related complications is still a key issue affecting surgical efficacy and patient prognosis. In minimally invasive gastric cancer surgery, the treatment and clearance of multiple groups of lymph nodes around the stomach is an important link in determining the treatment effect. During this process, many key blood vessels, such as the portal vein, need to be processed. Therefore, avoiding damage to blood vessels is an urgent problem to be solved in minimally invasive surgery to improve surgical safety and efficiency. Similarly, other types of surgery also have similar technical problems, which will not be elaborated here.

[0035] Therefore, in view of the deficiencies in the prior art, it is urgent to propose a new technical solution to solve at least one of the above technical problems.

[0036] The embodiments of the present application provide a blood vessel identification method and related devices provided by the present application.

[0037] In the method, first, the image to be processed is obtained. For example, the image to be processed should at least include a surgical image. The surgical image here can come from a real-time acquired surgical image stream, which helps to ensure that the processing process can immediately reflect the current surgical status, reduce delays, and provide a real-time data basis for subsequent auxiliary analysis steps. Then, target detection and tracking positioning are performed based on the image to be processed to obtain tracking positioning information of each vascular structure in the image to be processed. In this step, the specific position information of each vascular structure is obtained through target tracking and positioning, which is crucial for subsequent image analysis and decision support, and can ensure accurate processing and tracking of specific vascular structures, which helps to assist doctors to perform surgical operations and make decisions better. After obtaining the tracking positioning information, the image to be processed is compressed and encoded in real time, and the compressed encoded image is mask-decoded according to the tracking positioning information to obtain a first mask image. Among them, the first mask image is used to indicate the pixel-level segmentation result corresponding to the vascular structure in the real-time surgical field of view, and the first mask image is optimized using a multiple attention mechanism, and the multiple attention mechanism at least includes a self-attention mechanism and a cross-attention mechanism. The multiple attention mechanism is used to process the multi-scale image feature information of each vascular structure on different spatial axes. Here, the mixed use of self-attention mechanism and cross-attention mechanism can effectively process multi-scale image feature information on different spatial axes, and enhance the adaptability to complex anatomical structures, such as the changes of vascular structures in surgical images at different scales. For example, image feature information of smaller size can be used to capture vascular structure, while image feature information of larger size can be used to capture structures such as surgical instruments, tissues and organs, so as to eliminate the visual impact of these structures on vascular structure. Finally, mask optimization processing is performed based on the first mask image to obtain a second mask image; the second mask image is used to indicate the pixel-level post-processing result corresponding to the target vascular structure in the real-time surgical field of view. In the above steps, the vascular structure is indicated by the first mask image and the target vascular structure is further extracted to obtain the second mask image, which can provide doctors with real-time intraoperative decision-making reference, which helps to reduce the difficulty of surgery, especially in complex minimally invasive surgery. In addition, the above steps use real-time compression coding and mask decoding, combined with multiple attention mechanisms, to quickly and accurately extract vascular structures in real-time surgical images, which helps to cope with the increased difficulty of surgery caused by changes in vascular position during surgery, as well as the problem of insufficient observation field of view, and provide more accurate visual guidance and support.

[0038] The blood vessel identification method provided in the embodiment of the present application provides real-time decision support and visual guidance by combining real-time image stream acquisition, multi-scale feature processing, target tracking and positioning, real-time compression coding, mask decoding and other technologies, significantly improving the visual effect of blood vessel structure in surgery, especially, strengthening the visual distinction of blood vessel structure in real time in complex surgeries such as minimally invasive surgery. This method helps to achieve real-time auxiliary analysis feedback during surgery, helps to improve the visual effect of blood vessel structure during surgery, strengthens the visual distinction of blood vessel structure, thereby providing doctors with intraoperative decision references, and assists in reducing the difficulty of surgery through real-time visual guidance and support.

[0039] It should be noted that the blood vessel identification method provided in the embodiment of the present application uses real-time compression coding and mask decoding, combined with self-attention and cross-attention mechanisms, which can greatly improve the image processing quality in the field of image segmentation and real-time auxiliary analysis in various surgical processes. In addition, the blood vessel identification method provided in the embodiment of the present application is not limited to specific organs or surgical types, and has the potential to be promoted and applied in other minimally invasive surgeries.

[0040] The blood vessel identification scheme provided in the embodiments of the present application may also be executed by an electronic device, which may be a server, a server cluster, or a cloud server. The electronic device may also be a terminal device such as a computer or a dedicated device (such as a dedicated terminal device with a blood vessel identification system). These electronic devices may also be equipped with the chips described in the above embodiments. Alternatively, these electronic devices may also be installed with a service program for executing the blood vessel identification scheme.

[0041] To solve the above problems, please refer to Figure 1 , Figure 1 FIG. 1 is a flow chart of a blood vessel identification method according to an embodiment of the present application. Figure 1 As shown, at least the following steps are included:

[0042] S101, obtaining an image to be processed;

[0043] S102, performing target detection and tracking and positioning based on the image to be processed, and obtaining tracking and positioning information of each blood vessel structure contained in the image to be processed;

[0044] S103, performing real-time compression encoding on the image to be processed, and performing mask decoding on the compressed encoded image according to the tracking and positioning information to obtain a first mask image;

[0045] S104, performing mask optimization processing based on the first mask image to obtain a second mask image; the second mask image is used to indicate a pixel-level post-processing result corresponding to the target vascular structure in the real-time surgical field of view.

[0046] In the embodiment of the present application, the image to be processed should at least include a surgical image. In practical applications, the image to be processed may be a minimally invasive surgical image, such as an image frame in a real-time acquired surgical image stream. These image frames may contain various surgical scenes, such as surgical operation areas, instruments, tissue structures, etc. In order to accurately capture the details of the surgical scene, the image to be processed usually has a high resolution so that the doctor can clearly observe the subtle changes and structures during the operation. Since these images are from surgical images, the algorithm or system for processing the image to be processed needs to have a relatively powerful computing power to meet the needs of real-time processing to maintain the continuity and smoothness of the image optimization effect during the operation. The image to be processed may cover a variety of medical information, such as vascular structure, instrument position, tissue characteristics, etc. Therefore, the processing algorithm needs to be able to effectively identify and process different types of target structures. Due to the dynamics and complexity of the surgical process, the image to be processed may be affected by noise interference or motion blur, so the processing algorithm needs to have a certain degree of robustness and stability. In the image optimization scenario for minimally invasive surgery, the image to be processed is the core input of the entire system and is crucial to the success and accuracy of the operation. Therefore, the algorithms and technologies for processing the images to be processed need to be optimized according to the characteristics of the surgery to ensure real-time monitoring, analysis, and assistance of the surgical process.

[0047] In an embodiment of the present application, the first mask image is used to indicate the pixel-level segmentation results corresponding to the vascular structure within the real-time surgical field of view. That is, the first mask image reflects the vascular structure information within the real-time surgical field of view. These vascular structures may be important blood vessels in the surgical operation area, and need to be accurately identified and segmented during the operation to assist doctors in performing accurate surgical operations. Further optionally, the pixel-level segmentation results in the first mask image can be used to represent the probability or classification of each pixel point belonging to a vascular structure in the image. Through this segmentation result, the vascular structure can be analyzed and located more finely, providing doctors with more intuitive and accurate surgical field information.

[0048] Further optionally, the first mask image is optimized using a multiple attention mechanism, and the multiple attention mechanism at least includes a self-attention mechanism and a cross-attention mechanism.

[0049] Specifically, the use of a multi-attention mechanism combining the self-attention mechanism and the cross-attention mechanism is to further improve the quality and accuracy of the first mask image. Among them, the self-attention mechanism allows each pixel in the image to establish an association with other pixels and weight them according to their relative importance. In the process of generating the first mask image, the self-attention mechanism can help the system better capture the spatial relationship and local features inside the image, thereby improving the accuracy of segmentation and positioning of vascular structures. The cross-attention mechanism allows information interaction and association between different images, especially the learning and application of cross-attention between multiple image frames. In the real-time surgical image stream, image frames at different time points may contain information about the same vascular structure, so the cross-attention mechanism can help the system fuse and complement information between different image frames. Therefore, the hybrid fusion of the self-attention mechanism and the cross-attention mechanism can enable the system to more comprehensively utilize the information inside and between images, thereby generating a more accurate and stable first mask image. This fusion of multiple attention mechanisms can improve the segmentation accuracy, continuity and robustness of vascular structures. By adopting a multi-attention mechanism that is a mixture of self-attention mechanism and cross-attention mechanism, it helps to integrate more image information, thereby providing a data basis for the pixel-level segmentation of vascular structures in the first mask image, further assisting in improving the accuracy and real-time performance of minimally invasive surgical image optimization processing, and providing more reliable auxiliary information for the surgical process.

[0050] In the embodiment of the present application, the multiple attention mechanism is used to process the multi-scale image feature information of each vascular structure in different spatial axes.

[0051] It is understandable that in practical applications, the vascular structures in the surgical scene have features of different scales, such as small blood vessels and larger blood vessels. Multi-scale image feature information covers various features from subtle structures to overall structures. In surgical images (such as the images to be processed involved in the embodiments of the present application), vascular structures may exist in multiple spatial axes such as horizontal, vertical and depth. Therefore, multiple attention mechanisms are used to simultaneously process information on these different spatial axes to ensure a comprehensive understanding and accurate segmentation of vascular structures. The self-attention mechanism is used to establish associations between pixels on each spatial axis to capture local features and spatial relationships. The cross-attention mechanism is used for information interaction and fusion between different spatial axes to ensure the global consistency and continuity of multi-scale features. By fusing the self-attention mechanism and the cross-attention mechanism, the system can comprehensively utilize multi-scale image feature information on different spatial axes to produce a more accurate and complete first mask image. This fusion can effectively process the multi-scale features of vascular structures on each spatial axis and fuse image visual information in more dimensions. In summary, in the embodiments of the present application, multiple attention mechanisms are used to process the multi-scale feature information of each vascular structure in the surgical image in different spatial axes. Through the fusion of self-attention mechanism and cross-attention mechanism, accurate segmentation and comprehensive understanding of the vascular structure are ensured, providing reliable auxiliary information for minimally invasive surgery.

[0052] In the embodiments of the present application, by combining real-time image stream acquisition, multi-scale feature processing, target tracking and positioning, real-time compression coding, mask decoding and other technologies, real-time decision support and visual guidance are provided, which significantly improves the visual effect of vascular structure in surgery, especially in complex operations such as minimally invasive surgery. The visual distinction of vascular structure is enhanced in real time. The vascular identification method provided in the embodiments of the present application helps to achieve real-time auxiliary analysis feedback during surgery, helps to improve the visual effect of vascular structure during surgery, and strengthens the visual distinction of vascular structure, thereby providing doctors with intraoperative decision-making references, and assists in reducing the difficulty of surgery through real-time visual guidance and support. In addition, the vascular identification method provided in the embodiments of the present application is not limited to specific organs or types of surgery, and has the potential to be promoted and applied in other minimally invasive surgeries.

[0053] The specific implementation of each step is introduced below with reference to specific examples.

[0054] S101, obtaining an image to be processed.

[0055] In an embodiment of the present application, the image to be processed may be a surgical image. For example, if it is necessary to identify the vascular structure in a laparoscopic image, the image to be processed may be a surveillance video image of the laparoscopic surgery, or an image obtained after image preprocessing based on the surveillance video image. Specifically, in order to facilitate subsequent recognition operations, the image originally acquired by the image acquisition device may be preprocessed, such as horizontal / vertical flipping, rotation, blurring, sharpening, noise, contrast enhancement, color channel conversion, and the like. Those skilled in the art may select a suitable image enhancement processing method according to actual needs, which is not limited here.

[0056] It can be seen that the image to be processed in the embodiment of the present application can be an image originally captured by the image capture device, or it can be an image obtained by processing the originally captured image.

[0057] In practical applications, the images to be processed may be image frames in a real-time surgical image stream collected during minimally invasive surgery. These image frames may contain various surgical scenes, such as surgical operation areas, instruments, vascular structures, tissue structures, etc. This application mainly focuses on vascular structures within the surgical field of view.

[0058] S102, performing target detection and tracking and positioning based on the image to be processed, and obtaining tracking and positioning information of each blood vessel structure contained in the image to be processed.

[0059] For example, suppose that in a minimally invasive surgery, the doctor needs to accurately identify the gastric vascular structure to ensure the safety and effectiveness of the surgical operation. Through step S102, each vascular structure can be detected in the image to be processed, and the specific position of each vascular structure and the position change caused by the change of the motion trajectory of the laparoscope can be tracked and located. For example, the specific position of structures such as the portal vein, the trend of position change, and the blood flow conditions. These information on tracking and locating each vascular structure can be updated in real time and reflected in the tracking and positioning information to assist in the accuracy of identifying the position and changes of the vascular structure in the subsequent analysis process, thereby assisting doctors to avoid accidental injury to key blood vessels and ensure the high accuracy and safety of the operation.

[0060] As an optional embodiment, in S102, target detection and tracking and positioning are performed based on the image to be processed to obtain tracking and positioning information of each blood vessel structure contained in the image to be processed, such as Figure 2 As shown, it can be implemented as follows:

[0061] S201, performing stage-by-stage feature extraction based on the image to be processed to obtain multi-scale first image feature information;

[0062] S202, performing feature fusion on the first image feature information, and performing attention enhancement on the blood vessel structure in the fused first image feature information to obtain second image feature information;

[0063] S203, performing feature extraction based on the second image feature information to obtain positioning feature information corresponding to each blood vessel structure;

[0064] S204, screening out positioning feature information that meets preset conditions as target positioning feature information corresponding to the image to be processed;

[0065] S205: Perform feature matching on the target positioning feature information and the reference positioning feature information to obtain a tracking and positioning result of the image to be processed.

[0066] Specifically, in S201, the first image feature information is obtained by inputting the image to be processed into a staged feature extraction model. A channel attention module is added to the backbone network of the staged feature extraction model. That is, in S201, the image to be processed is input into a staged feature extraction model, which is used to extract the first image feature information. The staged feature extraction model adopts a multi-scale feature extraction method, which means that it can extract image features from different scales (or different resolutions). This method can be achieved by using convolution filters of different sizes in the backbone network or by pooling operations. For example, the CSPDarknet53 backbone network in Yolov8 can perform staged feature extraction and extract multi-scale feature maps by gradually reducing the resolution of the image (or called downsampling). In some examples, the single-stage target detection model (Single Shot MultiBox Detector, SSD) efficient target detection model (EfficientDet) and target detection model (RetinaNet) can also be used as the basic network to implement similar functions of the above-mentioned Yolov8, which will not be repeated here.

[0067] Exemplarily, the channel attention module added to the backbone network of the staged feature extraction model is an important component for enhancing the feature capabilities in the feature map. The channel attention module can be implemented using a SE module such as SENet (Squeeze-and-Excitation Networks), which can dynamically adjust the weights of features according to the importance of feature channels. This helps the model focus on features that are more important to the task while suppressing less important features. Adding a channel attention module to the CSPDarknet53 backbone network of Yolov8 can further enhance the network's ability to extract features, thereby improving the accuracy of target detection. Of course, the above functions can also be achieved by adding a channel attention module to other similar models, such as SSD, EfficientDet, RetinaNet, etc.

[0068] Specifically, when an image is input to the staged feature extraction model, the staged feature extraction model first performs initial feature extraction, and then obtains feature information at different levels through a multi-scale feature extraction process. During the extraction process, the channel attention module can help the model better understand which features are more critical for target detection and localization, thereby enhancing the performance of the detection model. Through this combination of staged feature extraction and channel attention mechanism, the model can capture the key features of vascular structure at different scales, improving the accuracy and robustness of detection.

[0069] Further optionally, the second image feature information is obtained by inputting the first image feature information into a global attention model. Exemplarily, the global attention model is constructed based on a global attention mechanism (GAM). Assume that the global attention model includes the following structures: an image fusion module, a channel attention module, and a spatial attention module. Based on the above assumed structure, in S202, an image fusion module is used to fuse the multi-scale image features in the first image feature information into the first reconstructed feature information. This process generates the first reconstructed feature information by integrating features from different sizes and different resolutions. This fusion helps to integrate information of features of different scales and improve the global representation capability of features, so that the model can better understand the overall picture and details of the image.

[0070] Furthermore, in S202, a channel attention module is used to divide the first reconstructed feature information into corresponding multi-channel feature blocks, and deep feature extraction and feature mixing are performed on the multi-channel feature blocks respectively to reconstruct the second reconstructed feature information. In the channel attention module, the first reconstructed feature information is divided into a plurality of corresponding channel feature blocks. Then, through convolution operations of different step lengths, deep feature extraction and feature mixing are performed on each channel feature block respectively. This not only takes into account the attention weight of each channel, but also captures the different size features of multiple types of targets, especially the fine structure of blood vessels, through convolution operations of different step lengths, thereby effectively enhancing the detection accuracy of key vascular images. This method applies different weights to each channel, so that the model pays more attention to those feature channels that are more important in the target detection task, thereby improving the overall feature representation ability of the model. Exemplarily, the channel attention module can be constructed using the SE (Squeeze-and-Excitation) module, which mainly calculates the attention weight of each channel through compression and extraction.

[0071] Furthermore, in S202, a spatial attention module is used to calculate the bidirectional cross-attention information of the second reconstructed feature information between the parallel axial attentions to reconstruct the second image feature information. This module further enhances the model's attention to key vascular structures by focusing on the relevance and importance of different spatial positions in the image. Through this bidirectional cross-attention mechanism, the spatial attention module can effectively capture the relationship between feature information in the spatial dimension, thereby reconstructing more accurate and detailed second image feature information.

[0072] Based on the above steps, in S202, the global attention model can fully utilize and reconstruct the first image feature information through the joint use of the image fusion module, the channel attention module and the spatial attention module. Through the fusion of multi-scale features, the calculation of feature weights of different channels and the attention mechanism in the spatial dimension, the model can more accurately extract and characterize the vascular structure, thereby improving the accuracy and reliability of medical image analysis and providing strong support for surgical assistance and medical diagnosis.

[0073] As an optional embodiment, in the above S202, a channel attention module is used to divide the first reconstructed feature information into corresponding multi-channel feature blocks, and deep feature extraction and feature mixing are performed on the multi-channel feature blocks respectively to reconstruct the second reconstructed feature information, which can be implemented as follows:

[0074] The first reconstructed feature information is divided into corresponding multi-channel feature blocks according to a preset number of channels; each multi-channel feature block is input into a multi-perception unit to perform feature conversion, and the converted multi-channel feature blocks are subjected to depthwise separable dilated convolution according to corresponding sizes to obtain image context features at different sizes; the image context features at different sizes are normalized to obtain normalized image context features; the normalized image context features are reorganized and mixed, and the image context features corresponding to different channels are cross-group fused and full-channel fused according to channel indexes to obtain the second reconstructed feature information.

[0075] Specifically, according to the preset number of channels, the first reconstructed feature information is divided into corresponding multi-channel feature blocks. For example, the specific calculation process of the channel attention module in the following example is shown in Figure 3 The calculation process is shown.

[0076] In this example, if Figure 3 As shown, assuming that the number of channels is four, then the size of the input feature map Fin is H×W×C, and it is divided into four equal parts along the channel to generate Fin_1 to Fin_4∈R H×W×C / 4. This four-equal division method is more balanced, and features of objects of different sizes can be extracted through convolutions of different step sizes. For example, convolutions with a step length of 8 are used to extract features of large objects, while convolutions with a small step length (such as 1) are used to extract features of small objects such as lymph nodes or thin blood vessels. Each multi-channel feature block is input into a multi-layer perceptron (MLP) to perform feature conversion. It should be noted that a multi-layer perceptron is mainly composed of multiple fully connected layers and can be used for classification and regression tasks. MLP can extract useful information from the data and convert it into a form that is easier to handle, thereby facilitating channel attention to learn the features of the target. The converted multi-channel feature blocks are subjected to depthwise separable dilated convolutions, such as dilated depthwise separable convolution (dilatedDwConv), and different convolution steps (1, 2, 4, 8) are used to extract image context features of different sizes. When implementing depth-wise separable convolution, first perform independent convolution on each channel, and then merge all channels into the output feature map through point convolution (PWConv), so as to reduce the amount of calculation and improve the calculation efficiency. This operation helps to capture the target feature information of different sizes. Then, the image context features at different sizes are normalized, and the standard normalization layer (Batch Normalization, BN) and rectified linear unit (Rectified Linear Unit, Relu) operations are used to obtain the normalized image context features Fc_1 to Fc_4. The normalized image context features are reorganized and mixed, such as using the Focus operation similar to Yolov5, to achieve the diversity enhancement of the feature map in the channel dimension. The Focus operation mainly focuses on certain areas of the input image to extract more useful information in the network and compress the spatial size of the feature map.

[0077] In this example, Figure 3 As shown, assuming that the channel indexes of Fc_1 to Fc_4 are 1 to 4, the jth channel of Fc_i' is j = 4*k+i, where k∈(1…4). Where i is one of 1, 2, 3, and 4. Finally, a 1×1 convolution is performed to fuse the diverse information within the group. In the embodiment of the present application, after the channel reorganization, each group contains the feature information of targets of different scales, and then through a 1×1 convolution (for example Figure 31×1Conv in ) to perform channel information fusion. Finally, through the tensor splicing operation under the channel dimension and a 1×1 convolution, cross-group information fusion and the output of the overall channel attention module are realized, and finally the second reconstructed feature information is obtained. For example, the tensor splicing operation can be a Concat operation, that is, an operation of tensor splicing in deep learning, which can be used to connect multiple feature maps or feature vectors according to a certain dimension (usually the channel dimension). For example, in Figure 3 In the example, the second reconstructed feature information (i.e., Fout) is obtained through the Concat and 1×1Conv operations.

[0078] In this way, through multi-channel division, feature conversion, depth-separable dilated convolution, normalization processing, reorganization mixing and diverse information fusion of the first reconstructed feature information, the algorithm can effectively enhance its attention to key vascular image information, improve the accuracy of feature extraction and representation, and thus generate more accurate second reconstructed feature information.

[0079] In the prior art, the spatial attention module performs global average pooling and global maximum pooling on the input feature map in the channel dimension to generate two feature maps with the same size as the input feature map but with 1 channel number. After splicing, the two feature maps are convolved with a 7×7 convolution kernel, and then the spatial attention weights are generated after passing through the Sigmoid activation function. It can be seen that the existing spatial attention mechanism is not sensitive enough to long strip targets such as blood vessels or surgical instruments, and the recognition accuracy is not high. In order to overcome this defect, in order to deal with the significant changes in the size and shape of individual lesions or blood vessels in this application, strip convolution kernels of different sizes are used in each axial attention path for multiple convolutions to effectively extract features. The embodiment of the present application provides a spatial attention mechanism module for processing image feature information in different axes to improve the detection accuracy of the target.

[0080] As an optional embodiment, in the above S202, a spatial attention module is used to calculate the bidirectional cross attention information of the second reconstructed feature information between the parallel axial attentions to reconstruct the second image feature information, which can be implemented as follows:

[0081] Perform multi-scale convolution operations on the second reconstructed feature information in parallel axes, and perform feature aggregation on the operation results to obtain aggregated feature maps on different parallel axes; perform matrix addition on the aggregated feature maps on different parallel axes, and input the addition result matrix into the fully connected layer of the multi-layer perception unit for feature fusion processing to obtain feature fusion matrices on different parallel axes; perform matrix dot multiplication operations on the feature fusion matrices on different parallel axes to obtain an attention feature weight matrix for each image feature point on the parallel axis; perform matrix dot multiplication on the attention feature weight matrix and the aggregated feature maps on different parallel axes to obtain enhanced feature maps on different parallel axes; perform convolution fusion processing on the enhanced feature maps on different parallel axes to obtain the second image feature information.

[0082] Specifically, in S202, the spatial attention module is used to process the second reconstructed feature information, aiming to reconstruct the feature information of the second image through the bidirectional cross attention between the parallel axial attentions. For example, the specific calculation process of the spatial attention module in the above example can be found in Figure 4 The calculation process is shown in Figure 4 In the first step, for the second reconstructed feature information (such as Fout introduced above), Figure 4 F H×W×C ) is processed into a standard quantity (i.e., Norm) on the parallel axis, and convolution operations are performed using convolution kernels of different sizes on the parallel axis (assuming that they are in the X and Y directions). For example, a 1×7 convolution kernel is used for convolution in the X direction (i.e., Figure 4 In 1×7Conv, a 7×1 convolution kernel is used for convolution in the Y direction (i.e. Figure 4 7×1Conv in the middle); use 1×13 convolution kernel for convolution in the X direction (i.e. Figure 4 In 1×13Conv, a 13×1 convolution kernel is used for convolution in the Y direction (i.e. Figure 4 13×1Conv in the middle); use 1×21 convolution kernel for convolution in the X direction (i.e. Figure 4 In 1×21Conv, a 21×1 convolution kernel is used for convolution in the Y direction (i.e. Figure 4 21×1Conv in the figure). The convolution operation on each axis is designed to extract features of different scales. Then, the feature maps (Fx and Fy) obtained on each axis are convolved by 1×1 convolution (such as Figure 4 1×1Conv) performs feature aggregation to obtain the aggregated feature maps on different parallel axes, that is, Figure 4 Two Fs in the middle H×W×C Then, the aggregate feature maps along different parallel axes (such as Figure 43×3Conv) in the image to obtain a fused feature matrix. This step aims to integrate information in the X and Y directions. Next, the feature fused matrix is ​​input into the fully connected layer of the multi-layer perception unit mlp for higher-level feature fusion processing. The fully connected layer can help the model learn and combine complex feature relationships in the X and Y axes. Furthermore, the attention feature weight matrix of each image feature point in the parallel axes (X and Y) is generated from the output of the fully connected layer. These weights will be used for subsequent feature enhancement. For example, Figure 4 In one axis, H×CW, CW×H, and in the other axis, W×HC, HC×W. Next, using the attention feature weight matrix, matrix dot multiplication is performed on the aggregated feature maps in the X and Y axes respectively to obtain the enhanced feature maps in each axis. This step enables the model to focus on important feature parts. The above processing also includes Figure 4 Then, the enhanced feature maps on different parallel axes are concatenated through the Concat operation, and a convolution kernel (such as 1×1Conv) is used for further feature fusion processing. For example, Figure 4 After the Concat operation, H×W×2C and H×W×C are obtained. Finally, the feature map after convolution fusion is the reconstructed feature information of the second image (i.e. Figure 4 The reconstructed output F H×W×C’ ), which includes a refined feature representation enhanced by a parallel axial attention mechanism.

[0083] Through the above steps, the spatial attention module in S202 can effectively process the second reconstructed feature information, and use the axial attention mechanism to extract multi-scale features and learn complex relationships, thereby improving the recognition accuracy and sensitivity of long-shaped targets (such as blood vessels). Through the above improved spatial attention module, it is possible to better capture global information and axial features, improve the ability to handle significant changes in the size and shape of individual lesions or blood vessels, and thus improve the accuracy and sensitivity of target detection.

[0084] In S203, feature extraction is performed based on the second image feature information to obtain positioning feature information corresponding to each vascular structure. Further optionally, it is assumed that the positioning feature information corresponding to each vascular structure includes: a target positioning frame corresponding to each vascular structure. Based on this, in S203, multiple convolution operations are performed on the second image feature information to extract candidate positioning frames corresponding to each vascular structure.

[0085] For example, a series of multiple convolution operations are performed using the feature information of the second image as input. This process involves the superposition of multiple convolutional layers, and high-level features in the image are extracted through convolution kernels of different scales and depths. Each layer of convolution operation is usually followed by an activation function (such as ReLU) and a pooling layer (such as Max Pooling) for nonlinear transformation and dimensionality reduction, thereby extracting richer and more semantic feature representations. Furthermore, in the last layer of the multi-layer convolutional network, the high-dimensional features are mapped to a predefined target positioning box through a specific detection head or a fully connected layer. For each vascular structure, its four boundaries (the coordinates of the upper left corner and the lower right corner) are predicted by regression to form candidate positioning boxes (Bounding Boxes). These candidate boxes represent areas that may contain vascular structures.

[0086] S204, filter out the positioning feature information that meets the preset conditions as the target positioning feature information corresponding to the image to be processed. Further optionally, continue to assume that the positioning feature information corresponding to each vascular structure includes: a target positioning frame corresponding to each vascular structure. Based on this, in S204, traverse the candidate positioning frames corresponding to each vascular structure, calculate the confidence of each candidate positioning frame, and select the candidate positioning frame whose confidence is not less than the set confidence threshold as the target positioning frame corresponding to the image to be processed.

[0087] For example, the candidate positioning frames corresponding to each vascular structure generated in step S203 are traversed, and each candidate frame is further verified and screened. For each candidate positioning frame, its confidence (Confidence) is calculated through a classification head (ClassificationHead). The confidence indicates the probability that the candidate positioning frame does contain the target object (such as a vascular structure). The confidence is usually calculated by the Softmax function or the Sigmoid function, and depends on the classification labels of the target object and the background during the training process. The confidence of each candidate positioning frame is compared with the preset confidence threshold (ConfidenceThreshold). Only candidate positioning frames whose confidence is not lower than the threshold will be retained. This step ensures that the final output positioning frame has a high accuracy and reduces the possibility of false detection or missed detection.

[0088] Furthermore, in order to avoid repeated detection, the non-maximum suppression (NMS) algorithm can also be applied. This algorithm calculates the overlap area (Intersection over Union, IoU) between candidate positioning frames, retains the candidate positioning frame with the maximum confidence, and suppresses other candidate positioning frames with higher overlap, thereby further improving the detection accuracy. For each type of target, the candidate positioning frames are arranged in descending order of confidence, and the candidate positioning frame with the highest confidence is selected in turn as the final output target positioning frame, and all other candidate positioning frames whose overlap with the frame exceeds the preset threshold are deleted.

[0089] Alternatively, in another example, a non-maximum suppression algorithm is used to traverse the aforementioned candidate positioning frames, calculate the confidence of each candidate positioning frame, filter out candidate positioning frames with a threshold value less than 0.75, and obtain the best positioning frame of the current frame image, that is, the target positioning frame.

[0090] Through steps S203 and S204, multiple convolution operations are first performed based on the feature information of the second image to extract high-level feature information and generate candidate positioning frames. After the candidate positioning frames are generated, confidence calculation and screening, as well as technical means of non-maximum suppression, are used to finally obtain a target positioning frame that meets the preset conditions, ensuring a highly accurate and robust vascular structure detection result.

[0091] S205: Perform feature matching on the target positioning feature information and the reference positioning feature information to obtain a tracking and positioning result of the image to be processed.

[0092] Exemplarily, it is further assumed that the positioning feature information corresponding to each vascular structure includes: a target positioning frame corresponding to each vascular structure. Furthermore, it is assumed that the image to be processed includes multiple continuous image frames. In this case, the reference positioning feature information is predicted based on the historical target positioning feature information, and the historical target positioning feature information is the target positioning feature information corresponding to the previous frame of the image to be processed.

[0093] Based on the above assumptions, in S205, for the current frame image in the image to be processed, the target positioning frame is used to perform feature extraction to obtain a first reference positioning frame; based on the first reference positioning frame and the historical target positioning frame corresponding to the previous frame image, a second reference positioning frame corresponding to the current frame image is predicted; and a matching detection is performed on the target positioning frame and the second reference positioning frame to update the tracking and positioning result of the current frame image.

[0094] When processing multiple continuous image frames, it is necessary to ensure the continuous tracking of the target (such as vascular structure) in the time series through feature matching and tracking positioning. Specifically, feature extraction is performed on the current frame image to obtain the target positioning frame corresponding to each vascular structure. Assume that these target positioning frames have been generated through steps S203 and S204, indicating the area that may contain the vascular structure. Based on the historical target positioning feature information, that is, the target positioning frame of the previous frame image, as the reference positioning feature information of the current frame image. It is also possible to perform the step of extracting features using the target positioning frame for the current frame image in the image to be processed in S205 through a lightweight feature extraction network to obtain a first reference positioning frame. This step can be completed by a lightweight feature extraction network, which can further reduce the computational overhead. Furthermore, based on the first reference positioning frame and the historical target positioning frame of the previous frame image, a motion model (such as Kalman filtering or optical flow method) is used to predict the second reference positioning frame corresponding to the current frame image. The motion model predicts its position in the current frame by analyzing the motion trajectory of the target in the time series. Matching detection is performed on the target positioning frame and the second reference positioning frame. For example, the matching algorithm can be a Hungarian matching algorithm, a nearest neighbor matching algorithm, a Hungarian algorithm, etc. Then, the similarity between the target positioning frame and the second reference positioning frame is calculated (for example, by IoU or feature distance), and the matching positioning frame pair with the highest similarity (i.e., the target positioning frame and the second reference positioning frame) is selected. Then, based on the matching result, the tracking and positioning result of the current frame image is updated. If the similarity between the target positioning frame and the second reference positioning frame exceeds a preset threshold, the target is considered to be successfully tracked, and the position and size of the target are updated. If there is no suitable matching pair, the target is marked as lost, and a re-detection or tracking mechanism is initiated. The tracking and positioning result of the current frame image is updated to ensure the accuracy of target tracking between consecutive image frames.

[0095] Through the above step S205, the target (such as a blood vessel structure) can be accurately tracked in multiple consecutive image frames, thereby improving the accuracy and robustness of target detection and tracking.

[0096] Further optionally, in the above step S205, matching detection is performed on the target positioning frame and the second reference positioning frame to update the tracking positioning result of the current frame image, which can be implemented as follows:

[0097] The Mahalanobis distance between the target positioning frame and the second reference positioning frame is calculated; if the calculated Mahalanobis distance is not greater than a set distance threshold, it is determined that the target positioning frame and the second reference positioning frame are matched as the same detection target, and the target positioning frame is set as the tracking and positioning result of the image to be processed; if the calculated Mahalanobis distance is greater than the set distance threshold, it is determined that the target positioning frame and the second reference positioning frame do not belong to the same detection target, and the target positioning frame is updated and set as the newly added tracking and positioning result in the image to be processed.

[0098] In another optional example, in the above step S205, after calculating the Mahalanobis distance between the target positioning frame and the second reference positioning frame, further optionally, the target positioning frame and the second reference positioning frame can also be feature extracted through the re-identification model to obtain the first feature vector corresponding to the target positioning frame and the second feature vector corresponding to the second reference positioning frame. Then, the first feature vector and the second feature vector are input into the cosine distance cost function to calculate the appearance feature cosine distance between the first feature vector and the second feature vector. Finally, if the calculated appearance feature cosine distance is not greater than the set appearance feature cosine distance threshold, and the calculated Mahalanobis distance is not greater than the set distance threshold, it is determined that the target positioning frame and the second reference positioning frame are matched as the same detection target, and the target positioning frame is set as the tracking and positioning result of the image to be processed. If the calculated appearance feature cosine distance is greater than the set appearance feature cosine distance threshold, or the calculated Mahalanobis distance is greater than the set distance threshold, it is determined that the target positioning frame and the second reference positioning frame do not belong to the same detection target, and the target positioning frame is updated and set as the newly added tracking and positioning result in the image to be processed.

[0099] In this way, the above optimization method can more accurately measure the similarity between the two frames and improve the matching accuracy. By setting a reasonable distance threshold, different targets can be effectively distinguished and the possibility of false detection can be reduced. In addition, by combining historical data and current features, high tracking stability and robustness can be maintained in complex backgrounds. In summary, by using the Mahalanobis distance for matching detection, the similarity between the target positioning frame and the reference positioning frame can be more accurately judged, thereby improving the accuracy and stability of tracking positioning. This method not only takes into account the spatial position of the target, but also combines the feature distribution, making the matching process more reliable and effective.

[0100] S103, performing real-time compression encoding on the image to be processed, and performing mask decoding on the compressed encoded image according to the tracking and positioning information to obtain a first mask image.

[0101] Specifically, in S103, first, the image to be processed is input into a pre-trained lightweight encoder for real-time compression encoding to obtain the compressed encoded image. The lightweight encoder is pre-trained synchronously with the deep visual converter on the same image training set and is obtained by using fewer training parameters. Here, the lightweight encoder is pre-trained synchronously with the deep visual converter on the same image training set, and the number of parameters is small, thereby reducing the computational overhead while maintaining performance. The image to be processed is processed by the lightweight encoder to obtain a compressed encoded image. This step is to reduce the amount of data while retaining necessary information to meet the requirements of real-time processing. Then, the target positioning frame corresponding to each vascular structure is mapped to the compressed encoded image to obtain the image mapping feature of the compressed encoded image. This step ensures the effective expression of the positioning information of each vascular structure in the compression domain. Then, the self-attention mechanism is used to update the association between different target positioning frames in the image mapping feature, which helps to capture the spatial dependency between different vascular structures and improve the accuracy of segmentation. And the cross-attention mechanism is used to update the interactive relationship between the mapping feature information and the target positioning frame in the image mapping feature to improve the accuracy of segmentation. In this way, the association between the image mapping features and the positioning frame information can be strengthened, and the segmentation accuracy can be further improved. Finally, the updated image mapping features are input into the multi-layer perceptron and the dynamic linear classifier to predict the first mask image. These classifiers can accurately identify and segment the vascular structure based on the updated feature mapping, and the first mask image finally output has a pixel-level vascular segmentation result.

[0102] In summary, the use of lightweight encoders for real-time compression encoding greatly improves the processing speed and response time, and is suitable for application scenarios that require fast feedback, such as real-time minimally invasive surgery. The application of self-attention mechanism and cross-attention module effectively enhances the expression ability of image mapping features, improves the precision and accuracy of vascular segmentation, and helps to accurately identify tiny vascular structures during surgery. Through pre-training and parameter optimization, it is ensured that the lightweight encoder reduces the requirements for hardware equipment while maintaining performance, and enhances the reliability and stability of the system. Through the above steps, real-time compression encoding and accurate vascular segmentation of the processed images can be effectively achieved, providing powerful image processing capabilities and support for fields such as real-time minimally invasive surgery.

[0103] S104, performing mask optimization processing based on the first mask image to obtain a second mask image; the second mask image is used to indicate a pixel-level post-processing result corresponding to the target vascular structure in the real-time surgical field of view.

[0104] Specifically, in S104, the pixels in the mask area of ​​the first mask image are filtered to obtain the first mask image after filtering. The filtering granularity in the filtering process is dynamically set based on the observation requirements of minimally invasive surgery. Furthermore, the mask edge area in the first mask image after filtering is subjected to bilateral filtering to obtain a grayscale mask image. Finally, multi-layer edge detection is performed on the grayscale mask image to obtain the second mask image.

[0105] In the above step S104, first, the pixels of the mask area in the first mask image are filtered to obtain the first mask image after filtering. The granularity of the filter is dynamically set according to the observation requirements of minimally invasive surgery, which means that the parameters of the filter can be adjusted according to the needs of the surgical field and the target vascular structure. Next, the mask edge area in the first mask image after filtering is subjected to bilateral filtering. Bilateral filtering is a nonlinear filtering technique that can retain the edge information of the image while eliminating noise. After bilateral filtering, a grayscale mask image is obtained. This step is to convert the processed mask area into a grayscale image for the next step of edge detection. Multi-layer edge detection is performed on the grayscale mask image to obtain a second mask image. Multi-layer edge detection is usually performed by edge detection through convolution kernels of different scales to capture image edge information at different levels. Finally, the second mask image is used to indicate the pixel-level post-processing results of the target vascular structure in the real-time surgical field.

[0106] The specific implementation of S104 in this process combines filtering, bilateral filtering, grayscale conversion and edge detection to achieve accurate recognition and segmentation of target vascular structures in minimally invasive surgery, providing important visual support and guidance for real-time surgery.

[0107] Exemplarily, in the above-mentioned step of performing bilateral filtering on the mask edge area in the first mask image after filtering to obtain a grayscale mask image, for the first mask image after filtering, the spatial proximity between each mask area pixel and the central pixel is calculated to obtain the spatial proximity corresponding to each mask area pixel. These proximity values ​​reflect the distance relationship between pixels, which are usually calculated by a Gaussian function, and the closer the distance, the higher the proximity of the pixel. At the same time, the similarity of the pixel values ​​of each mask area pixel is calculated. This spatial proximity calculation step evaluates the similarity of the pixel points in grayscale, which is measured by the difference between the pixel values. Finally, based on the product of the spatial proximity and the pixel value similarity, the weight value of each mask area pixel is adjusted, and the adjusted mask area pixel points are convolved to obtain the grayscale mask image after the mask edge is smoothed. In this step, the product of spatial proximity and pixel value similarity is used as a weight value to adjust the attention of each pixel in the mask area, so that the image information of pixels with close distances and similar pixel values ​​is applied to subsequent processing to retain image edge information and details. The purpose of using the above weight value to perform a convolution operation on the first mask image after filtering is to calculate the updated value of the output pixel through the convolution operation using the weight value and the value of each pixel in the image, thereby achieving smoothing of the mask edge.

[0108] After the above smoothing process, a grayscale mask image after bilateral filtering is obtained. This image retains the smoothing effect of the mask edge, aiming to improve the subsequent segmentation algorithm's ability to accurately identify targets such as vascular structures, while reducing interference to doctors' vision.

[0109] The above steps S103 and S104 can be implemented by a blood vessel boundary segmentation model based on an efficient segmentation everything model (Efficient SAM). The data processing flow of the blood vessel boundary segmentation model is as follows: Figure 5 As shown. Figure 5 In the data processing flow shown, the input image P i That is, the image to be processed mentioned above, the image to be processed P i After being encoded by the image encoder (i.e. the lightweight encoder mentioned above), the encoded result and the target positioning box bbox are i Input to the image decoder for image segmentation prediction processing to obtain the prediction mask M i1 (i.e., the first mask image described above). Then, the optimized mask M is obtained by using the mask filter processing. i2 (ie the second mask image mentioned above).

[0110] For example, an image encoder pre-trained with the high-performance feature extractor ViT (also known as the visual transformer) in the efficient segmentation model SAM is used to process the input image. The specific steps are as follows: use the SAM high-performance encoder ViT-H (also known as the high-performance visual transformer) in the SAM model and a lightweight image encoder for pre-training. Through the Masked Autoencoders (MAE) technology, the lightweight encoder is trained on the same image training set as ViT-H to reduce the number of parameters and computational overhead. MAE is a method for feature learning and data representation, which is often used in the fields of image processing and computer vision. MAE combines the concept of autoencoder and the technology of mask learning. It realizes the learning and reconstruction of specific areas or features by controlling the reconstruction ability of the autoencoder. It is suitable for scenarios that require fine control and efficient reconstruction. During the training process, the loss difference between ViT-H and the lightweight encoder is calculated. Through multiple rounds of iterative training, the feature extraction ability of the lightweight encoder is made close to ViT-H. For example, the lightweight encoder pre-training process introduced in the above example, such as Figure 6 shown.

[0111] For example, in Figure 6 In the pre-training process of the SAM high-performance encoder ViT-H and the lightweight image encoder, the original image is processed into a masked original image, and the image encoder (lightweight ViT) is used to process it to obtain the masked image features. The masked image features are input into the cross-attention decoder for image segmentation to obtain the segmentation results. At the same time, the original image is processed by the SAM high-performance encoder to obtain the comparison pre-training results, the segmentation processing and the comparison pre-training results are calculated by the loss function to obtain the configuration parameters, and the configuration parameters are input into the image encoder (lightweight ViT) for parameter optimization.

[0112] The lightweight encoder significantly reduces the equipment performance requirements while maintaining efficient segmentation capabilities and achieves real-time segmentation. The self-attention and cross-attention mechanisms ensure highly accurate segmentation results. Through multiple rounds of iterative training and MAE pre-training methods, the lightweight encoder's segmentation performance remains stable under complex backgrounds. In summary, EfficientSAM achieves efficient and accurate vascular boundary segmentation in real-time minimally invasive surgery through a lightweight encoder and advanced attention mechanism.

[0113] As an optional embodiment, after S104, a surgery optimization image may be generated and displayed based on the second mask image. The surgery optimization image at least includes a highlight icon for indicating the target vascular structure. Figure 5 In the data processing flow shown, the optimized mask Mi2 (i.e. the second mask image) for edge extraction.

[0114] Specifically, this surgical optimization image should at least include a highlight icon for indicating the target vascular structure to provide clearer visual guidance. Specifically, based on the second mask image, an image processing algorithm can be used to convert it into a surgical optimization image. This process may involve adjusting, enhancing or adding marks to the image to highlight the position and characteristics of the target vascular structure. For example, in the surgical optimization image, at least a highlight icon for indicating the target vascular structure needs to be added. These highlight icons can be marks of specific colors or shapes to help doctors identify the areas that need attention during surgery more quickly and accurately. Here, the purpose of providing a surgical optimization image is to provide clearer and more intuitive visual guidance so that doctors can better understand the structure and characteristics of the surgical area. Therefore, during the generation process, it is necessary to consider how to maximize the visibility and information transfer of the image.

[0115] The resulting surgical optimization images should be able to be displayed in real time on a monitor or display device during the operation so that doctors and surgical teams can view and refer to them at any time, providing more effective visual support and guidance for minimally invasive surgery. With such surgical optimization images, doctors can more accurately locate and treat target vascular structures, improving the success rate and effectiveness of the operation.

[0116] As an optional embodiment, the processing process from the image to be processed to the second mask image is accelerated by inference operation through an edge device.

[0117] In practical applications, it is an effective way to accelerate the inference operation of the processing process from the image to be processed to the second mask image through edge devices, which can improve the efficiency and real-time performance of image processing. Edge devices usually have strong computing power and specialized hardware accelerators, such as GPU, TPU, etc., which can perform efficient inference calculations locally, reduce communication delays with the cloud, and improve real-time performance. Using acceleration algorithms for inference on edge devices, such as lightweight model compression, quantization, pruning, etc., can reduce the amount of calculation and the number of parameters of the model and speed up the inference speed. Decomposing the image processing process into several stages, and executing the calculations of each stage in parallel through pipeline processing, the entire processing process is more efficient and reduces delays. On edge devices, local computing and data transmission can be achieved, reducing dependence on external resources, speeding up processing speed and reducing communication costs. Using hardware accelerators (such as GPU, TPU) on edge devices to accelerate specific computing tasks, such as image filtering, convolution operations, etc., greatly improves processing efficiency. By accelerating inference operations on edge devices, the real-time nature of the image processing process can be guaranteed, ensuring that processing results can be provided in a timely manner in scenarios that require rapid response, such as minimally invasive surgery.

[0118] By accelerating inference operations on edge devices, we can fully utilize the computing resources and hardware accelerators of the device, improve the efficiency and real-time performance of the image processing process, and provide faster and more accurate visual assistance for applications such as minimally invasive surgery.

[0119] In order to adapt to the edge deployment requirements in the operating room and reduce computing overhead, model compression is performed on the staged feature extraction model, global attention model, and vascular boundary segmentation model, which are converted into ONNX models, and the ONNX Runtime framework is used to implement inference operations on various edge devices.

[0120] In addition, by applying the tensorRT high-performance deep learning inference library, the ONNX model is further accelerated, improving the inference speed and data throughput per unit time. These optimization measures not only provide surgeons with high-precision and low-latency real-time visual guidance, but also ensure that key image processing results can be provided in a timely and accurate manner in scenarios that require efficient processing, such as minimally invasive surgery.

[0121] See also Figure 7 As shown, the embodiment of the present application provides a blood vessel identification system. The blood vessel identification system includes at least the following structures:

[0122] Image acquisition module, used to obtain the image P to be processed i ; Further, Figure 7 In the system shown, the image P to be processed i Real-time streaming images from minimally invasive surgery.

[0123] A tracking and positioning module is used to track and locate the image P to be processed based on the image P to be processed. i Perform target detection and tracking to obtain the image P to be processed. i Contains the tracking and positioning information of each vascular structure bbox i ; Further, if Figure 7 The tracking and positioning module shown can be implemented using the Yolov8 and Deepsort network structures introduced above.

[0124] The inference acceleration module is used to accelerate the inference of the tracking and positioning module and the boundary segmentation module. Furthermore, the inference acceleration module can use the ONNXRuntime framework and the tensorRT high-performance deep learning inference library to accelerate the inference operation.

[0125] Boundary segmentation module, used to segment the bbox according to the tracking positioning information i For the image P to be processed i Perform mask decoding and optimization to obtain the mask image M iFurthermore, EfficientSAM can be used to construct.

[0126] A display module is used to display the mask image M i Displays real-time optimized images. For example, Figure 7 In the system shown, the real-time optimized image can be a real-time image of minimally invasive surgery.

[0127] Further optionally, Figure 7 In the blood vessel identification system shown, the specific structure of the tracking and positioning module is as follows Figure 8 As shown. Figure 8 In the tracking and positioning module shown, based on the image to be processed P i Perform stage-by-stage feature extraction to obtain multi-scale first image feature information F corresponding to stages 1 to 5 i ; The first image feature information F i The feature fusion is performed, and the GAM module composed of the channel attention module and the spatial attention module is used to integrate the first image feature information F i The attention of the blood vessel structure is enhanced to obtain the second image feature information F i Based on the second image feature information F i Perform target detection to obtain the positioning feature information bbox corresponding to each vascular structure i ; Use the maximum suppression algorithm to filter out the positioning feature information bbox that meets the preset conditions i , as the target positioning feature information bbox corresponding to the image to be processed i . Simultaneously, in the Deepsort module, the image P to be processed is sorted. i Perform feature extraction to obtain candidate reference positioning feature information bbox i-1 , reference the positioning feature information bbox i-1 After Kalman filtering and result matching processing, it is updated to the reference positioning feature information bbox i Finally, in the Deepsort module, the target positioning feature information bbox i Reference positioning feature information bbox i Perform feature matching to obtain the tracking and positioning result bbox of the image to be processed i .

[0128] The functions of each module in the above-mentioned blood vessel identification system are similar to the relevant descriptions in steps S101 to S104 of the above-mentioned embodiment. The similarities can be found in the above introduction and will not be expanded here.

[0129] In the embodiment of the present application, it is possible to automatically track and locate key blood vessels, and provide accurate boundary segmentation information to provide visual guidance for surgeons, which can effectively enhance the doctor's attention to key blood vessels, focus on key blood vessel boundaries, reduce the impact of adverse factors such as bleeding in the surgical area, anatomical variation, and changes in brightness and darkness of the screen, reduce the incidence of adverse complications such as intraoperative bleeding, improve the efficiency and accuracy of minimally invasive surgery, and reduce the learning curve of young doctors. At the same time, in the embodiment of the present application, a variety of optimization techniques are used to improve the model's visual effect and real-time reasoning speed, which can further accelerate the optimization efficiency of surgical images and help ensure the real-time image optimization effect during surgery. At the same time, in view of the fact that hardware equipment in various operating room environments does not support image optimization, the embodiment of the present application reduces the computing power required for surgical image optimization through the various optimization techniques introduced above, so that the technical solution proposed in this application can be applied to various operating room environments, thereby providing assistance to more doctors. In summary, in the embodiment of the present application, by combining real-time image stream acquisition, multi-scale feature processing, target tracking and positioning, real-time compression coding, mask decoding and other technologies, real-time decision support and visual guidance are provided, which significantly improves the application effect and medical value in complex operations such as minimally invasive surgery, and provides doctors with powerful technical tools and support. In the embodiments of the present application, real-time auxiliary analysis feedback can be provided to doctors during surgery, which helps to provide doctors with reference for intraoperative decision-making, as well as real-time visual guidance and support, and also helps to reduce the difficulty of surgery. In addition, the embodiments of the present application are not limited to specific organs or types of surgery, but also have the potential to be promoted and applied in other minimally invasive surgeries.

[0130] In order to perform the blood vessel identification method corresponding to the above method embodiment, to achieve the corresponding functions and technical effects. Fig. 9 , Fig. 9 The structure block diagram of a blood vessel identification device 700 provided in an embodiment of the present application is shown. For the convenience of description, only the parts related to this embodiment are shown. The blood vessel identification device 700 provided includes:

[0131] An acquisition module, used for acquiring images to be processed; the images to be processed should at least include surgical images;

[0132] A tracking module, used for performing target detection and tracking positioning based on the image to be processed, and obtaining tracking positioning information of each blood vessel structure contained in the image to be processed;

[0133] A segmentation module, used for performing real-time compression encoding on the image to be processed, and performing mask decoding on the compressed encoded image according to the tracking positioning information to obtain a first mask image; wherein the first mask image is used to indicate the pixel-level segmentation result corresponding to the vascular structure in the real-time surgical field, and the first mask image is obtained by optimizing a multiple attention mechanism, and the multiple attention mechanism at least includes a self-attention mechanism and a cross-attention mechanism; the multiple attention mechanism is used to process multi-scale image feature information of each vascular structure on different spatial axes;

[0134] The optimization module is used to perform mask optimization processing based on the first mask image to obtain a second mask image; the second mask image is used to indicate the pixel-level post-processing result corresponding to the target vascular structure in the real-time surgical field of view.

[0135] As an optional embodiment, the tracking module is specifically used to: perform stage-by-stage feature extraction based on the image to be processed to obtain multi-scale first image feature information;

[0136] Performing feature fusion on the first image feature information, and performing attention enhancement on the blood vessel structure in the fused first image feature information to obtain second image feature information;

[0137] Perform feature extraction based on the second image feature information to obtain positioning feature information corresponding to each blood vessel structure;

[0138] Screening out positioning feature information that meets preset conditions as target positioning feature information corresponding to the image to be processed;

[0139] The target positioning feature information is feature matched with the reference positioning feature information to obtain a tracking and positioning result of the image to be processed.

[0140] As an optional embodiment, the tracking module obtains the first image feature information by inputting the image to be processed into a staged feature extraction model; and a channel attention module is added to the backbone network of the staged feature extraction model.

[0141] As an optional embodiment, the tracking module obtains the second image feature information by inputting the first image feature information into a global attention model; wherein the global attention model includes an image fusion module, a channel attention module, and a spatial attention module.

[0142] The tracking module performs attention enhancement on the blood vessel structure in the fused first image feature information to obtain second image feature information, which is specifically used for:

[0143] Using an image fusion module, the multi-size image features in the first image feature information are fused into first reconstructed feature information;

[0144] Using a channel attention module, the first reconstructed feature information is divided into corresponding multi-channel feature blocks, and deep feature extraction and feature mixing are performed on the multi-channel feature blocks respectively to reconstruct the second reconstructed feature information;

[0145] A spatial attention module is used to calculate the bidirectional cross-attention information of the second reconstructed feature information between the parallel axial attentions to reconstruct the second image feature information.

[0146] As an optional embodiment, the tracking module uses a channel attention module to divide the first reconstructed feature information into corresponding multi-channel feature blocks, and performs deep feature extraction and feature mixing on the multi-channel feature blocks respectively to reconstruct the second reconstructed feature information, specifically for:

[0147] Dividing the first reconstructed feature information into corresponding multi-channel feature blocks according to a preset number of channels;

[0148] Each multi-channel feature block is input into the multi-sensory unit to perform feature conversion, and the converted multi-channel feature blocks are subjected to depthwise separable dilated convolution according to the corresponding size to obtain image context features at different sizes;

[0149] Normalizing the image context features at different sizes to obtain normalized image context features;

[0150] The normalized image context features are reorganized and mixed, and the image context features corresponding to different channels are cross-group fused and full-channel fused according to the channel index to obtain the second reconstructed feature information.

[0151] As an optional embodiment, the tracking module uses a spatial attention module to calculate the bidirectional cross attention information of the second reconstructed feature information between the parallel axial attentions to reconstruct the second image feature information, specifically for:

[0152] Performing a multi-scale convolution operation on the second reconstructed feature information in parallel axes, and performing feature aggregation on the operation results to obtain aggregated feature maps on different parallel axes;

[0153] The aggregated feature maps on different parallel axes are respectively matrix-added, and the addition result matrix is ​​input into the fully connected layer of the multi-layer perception unit for feature fusion processing to obtain the feature fusion matrix on different parallel axes;

[0154] Perform matrix dot multiplication on the feature fusion matrices on different parallel axes to obtain the attention feature weight matrix of each image feature point on the parallel axis;

[0155] Perform matrix dot multiplication of the attention feature weight matrix with the aggregated feature maps on different parallel axes to obtain enhanced feature maps on different parallel axes;

[0156] The feature maps enhanced along different parallel axes are subjected to convolution fusion processing to obtain the second image feature information.

[0157] As an optional embodiment, the positioning feature information corresponding to each vascular structure includes: a target positioning frame corresponding to each vascular structure; the tracking module extracts features based on the second image feature information to obtain the positioning feature information corresponding to each vascular structure, which is specifically used for:

[0158] Performing multiple convolution operations on the second image feature information to extract candidate positioning frames corresponding to each vascular structure;

[0159] The step of screening out the positioning feature information that meets the preset conditions as the target positioning feature information corresponding to the image to be processed includes:

[0160] The candidate positioning frames corresponding to the respective vascular structures are traversed, the confidence of each candidate positioning frame is calculated, and the candidate positioning frame whose confidence is not less than a set confidence threshold is selected as the target positioning frame corresponding to the image to be processed.

[0161] As an optional embodiment, the image to be processed includes a plurality of continuous image frames; the reference positioning feature information is predicted based on historical target positioning feature information, and the historical target positioning feature information is the target positioning feature information corresponding to the previous frame of the image to be processed;

[0162] The tracking module performs feature matching on the target positioning feature information and the reference positioning feature information to obtain the tracking positioning result of the image to be processed, which is specifically used for:

[0163] For the current frame image in the image to be processed, the target positioning frame is used to perform feature extraction to obtain a first reference positioning frame;

[0164] Predicting a second reference positioning frame corresponding to the current frame image based on the first reference positioning frame and the historical target positioning frame corresponding to the previous frame image;

[0165] A matching detection is performed on the target positioning frame and the second reference positioning frame to update the tracking positioning result of the current frame image.

[0166] As an optional embodiment, the tracking module performs matching detection on the target positioning frame and the second reference positioning frame to update the tracking positioning result of the current frame image, specifically for:

[0167] Calculating the Mahalanobis distance between the target positioning frame and the second reference positioning frame;

[0168] If the calculated Mahalanobis distance is not greater than the set distance threshold, determining that the target positioning frame and the second reference positioning frame match the same detection target, and setting the target positioning frame as the tracking positioning result of the image to be processed;

[0169] If the calculated Mahalanobis distance is greater than the set distance threshold, it is determined that the target positioning frame and the second reference positioning frame do not belong to the same detection target, and the target positioning frame is updated and set to the newly added tracking positioning result in the image to be processed.

[0170] As an optional embodiment, the segmentation module performs real-time compression encoding on the image to be processed, and performs mask decoding on the compressed encoded image according to the tracking positioning information to obtain a first mask image, which is specifically used for:

[0171] Inputting the image to be processed into a pre-trained lightweight encoder for real-time compression encoding to obtain the compressed encoded image; wherein the lightweight encoder is pre-trained synchronously with the deep visual converter on the same image training set and is obtained by using fewer training parameters;

[0172] Mapping the target positioning frame corresponding to each blood vessel structure to the compressed coded image to obtain the image mapping feature of the compressed coded image;

[0173] A self-attention mechanism is used to update the association between different target positioning frames in the image mapping features, and a cross-attention module is used to update the interaction between the mapping feature information and the target positioning frames in the image mapping features to improve segmentation accuracy;

[0174] The updated image mapping features are input into a multi-layer perceptron and a dynamic linear classifier to predict and obtain the first mask image.

[0175] As an optional embodiment, the optimization module performs mask optimization processing based on the first mask image to obtain the second mask image, which is specifically used to:

[0176] Performing filtering processing on pixels in the mask area of ​​the first mask image to obtain a first mask image after filtering processing; the filtering granularity in the filtering processing is dynamically set based on the observation requirements of minimally invasive surgery;

[0177] Performing bilateral filtering on the mask edge region in the first mask image after filtering to obtain a grayscale mask image;

[0178] Perform multi-layer edge detection on the grayscale mask image to obtain the second mask image.

[0179] As an optional embodiment, the optimization module performs bilateral filtering on the mask edge region in the first mask image after filtering to obtain a grayscale mask image, which is specifically used for:

[0180] For the first mask image after filtering, the spatial proximity between each pixel point in the mask area and the central pixel point is calculated to obtain the spatial proximity corresponding to each pixel point in the mask area;

[0181] Calculate the similarity of pixel values ​​of pixels in each mask area;

[0182] Based on the product of spatial proximity and pixel value similarity, the weight value of each mask area pixel is adjusted, and a convolution operation is performed on the adjusted mask area pixel to obtain a grayscale mask image after mask edge smoothing.

[0183] As an optional embodiment, it also includes a display module, which is used to generate and display a surgical optimization image based on the second mask image after the segmentation module performs mask optimization processing based on the first mask image to obtain a second mask image; wherein the surgical optimization image at least includes a highlight icon for indicating the target vascular structure.

[0184] As an optional embodiment, it also includes an inference module, which is used to accelerate the inference operation of the processing process from the image to be processed to the second mask image through an edge device.

[0185] The above-mentioned blood vessel identification device 700 can implement the blood vessel identification method of the above-mentioned method embodiment. The options in the above-mentioned method embodiment are also applicable to this embodiment and will not be described in detail here. The rest of the contents of the embodiment of this application can refer to the contents of the above-mentioned method embodiment and will not be described in detail in this embodiment.

[0186] In order to perform the blood vessel identification method corresponding to the above method embodiment, to achieve the corresponding functions and technical effects. Fig.10 , Fig.10 The structure block diagram of a blood vessel identification device 800 provided in an embodiment of the present application is shown. For the convenience of description, only the parts related to the present embodiment are shown. The blood vessel identification device 800 provided in an embodiment of the present application includes at least a processor 801, a memory 802, and a communication interface 803.

[0187] The above-mentioned blood vessel identification device can implement the blood vessel identification method of the above-mentioned method embodiment. The options in the above-mentioned method embodiment are also applicable to this embodiment and will not be described in detail here. The rest of the contents of the embodiment of this application can refer to the contents of the above-mentioned method embodiment and will not be described in detail in this embodiment.

[0188] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a storage controller, the steps in any of the above method embodiments are implemented.

[0189] An embodiment of the present application provides a computer program product. When the computer program product runs on a storage controller, the storage controller implements the steps in the above-mentioned method embodiments when executing.

[0190] In several embodiments provided in the present application, it is understood that each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved.

[0191] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0192] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A blood vessel identification method, characterized in that: The method comprises: Acquire an image to be processed; the image to be processed includes at least a surgical image; the surgical image includes a vascular structure, and the vascular structure exists in multiple spatial axes including horizontal, vertical and depth; Performing target detection and tracking and positioning based on the image to be processed to obtain tracking and positioning information of each blood vessel structure contained in the image to be processed; The image to be processed is compressed and encoded in real time, and the compressed and encoded image is mask-decoded according to the tracking and positioning information to obtain a first mask image; wherein the first mask image is used to indicate the pixel-level segmentation result corresponding to the vascular structure in the real-time surgical field, and the pixel-level segmentation result is used to indicate the probability that each pixel point in the image belongs to the vascular structure, and the first mask image is obtained by optimizing a multiple attention mechanism, and the multiple attention mechanism at least includes a self-attention mechanism and a cross-attention mechanism; the multiple attention mechanism is used to process the multi-scale image feature information of each vascular structure in different spatial axes, the self-attention mechanism is used to establish the association between pixels in each spatial axis, and the cross-attention mechanism is used for information interaction and fusion between different spatial axes; Mask optimization processing is performed based on the first mask image to obtain a second mask image; the second mask image is used to indicate a pixel-level post-processing result corresponding to the target vascular structure in the real-time surgical field of view.

2. The blood vessel identification method according to claim 1, characterized in that: The target detection and tracking and positioning are performed based on the image to be processed to obtain the tracking and positioning information of each blood vessel structure in the image to be processed, including: Performing staged feature extraction based on the image to be processed to obtain multi-scale first image feature information; Performing feature fusion on the first image feature information, and performing attention enhancement on the blood vessel structure in the fused first image feature information to obtain second image feature information; Perform feature extraction based on the second image feature information to obtain positioning feature information corresponding to each blood vessel structure; Screening out positioning feature information that meets preset conditions as target positioning feature information corresponding to the image to be processed; The target positioning feature information is feature matched with the reference positioning feature information to obtain a tracking and positioning result of the image to be processed.

3. The blood vessel identification method according to claim 2, characterized in that: The method obtains the first image feature information by inputting the image to be processed into a staged feature extraction model; A channel attention module is added to the backbone network of the staged feature extraction model.

4. The blood vessel identification method according to claim 2, characterized in that: The method obtains the second image feature information by inputting the first image feature information into a global attention model; The global attention model includes an image fusion module, a channel attention module, and a spatial attention module; the step of enhancing the attention of the vascular structure in the fused first image feature information to obtain the second image feature information includes: Using an image fusion module, the multi-size image features in the first image feature information are fused into first reconstructed feature information; Using a channel attention module, the first reconstructed feature information is divided into corresponding multi-channel feature blocks, and deep feature extraction and feature mixing are performed on the multi-channel feature blocks respectively to reconstruct the second reconstructed feature information; A spatial attention module is used to calculate the bidirectional cross-attention information of the second reconstructed feature information between the parallel axial attentions to reconstruct the second image feature information.

5. The blood vessel identification method according to claim 4, characterized in that: The method of using a channel attention module to divide the first reconstructed feature information into corresponding multi-channel feature blocks, and performing deep feature extraction and feature mixing on the multi-channel feature blocks respectively to reconstruct the second reconstructed feature information includes: Dividing the first reconstructed feature information into corresponding multi-channel feature blocks according to a preset number of channels; Each multi-channel feature block is input into the multi-sensory unit to perform feature conversion, and the converted multi-channel feature blocks are subjected to depthwise separable dilated convolution according to the corresponding size to obtain image context features at different sizes; Normalizing the image context features at different sizes to obtain normalized image context features; The normalized image context features are reorganized and mixed, and the image context features corresponding to different channels are cross-group fused and full-channel fused according to the channel index to obtain the second reconstructed feature information.

6. The blood vessel identification method according to claim 4, characterized in that: The adopting of the spatial attention module to calculate the bidirectional cross attention information of the second reconstructed feature information between the parallel axial attentions to reconstruct the second image feature information includes: Performing a multi-scale convolution operation on the second reconstructed feature information in parallel axes, and performing feature aggregation on the operation results to obtain aggregated feature maps on different parallel axes; The aggregated feature maps on different parallel axes are respectively matrix-added, and the addition result matrix is ​​input into the fully connected layer of the multi-layer perception unit for feature fusion processing to obtain the feature fusion matrix on different parallel axes; Perform matrix dot multiplication on the feature fusion matrices on different parallel axes to obtain the attention feature weight matrix of each image feature point on the parallel axis; Perform matrix dot multiplication of the attention feature weight matrix with the aggregated feature maps on different parallel axes to obtain enhanced feature maps on different parallel axes; The feature maps enhanced along different parallel axes are subjected to convolution fusion processing to obtain the second image feature information.

7. The blood vessel identification method according to claim 2, characterized in that: The positioning feature information corresponding to each vascular structure includes: a target positioning frame corresponding to each vascular structure; The extracting features based on the second image feature information to obtain the positioning feature information corresponding to each blood vessel structure includes: Performing multiple convolution operations on the second image feature information to extract candidate positioning frames corresponding to each vascular structure; The step of screening out the positioning feature information that meets the preset conditions as the target positioning feature information corresponding to the image to be processed includes: The candidate positioning frames corresponding to the respective vascular structures are traversed, the confidence of each candidate positioning frame is calculated, and the candidate positioning frame whose confidence is not less than a set confidence threshold is selected as the target positioning frame corresponding to the image to be processed.

8. The blood vessel identification method according to claim 7, characterized in that: The image to be processed includes a plurality of continuous image frames; the reference positioning feature information is predicted based on historical target positioning feature information, and the historical target positioning feature information is the target positioning feature information corresponding to the previous frame of the image to be processed; The step of performing feature matching on the target positioning feature information and the reference positioning feature information to obtain the tracking and positioning result of the image to be processed includes: For the current frame image in the image to be processed, the target positioning frame is used to perform feature extraction to obtain a first reference positioning frame; Predicting a second reference positioning frame corresponding to the current frame image based on the first reference positioning frame and the historical target positioning frame corresponding to the previous frame image; A matching detection is performed on the target positioning frame and the second reference positioning frame to update the tracking positioning result of the current frame image.

9. The blood vessel identification method according to claim 8, characterized in that: The matching detection of the target positioning frame and the second reference positioning frame to update the tracking positioning result of the current frame image includes: Calculating the Mahalanobis distance between the target positioning frame and the second reference positioning frame; If the calculated Mahalanobis distance is not greater than the set distance threshold, determining that the target positioning frame and the second reference positioning frame match the same detection target, and setting the target positioning frame as the tracking positioning result of the image to be processed; If the calculated Mahalanobis distance is greater than the set distance threshold, it is determined that the target positioning frame and the second reference positioning frame do not belong to the same detection target, and the target positioning frame is updated and set to the newly added tracking positioning result in the image to be processed.

10. The blood vessel identification method according to claim 9, characterized in that: After calculating the Mahalanobis distance between the target positioning frame and the second reference positioning frame, the method further includes: By using the re-identification model, feature extraction is performed on the target positioning frame and the second reference positioning frame to obtain a first feature vector corresponding to the target positioning frame and a second feature vector corresponding to the second reference positioning frame; Inputting the first feature vector and the second feature vector into a cosine distance cost function to calculate an appearance feature cosine distance between the first feature vector and the second feature vector; If the calculated appearance feature cosine distance is not greater than the set appearance feature cosine distance threshold, and the calculated Mahalanobis distance is not greater than the set distance threshold, it is determined that the target positioning frame matches the second reference positioning frame as the same detection target, and the target positioning frame is set as the tracking positioning result of the image to be processed.

11. The blood vessel identification method according to claim 1, characterized in that: The real-time compression encoding of the image to be processed and mask decoding of the compressed encoded image according to the tracking and positioning information to obtain a first mask image include: Inputting the image to be processed into a pre-trained lightweight encoder for real-time compression encoding to obtain the compressed encoded image; wherein the lightweight encoder is pre-trained synchronously with the deep visual converter on the same image training set and is obtained by using fewer training parameters; Mapping the target positioning frame corresponding to each blood vessel structure to the compressed coded image to obtain the image mapping feature of the compressed coded image; A self-attention mechanism is used to update the association between different target positioning frames in the image mapping features, and a cross-attention module is used to update the interaction between the mapping feature information and the target positioning frames in the image mapping features to improve segmentation accuracy; The updated image mapping features are input into a multi-layer perceptron and a dynamic linear classifier to predict and obtain the first mask image.

12. The blood vessel identification method according to claim 1, characterized in that: The performing mask optimization processing based on the first mask image to obtain a second mask image includes: Performing filtering processing on pixels in the mask area of ​​the first mask image to obtain a first mask image after filtering processing; the filtering granularity in the filtering processing is dynamically set based on the observation requirements of minimally invasive surgery; Performing bilateral filtering on the mask edge region in the first mask image after filtering to obtain a grayscale mask image; Perform multi-layer edge detection on the grayscale mask image to obtain the second mask image.

13. The blood vessel identification method according to claim 12, characterized in that: The step of performing bilateral filtering on the mask edge region in the first mask image after filtering to obtain a grayscale mask image includes: For the first mask image after filtering, the spatial proximity between each pixel point in the mask area and the central pixel point is calculated to obtain the spatial proximity corresponding to each pixel point in the mask area; Calculate the similarity of pixel values ​​of pixels in each mask area; Based on the product of spatial proximity and pixel value similarity, the weight value of each mask area pixel is adjusted, and a convolution operation is performed on the adjusted mask area pixel to obtain a grayscale mask image after mask edge smoothing.

14. The blood vessel identification method according to claim 1, characterized in that: After performing mask optimization processing based on the first mask image to obtain a second mask image, the method further includes: Based on the second mask image, a surgery optimization image is generated and displayed; wherein the surgery optimization image at least includes a highlight icon for indicating the target vascular structure.

15. The blood vessel identification method according to claim 1, characterized in that: The method also accelerates the processing from the image to be processed to the second mask image by performing inference operations through an edge device.

16. A blood vessel identification device, characterized in that: The device comprises: An acquisition module, used for acquiring images to be processed; the images to be processed include at least surgical images; the surgical images include vascular structures, and the vascular structures exist in multiple spatial axes including horizontal, vertical and depth; A tracking module, used for performing target detection and tracking positioning based on the image to be processed, and obtaining tracking positioning information of each blood vessel structure contained in the image to be processed; A segmentation module, used for performing real-time compression encoding on the image to be processed, and performing mask decoding on the compressed encoded image according to the tracking positioning information to obtain a first mask image; wherein the first mask image is used to indicate the pixel-level segmentation result corresponding to the vascular structure in the real-time surgical field, and the pixel-level segmentation result is used to indicate the probability that each pixel point in the image belongs to the vascular structure, and the first mask image is obtained by optimizing a multiple attention mechanism, and the multiple attention mechanism at least includes a self-attention mechanism and a cross-attention mechanism; the multiple attention mechanism is used to process multi-scale image feature information of each vascular structure in different spatial axes, the self-attention mechanism is used to establish associations between pixels in each spatial axis, and the cross-attention mechanism is used for information interaction and fusion between different spatial axes; The optimization module is used to perform mask optimization processing based on the first mask image to obtain a second mask image; the second mask image is used to indicate the pixel-level post-processing result corresponding to the target vascular structure in the real-time surgical field of view.

17. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor is used to read and execute the computer software program, thereby implementing the blood vessel identification method according to any one of claims 1 to 15.

18. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores a computer software program, and when the computer software program is executed by the processor, the blood vessel identification method according to any one of claims 1 to 15 is implemented.

Images