Intraoperative blood vessel real-time automatic identification method and system based on deep neural network learning

Abstract

The invention belongs to the technical field of medical image processing and computer vision, discloses an intraoperative blood vessel real-time automatic identification method and system based on deep neural network learning, and is particularly suitable for automatic identification of submesenteric arteries and submesenteric veins in laparoscopic rectal resection. The intraoperative blood vessel recognition method comprises the following steps: acquiring a laparoscopic surgery video, extracting a frame sequence containing a target blood vessel, and marking an image to construct a training data set; using a 3D U-Net segmentation neural network training model to pre-process a video frame, and inputting the pre-processed video frame to a three-dimensional convolutional neural network model for semantic segmentation; and running the model in a test video in real time, outputting a pixel-level prediction result of the target blood vessel, and displaying the result in an original video in a visual overlapping manner to assist surgical navigation. According to the method, accurate recognition and real-time visual navigation of the blood vessel structure are achieved, a surgeon can be effectively assisted in reducing the risk of blood vessel misjudgment in the operation, and the safety and standardization level of the minimally invasive operation are improved.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer vision technology, specifically to a real-time recognition and visualization system and method for vascular structures in surgical videos based on deep neural network learning, which is particularly suitable for the automatic recognition of inferior mesenteric artery and inferior mesenteric vein in laparoscopic rectal resection. Background Technology

[0002] In laparoscopic rectal resection, the inferior mesenteric artery (IMA) and inferior mesenteric vein (IMV) are among the most critical anatomical landmarks, and their accurate identification directly affects surgical safety, bleeding risk, and the quality of lymph node dissection. However, current surgical methods mainly rely on the surgeon's experience for visual identification, which presents the following problems: the intraoperative field of vision is complex, and blood vessels are easily obscured by fat, smoke, and instruments; there are significant differences in anatomical structures among different patients, resulting in a steep learning curve; and misjudgment can easily lead to severe intraoperative bleeding.

[0003] In existing technologies, some methods rely on preoperative three-dimensional CT angiography (3D-CTA), which is an invasive examination, cannot be dynamically updated in real time, and cannot be directly applied to intraoperative real-time navigation. Some studies have attempted to use deep learning for surgical instrument or organ recognition, but a complete system solution for real-time automatic recognition of IMA and IMV in laparoscopic rectal surgery is still lacking.

[0004] Therefore, there is an urgent need for a real-time, non-invasive, high-precision, automated method and system for vascular identification that can be integrated into surgical platforms. Summary of the Invention

[0005] This invention provides a method and system for real-time automatic intraoperative blood vessel identification based on deep neural network learning, in order to solve the problem of low accuracy in identifying key anatomical structures during surgery due to factors such as insufficient operational experience, complex surgical field, large anatomical variations, and instrument obstruction in the prior art.

[0006] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or to describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0007] According to a first aspect of the present invention, a method and system for real-time automatic identification of intraoperative blood vessels based on deep neural network learning are provided.

[0008] In one embodiment, the method and system for real-time automatic intraoperative vessel identification based on deep neural network learning includes the following steps:

[0009] After preprocessing the surgical video clips, boundary annotation operations were performed on key anatomical targets such as blood vessels. The annotation results were then combined with a 3D U-Net neural network structure to train and generate a key anatomical target region recognition model.

[0010] Based on surgical image data and medical anatomical feature fingerprints, three-dimensional image segmentation and real-time inference output models are constructed respectively.

[0011] An intraoperative visualization navigation platform is built by combining key anatomical structure region recognition models, 3D image segmentation processing and real-time reasoning output models with augmented reality technology.

[0012] In one embodiment, performing boundary annotation of key anatomical targets on surgical video clips and combining the annotation results with a 3D U-Net neural network structure to train and generate a key anatomical target region recognition model includes the following steps:

[0013] Video clips exposing the key blood vessels, the inferior mesenteric artery (IMA) and inferior mesenteric vein (IMV), were extracted and segmented into static images at a fixed frame rate.

[0014] Boundary delineation is performed on IMA and IMV in static images, and reverse tracking technology and occlusion removal logic are used to obtain the gold standard for annotation.

[0015] The annotation results are input into a 3D U-Net network, and a semantic segmentation model capable of recognizing multi-anatomical targets is generated by configuring hyperparameters based on the nnU-Net v2 framework.

[0016] In one embodiment, the preprocessing operation includes performing extreme value clipping and pixel value normalization on the RGB three channels, wherein the clipping intervals for the R channel, G channel and B channel are (72, 255), (28, 244) and (23, 231), respectively.

[0017] In one embodiment, constructing a three-dimensional image segmentation and real-time inference output model based on surgical image data and medical anatomical fingerprints includes the following steps:

[0018] The model's precision, recall, and Dice coefficient for different anatomical targets were validated using 5-fold cross-validation.

[0019] The inference pipeline is optimized based on the trained model parameters to achieve real-time recognition performance with a processing time of no more than 0.08 seconds per static image.

[0020] The Mamba architecture's States-Space model is introduced to capture long-range spatiotemporal features in operational videos, enhancing the robustness of identifying meandering blood vessels and neural structures.

[0021] In one embodiment, obtaining the corresponding anatomical features of surgical images using annotation methods includes the following steps:

[0022] The consistency of the annotations was verified by independently annotating the data by two senior surgeons and calculating the Cohen's Kappa coefficient.

[0023] For IMA and IMV, feature extraction is performed by tracing back the video stream from the vascular transection scene to its anatomical root.

[0024] Excluding areas obscured by surgical instruments, pixel-level marking is performed only on effectively exposed blood vessels, nerves, and fascia tissue.

[0025] In one embodiment, capturing three-dimensional spatial information and constructing an image processing model based on a 3D U-Net structure includes the following steps:

[0026] Define a U-Net network with a sampling depth of 9 layers, with a maximum configuration of 512 feature maps per layer, to capture deep anatomical semantic information;

[0027] Hybrid loss functions are defined during network training, where Dice Loss is used to optimize global region matching accuracy and Cross-Entropy Loss is used to enhance local edge pixel accuracy.

[0028] In one embodiment, setting up an intraoperative visual navigation platform includes the following steps:

[0029] The key anatomical target area recognition model is integrated into the edge computing workstation to achieve the overlay rendering of the recognition mask and real-time surgical video;

[0030] Develop a results display interface that combines virtual reality or augmented reality technologies, and integrate a distance warning function between the instrument and the target structure;

[0031] The system acquires real-time intraoperative identification results and highlights the location and outline of key anatomical structures such as IMA and IMV on the display terminal.

[0032] According to a second aspect of the present invention, a method and system for real-time automatic identification of intraoperative blood vessels based on deep neural network learning is provided.

[0033] In one embodiment, the automatic identification system for key anatomical structures includes:

[0034] The image recognition unit is used to preprocess and delineate regions in surgical videos, and generate key anatomical structure target recognition models.

[0035] Real-time inference evaluation unit, used to build high-performance real-time recognition and output models based on anatomical feature fingerprints and neural network architecture;

[0036] The visual navigation platform unit is used to combine recognition models with augmented reality technology to build a navigation and risk warning platform in the technology.

[0037] According to a third aspect of the present invention, a computer device is provided.

[0038] In some embodiments, the computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described above.

[0039] According to a fourth aspect of the present invention, a computer-readable storage medium is provided.

[0040] In one embodiment, a computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the above method.

[0041] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0042] This invention uses a 3D U-Net network and precise boundary annotation to identify key anatomical targets such as IMA and IMV in laparoscopic rectal resection in real time and accurately, reducing the risk of accidental bleeding and injury due to anatomical variations or cognitive errors. Its high Dice coefficient verifies the high accuracy of identification.

[0043] This invention achieves an inference speed of 12.7 fps by optimizing the algorithm architecture, meeting the needs of real-time intraoperative navigation. Combined with augmented reality technology, it provides surgeons with intuitive anatomical boundary references, which helps to shorten the learning curve and improve surgical safety and standardization. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0045] Figure 1 This is a flowchart illustrating an intraoperative real-time automatic blood vessel identification method and system based on deep neural network learning, according to an exemplary embodiment.

[0046] Figure 2This is a block diagram illustrating the principle of a method and system for real-time automatic identification of intraoperative blood vessels based on deep neural network learning, according to an exemplary embodiment.

[0047] Figure 3 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0048] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some embodiments may include or substitute parts and features of other embodiments. The scope of the embodiments herein encompasses the entire scope of the claims and all available equivalents thereof. Throughout this document, the terms “first,” “second,” etc., are used only to distinguish one element from another without requiring or implying any actual relationship or order between the elements. Indeed, a first element can also be referred to as a second element, and vice versa. Furthermore, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a structure, apparatus, or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a structure, apparatus, or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the structure, apparatus, or device that includes said element. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0049] The terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" used in this document to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings. They are used solely for the convenience of describing the document and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In the description herein, unless otherwise specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two elements; they can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0050] In this document, unless otherwise stated, the term "multiple" means two or more.

[0051] In this article, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0052] In this article, the term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0053] It should be understood that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0054] The modules in the apparatus or system of this application can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0055] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0056] Figure 1 An embodiment of the intraoperative automatic blood vessel identification method based on deep neural network learning of the present invention is shown.

[0057] In this optional embodiment, the intraoperative real-time automatic blood vessel identification method and system based on deep neural network learning includes the following steps:

[0058] Step S101: Obtain laparoscopic surgery video, extract frame sequences containing target blood vessels, and annotate images to construct a training dataset;

[0059] Step S103: Use 3D U-Net segmentation neural network to train the model, preprocess the video frames and input them into the three-dimensional convolutional neural network model for semantic segmentation;

[0060] Step S105: Run the model in real time in the test video, output the pixel-level prediction results of the target blood vessel, and display the results in a visual manner overlaid on the original video to assist surgical navigation.

[0061] In this optional embodiment, the data acquisition and preprocessing of the intraoperative real-time automatic blood vessel identification method and system based on deep neural network learning includes the following steps:

[0062] We acquired high-definition surgical videos of laparoscopic rectal resections certified by professional physicians and de-anonymized the videos to complete the raw data acquisition. For specific surgical stages where the IMA and IMV were fully exposed (such as the retroperitoneal space anatomy stage), static images were extracted at a frequency of 1 fps (1 frame per second). Using an image annotation tool (such as LabelMe), two or more experienced surgeons performed pixel-level contour annotations on the IMA and IMV regions in the images, and the background was set as an invalid area. All annotation discrepancies were resolved by both parties through review of the original video and discussion with reference to anatomical atlases, and a consensus was reached to form the final gold standard annotation.

[0063] In this optional embodiment, the system also embodies the depth annotation logic of multi-target anatomical structures in extensive laparoscopic surgery. Accurate annotation logic is the core of achieving "multi-target recognition." This invention has developed a rigorous logical system for blood vessels and extended anatomical structures, specifically including:

[0064] For the tracking and annotation logic of vascular targets (IMA and IMV), this invention adopts a reverse video tracking method. That is, the annotator first locates the moment when the root of the IMA is cut off, at which time the vascular anatomy is clearest. Then, starting from this scene, the surgical video is played in reverse to track the direction of the vascular pedicle. The annotation area extends from the dissected area around the beginning of the blood vessel to the undissected area on the outer side. For the IMV, the annotator starts tracking from the moment it is initially exposed and continues until the complete cutting stage.

[0065] In laparoscopic surgery, instrument occlusion is the main factor affecting recognition accuracy. Faced with the handling of occlusion and exposure in complex environments, the annotation logic of this invention is clear: the blood vessel parts occluded by surgical instruments are not labeled, and only the remaining exposed areas after excluding occlusion are labeled. In this way, the trained model can learn to identify "partially exposed" IMA and IMV in complex scenarios, which is crucial for improving the reliability of the model in real surgical environments.

[0066] Based on the system requirement of "multi-target recognition", this invention can further expand and evolve the annotation scope. For example, for neural targets, the pelvic autonomic nerve plexus will be annotated. By observing the spatial relationship between the nerve and the IMA root, pixel-level marking will be performed using its unique gray-white fibrous texture. For fascial layers, the anatomical plane and the "sacred plane" behind the peritoneum will be marked. At this time, the annotator will define the fascial boundary by identifying the fracture surface of loose connective tissue (i.e., the "bloodless plane"). This logic enables the model to not only identify "point" vessels, but also "surface" anatomical layers, thereby achieving full-process surgical navigation support. In addition, the identification of other important blood vessels, organs, ureters, and tumor bodies, as well as surgical instruments, in laparoscopic surgery can be completed through their own structural characteristics and similar logic.

[0067] In this optional embodiment, the system deep learning model construction steps (taking the nnU-Net v2 framework as an example) include:

[0068] Architecture selection: 3D U-Net is adopted as the basic backbone network, whose encoder-decoder structure can effectively capture the semantic features and spatial anatomical relationships of images; Automatic configuration strategy: Utilizing the self-configuration feature of the nnU-Net framework, it is automatically set according to the characteristics of the input dataset; Image preprocessing: Including grayscale normalization and resampling; Hyperparameter settings: Sliding window size, batch size, and initial learning rate; Data augmentation: Random rotation, scaling, noise addition, and color shift are applied in real time during training to improve the robustness of the model under different light sources and smoke conditions.

[0069] In this optional embodiment, the 3D U-Net convolutional network architecture and hyperparameter configuration of the present invention are based on the nnU-Netv2 framework. By extracting the fingerprint of the training data (including image modality, pixel value distribution, pixel spacing, and median resolution, etc.), an optimal set of hyperparameter configurations for the laparoscopic environment is calculated, specifically including:

[0070] Regarding network depth and convolution kernel features, the 3D U-Net used in this embodiment has a 9-layer sampling depth, and the convolution kernel dimension is uniformly set to 2D / 3*3. Each layer performs 2 convolution operations. This depth configuration ensures that the network can extract the color and texture features of blood vessels from the shallow layer and capture complex spatial anatomical relationships from the deep layer.

[0071] Regarding the stepwise growth of feature maps, in order to balance computational efficiency and feature extraction capability, the number of feature maps in each layer of the network is configured in a stepwise increasing manner, with the specific sequence being: 32, 64, 128, 256, 512, 512, 512, 512, 512. In the middle and deeper layers of the network (after the 5th layer), the number of feature maps is maintained at a peak of 512 to ensure that the model can still maintain robust recognition capability of IMA and IMV when the laparoscopic field of view is deformed or obstructed.

[0072] Regarding image preprocessing and prediction sliding window configuration, this embodiment performs strict channel-level preprocessing on the input data. For extreme value cropping, the RGB three channels are cropped at the 95th percentile, with the R channel ranging from (72, 255), the G channel from (28, 244), and the B channel from (23, 231).

[0073] For normalization, normalization is performed based on the mean and variance of each channel. For example, the mean of the R channel is 163.3 and the variance is 40.7, and the prediction slider size is set to (896, 1792).

[0074] This large sliding box allows the model to cover a wider surgical field of view during inference, thereby utilizing surrounding tissues (such as nerve plexuses and fascia) as contextual information to assist in the identification of blood vessels.

[0075] In this optional embodiment, the system model training and optimization method includes:

[0076] The semantic segmentation task is one of the following: three-class segmentation: artery, vein, and background; or two-class segmentation: artery and background, or vein and background.

[0077] A weighted combination of Dice Loss and Cross-Entropy Loss is used as the hybrid loss function. Dice Loss addresses the problem of class imbalance (blood vessels occupy a small area in the entire image), while Cross-Entropy Loss ensures pixel-level prediction accuracy. A 5-fold cross-validation strategy is used as the validation mechanism. Dice coefficient, recall, and precision are used to evaluate the accuracy of the blood vessel prediction model. The Dice coefficient reflects the degree of overlap between the model segmentation result and the actual result. 100% of this index represents perfect segmentation. Experimental data shows that the highest Dice coefficient of IMA in this embodiment reaches 0.940, and the IMV reaches 0.980.

[0078] In this optional embodiment, the present invention employs a refined combination of loss functions during the training process of the deep learning recognition engine, aiming to solve the common problems of class imbalance and boundary ambiguity in medical image segmentation. This demonstrates the mathematical implementation details of the algorithm model and loss functions, including:

[0079] This embodiment first introduces Dice Loss as the global optimization objective. Dice Loss originates from the Dice coefficient, and its physical meaning is to measure the degree of overlap between the model's predicted region and the manually labeled gold standard region. Its mathematical expression is as follows:

[0080]

[0081] In the formula, TP (true positive) represents the number of blood vessel pixels correctly identified by the model, FP (false positive) represents the background pixels that are falsely reported as blood vessels, and FN (false negative) represents the blood vessel pixels that are missed. This function optimizes from the global region level and is particularly suitable for targets such as blood vessels that occupy a very small proportion of the whole image. It can effectively prevent the model from getting stuck in the local optimum of predicting the whole background.

[0082] To further improve the clarity of blood vessel boundaries, this invention introduces cross-entropy loss for local pixel calibration. This method penalizes the classification accuracy of each pixel at the local pixel level, and the calculation formula is as follows:

[0083]

[0084] In the formula, For pixels The true label, To predict the probability that the pixel belongs to the target for the model. The introduction of this method forces the model to focus on classification details at the boundaries, which helps to generate more accurate anatomical contours.

[0085] Finally, this invention combines the two with an equal weight ratio of 0.5 to construct a hybrid loss function (TotalLoss), as shown in the following formula:

[0086]

[0087] This fusion strategy combines the advantages of region-level and pixel-level metrics, ensuring that the model maintains high overlap while possessing extremely high boundary recognition accuracy.

[0088] In this optional embodiment, the real-time inference and visualization system is implemented in the following ways:

[0089] The system captures video signals from the laparoscopic host in real time to analyze the video stream; the trained weight model is deployed in a computing unit equipped with a GPU (such as the NVIDIA GeForce RTX series), and by optimizing the inference pipeline, the processing speed reaches 12.7 fps, with a frame latency controlled within 0.08 seconds, achieving clinically imperceptible latency and thus completing high-performance inference; the system overlays the identified vascular contours onto the original surgical image with a semi-transparent color layer (such as red for IMA and blue for IMV), and feeds the results back to the surgical monitor in real time to achieve graphic overlay output.

[0090] In this optional embodiment, the clinical performance evaluation was conducted by 10 senior gastrointestinal surgeons in a blind review using a five-point Likert scale, scoring on three dimensions: “accuracy of vascular localization,” “boundary fit,” and “real-time interaction.” The results showed that the average scores for all three questions were in the highest score range, indicating that the surgeons had a positive assessment of the predictive performance of the developed model and verifying the application value of the invention in actual surgical scenarios.

[0091] Figure 2 This paper illustrates an embodiment of the intraoperative real-time automatic blood vessel identification method and system based on deep neural network learning according to the present invention.

[0092] In this optional embodiment, the intraoperative real-time automatic blood vessel identification method and system based on deep neural network learning includes:

[0093] Image acquisition module 201: used to acquire high-definition video streams of laparoscopic surgery in real time and extract specific scene frames from the video for annotation;

[0094] Preprocessing module 203: Normalizes, crops out extreme values, and enhances anatomical features of the image;

[0095] Core recognition engine module 205: Employs deep neural network learning algorithms (such as 3D U-Net) to automatically recognize pixel-level contours of key anatomical structures in surgical scenes;

[0096] Real-time visualization module 207: The identified blood vessel areas are superimposed on the original surgical image with preset color layers and output to the display terminal in real time.

[0097] In one embodiment, a computer device is provided, which may be an edge computing workstation, a surgical robot control host, or a remote auxiliary diagnostic server deployed in an operating room, and its internal structure diagram may be as follows: Figure 3As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities, particularly real-time inference and multi-target segmentation control of the laparoscopic surgical video stream. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The database stores static information (such as preset nnU-Net neural network hyperparameters, anatomical fingerprints of blood vessels and nerves) and dynamic information data (such as real-time acquired intraoperative images, recognition mask data, and surgeon scoring feedback). The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface communicates with external terminals (such as laparoscopic mainframes, AR head-mounted displays, and cloud-based medical data centers) via a network connection. When executed by the processor, the computer program performs the steps in the above method embodiments, including video preprocessing, multi-target recognition based on 3D U-Net or Mamba architecture, and visual navigation overlay.

[0098] Those skilled in the art will understand that Figure 3 The structures shown are merely block diagrams of some structures related to the present invention and do not constitute a limitation on the computer devices on which the present invention is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. For example, in actual clinical applications, the device may also include a dedicated hardware acceleration card (such as a GPU or TPU) to ensure that the inference speed meets the real-time performance requirement of 12.7 fps or higher.

[0099] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0100] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0101] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0102] This invention is not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this invention is limited only by the appended claims.

Claims

1. A method for real-time automatic intraoperative vessel identification based on deep neural network learning, characterized in that, include: Image acquisition module: used to acquire high-definition video streams of laparoscopic surgery in real time and extract specific scene frames from the video for annotation; Preprocessing module: Normalizes the image, performs extreme value cropping, and enhances anatomical features; Core recognition engine module: Employs deep neural network learning algorithms (such as 3D U-Net) to automatically recognize pixel-level contours of key anatomical structures in surgical scenes; Real-time visualization module: The identified blood vessel areas are overlaid on the original surgical image with preset color layers and output to the display terminal in real time.

2. The method for real-time automatic intraoperative vessel identification based on deep neural network learning according to claim 1, characterized in that, The system has a multi-target recognition function. The anatomical targets include, but are not limited to, blood vessels, ureters, nerve plexuses, peritoneum / fascia layers, and tumor bodies. By setting multi-channel prediction output in a single neural network model or constructing cascaded modules, it can achieve synchronous and non-interfering extraction and visualization of multiple key intraoperative anatomical structures such as blood vessel contours, nerve pathways, and fascial boundaries. Different semi-transparent color layers (such as red representing the inferior mesenteric artery IMA and blue representing the inferior mesenteric vein IMV) are superimposed on the original surgical image and fed back to the surgical monitor in real time to achieve graphic superposition output.

3. The method for real-time automatic intraoperative vessel identification based on deep neural network learning according to claim 1, characterized in that, include: The preprocessing module performs specific extreme value clipping (95th percentile) and normalization on the RGB channels: the clipping range for the R channel is (72, 255), for the G channel it is (28, 244), and for the B channel it is (23, 231). The core recognition engine automatically configures hyperparameters through the nnU-Net v2 framework. Its U-Net network has 9 layers, with a maximum configuration of 512 feature maps per layer, and the convolution kernel dimension is 3×3. The framework automatically calculates the optimal image preprocessing parameters, convolution kernel dimension, number of network downsampling layers, and number of feature maps per layer based on the fingerprint features of the training dataset.

4. The method for real-time automatic intraoperative vessel identification based on deep neural network learning according to claim 1, characterized in that, The system employs a hybrid loss function during the model training phase. The calculation formula is: In the formula, This represents the Dice loss function, used to optimize the overall matching degree of the vascular region; This represents the cross-entropy loss function, used to optimize pixel accuracy at blood vessel boundaries; This represents a hybrid loss function, which is formed by fusing the Dice loss function and the cross-entropy loss function with a weight of 0.

5. This represents a statistic that measures the similarity between two sample sets, with values ​​ranging from [0,1], where 1 indicates complete overlap. These represent the number of pixels correctly predicted as positive (Category 1), the number of pixels incorrectly predicted as positive, and the number of missed positive pixels, respectively. , These represent the number of samples and the sample index, respectively. Indicates the first The true label of each sample; Indicates the first The predicted probability of a sample, that is, the probability that the model predicts that the sample belongs to the positive class, takes a value in the range of [0,1]. The complement of the actual labels; This indicates the probability that the model predicts the sample belongs to the negative class (category 0).

5. The method for real-time automatic intraoperative vessel identification based on deep neural network learning according to claim 1, characterized in that, The core recognition engine is further integrated or replaced with the MambaVesselNet architecture, which uses a linear complexity state space model to process long-range spatiotemporal dependencies in surgical videos, thereby improving the continuity of blood vessel recognition in environments with smoke, reflection, and large-scale obstruction.

6. The method for real-time automatic intraoperative vessel identification based on deep neural network learning according to claim 1, characterized in that, The system can be packaged as a lightweight software module and directly integrated into the laparoscopic host hardware, robotic surgical system platform, or mobile augmented reality (AR) surgical navigation platform. By projecting the identified 3D topological structure of blood vessels onto the head-mounted display device or AR surgical monitor worn by the surgeon, the system enables anatomical guidance of blood vessels. It can also be connected to a data storage device to store historical surgical data and model parameters, and can be equipped with a human-computer interaction unit to control model switching, display transparency adjustment, etc.

7. The method for real-time automatic intraoperative vessel identification based on deep neural network learning according to claim 1, characterized in that, The core recognition engine also has a multimodal data fusion function, which can automatically and elastically register preoperative 3D-CTA vascular reconstruction images with intraoperative real-time laparoscopic images, and use AI prediction results to dynamically correct the deformation of the preoperative anatomical model.

8. A method for real-time automatic intraoperative vessel identification based on deep neural network learning, using the system described in claim 1, characterized in that, include: Spatiotemporal continuity annotation: A annotation set is established using static images of surgical videos at a fixed frame rate (e.g., 1 frame per second); Reverse tracking logic: For blood vessels, starting from the scene where they have been severed or are clearly exposed, backtracking and labeling are performed in the reverse direction of the video stream towards the root, only labeling the exposed blood vessel pixels; Layered labeling: Different recognition targets are classified and labeled with multiple color blocks, and areas obscured by instruments are excluded.

9. The method for real-time automatic intraoperative vessel identification based on deep neural network learning according to claim 1, characterized in that, Deployed on an edge computing surgical workstation, the system is accelerated by TensorRT or OpenVINO, enabling the average inference latency per image for IMA and IMV to be kept below 0.08 seconds, thereby ensuring the continuity of intraoperative anatomical navigation.

10. The system according to any one of claims 1-9, characterized in that, The system also includes a safety warning module. When the movement trajectory of the surgical instrument enters the preset danger distance (such as 5mm) of the identified blood vessel contour, the system will automatically flash a red warning signal on the screen or issue an audio warning.

Images