Surgical navigation method, device, computer readable storage medium and electronic device

Abstract

This application provides a surgical navigation method, apparatus, computer-readable storage medium, and electronic device. The specific implementation involves: acquiring a surgical video stream targeting a target tissue; determining a first vascular bifurcation point in the surgical video stream; matching the first vascular bifurcation point with a second vascular bifurcation point to determine the matching result; determining vascular information in the surgical video stream based on the matching result; and overlaying the vascular information in the surgical video stream while displaying the surgical video stream for surgical navigation. This solution achieves vascular positioning in the surgical video stream through vascular bifurcation point matching, thereby realizing surgical navigation, improving navigation accuracy, and contributing to improved surgical precision. This solution is implemented purely visually, without relying on external markers or additional equipment, significantly shortening preoperative preparation time, and is unaffected by electromagnetic interference, marker occlusion, etc., exhibiting strong operational stability.

Description

Technical Field

[0001] This application relates to the field of surgical navigation system technology, and more specifically, to a surgical navigation method, apparatus, computer-readable storage medium, and electronic device. Background Technology

[0002] Surgical navigation is a minimally invasive treatment method that uses image-guided and computer-aided precision positioning technology to help surgeons accurately locate target areas and perform surgery before or during the procedure. Surgical navigation is particularly important for surgeries involving complex anatomical structures or lesions requiring high precision in localization, such as thoracoscopic surgery.

[0003] In existing technologies, surgical navigation mostly employs magnetic navigation or optical tracking schemes.

[0004] Magnetic navigation systems utilize electromagnetic sensors mounted on surgical instruments to track position and orientation in real time using electromagnetic fields. However, this method is susceptible to interference from metallic objects and electromagnetic fields, leading to decreased positioning accuracy. Furthermore, magnetic navigation typically requires specialized electromagnetic source equipment, increasing the complexity and cost of equipment in the operating room.

[0005] Optical tracking solutions rely on external cameras and optical markers attached to surgical instruments or patients for three-dimensional spatial positioning. Optical tracking systems require a large number of markers to be placed in the surgical area, resulting in lengthy preoperative preparation times. Summary of the Invention

[0006] This application provides a surgical navigation method, device, computer-readable storage medium, and electronic device, aiming to at least solve one of the aforementioned technical deficiencies. The technical solution adopted in this application is as follows:

[0007] In a first aspect, embodiments of this application provide a surgical navigation method, the method comprising:

[0008] Acquire surgical video streams targeting the target tissue and identify the first vascular bifurcation point in the surgical video stream;

[0009] The first vascular bifurcation point and the second vascular bifurcation point are matched to determine the matching result. The second vascular bifurcation point is the vascular bifurcation point in the 3D vascular model. The 3D vascular model is used to characterize the vascular vessels in the target tissue.

[0010] Information about blood vessels in the surgical video stream is determined based on the matching results;

[0011] When displaying the surgical video stream, information about the blood vessels in the surgical video stream is overlaid to facilitate surgical navigation;

[0012] Obtain the preoperative planning file, generate a navigation path based on the preoperative planning file, and overlay the navigation path when displaying the surgical video stream.

[0013] As an optional method, obtain preoperative planning documents, including:

[0014] Retrieve preoperative planning documents from a removable storage device, which has pre-stored the preoperative planning documents.

[0015] Preoperative planning documents are obtained via a wired or wireless network connection to the management system for preoperative planning documents.

[0016] As an alternative approach, information about blood vessels in the surgical video stream is determined based on the matching results, including:

[0017] Determine the spatial angle of the target tissue in the surgical video stream;

[0018] Based on spatial angles and matching results, information about blood vessels in the surgical video stream is determined.

[0019] As an alternative method, determining the spatial angle of the target tissue in the surgical video stream includes:

[0020] Obtain the video processing model, which includes a backbone network and fully connected layers;

[0021] The video frames of the surgical video stream are input into the backbone network for feature extraction to obtain frame image features;

[0022] The frame image features are input into the fully connected layer to obtain the spatial angle of the target tissue.

[0023] As an alternative approach, the video processing model also includes a vessel segmentation network, and other methods include:

[0024] The frame image features are input into the blood vessel segmentation network to obtain the blood vessel segmentation results.

[0025] As an optional approach, the above method also includes at least one of the following:

[0026] The 3D model of the blood vessels is rotated based on spatial angles, and the rotated 3D model of the blood vessels is overlaid and displayed when the surgical video stream is shown.

[0027] Display a 3D model of blood vessels and label the target blood vessels in the 3D model. The target blood vessels are the blood vessels that correspond to the blood vessels in the surgical video stream.

[0028] When displaying the surgical video stream, the image regions corresponding to the blood vessel segmentation results are labeled and displayed.

[0029] As an optional approach, the first vascular bifurcation point is matched with the second vascular bifurcation point to determine the matching result, including:

[0030] The video frames of the surgical video stream, the first vascular bifurcation point, the points in the 3D model of the vascular system, and the second vascular bifurcation point are input into a pre-trained vascular bifurcation point matching model, and the similarity between the first vascular bifurcation point and the second vascular bifurcation point is output.

[0031] Matching results are determined based on similarity.

[0032] As an alternative approach, the vascular bifurcation point matching model is trained as follows:

[0033] The training samples include video frames of the sample surgical video stream, the third vascular bifurcation point in the video frames of the sample surgical video stream, points in the sample vascular 3D model, and the fourth vascular bifurcation point in the sample vascular 3D model.

[0034] The model for matching vascular bifurcation points is trained using training samples. The training objectives include minimizing the difference between the matching third and fourth vascular bifurcation points and maximizing the difference between the mismatched third and fourth vascular bifurcation points.

[0035] As an optional approach, the objectives of model training may also include at least one of the following:

[0036] Maximize the differences between different third vascular bifurcation points;

[0037] Maximize the differences between different fourth vascular bifurcation points;

[0038] Minimize the first vessel segmentation loss determined based on video frames from the sample surgical video stream;

[0039] Minimize the second vessel segmentation loss determined based on points in the sample vessel 3D model.

[0040] As an alternative approach, determining the first blood vessel bifurcation point in the surgical video stream includes:

[0041] Based on the surgical video stream, or based on the surgical video stream and the blood vessel segmentation results, determine the first blood vessel bifurcation point in the surgical video stream.

[0042] As an alternative approach, the above methods also include:

[0043] Obtain the surgical path and identify the vascular bifurcation points along the surgical path as the second vascular bifurcation point.

[0044] Secondly, embodiments of this application provide a surgical navigation device, the device comprising:

[0045] The blood vessel bifurcation point determination module is used to acquire the surgical video stream for the target tissue and determine the first blood vessel bifurcation point in the surgical video stream.

[0046] The vessel bifurcation point matching module is used to match the first vessel bifurcation point with the second vessel bifurcation point and determine the matching result. The second vessel bifurcation point is the vessel bifurcation point in the three-dimensional vessel model, which is used to represent the vessels in the target tissue.

[0047] The vascular information determination module is used to determine the vascular information in the surgical video stream based on the matching results;

[0048] The surgical navigation module is used to overlay information about blood vessels in the surgical video stream for surgical navigation.

[0049] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the surgical navigation method described above.

[0050] Fourthly, embodiments of this application provide an electronic device, which includes:

[0051] One or more processors; and

[0052] A memory associated with one or more of the processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the surgical navigation method.

[0053] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the surgical navigation method described above.

[0054] The beneficial effects of the technical solutions provided in this application are:

[0055] The solution provided in this application involves acquiring a surgical video stream targeting a target tissue, determining a first vascular bifurcation point in the video stream, matching the first vascular bifurcation point with a second vascular bifurcation point in a 3D vascular model, and determining the matching result. Based on the matching result, information about the blood vessels in the surgical video stream is determined. When displaying the surgical video stream, the information about the blood vessels in the video stream is overlaid for surgical navigation. This solution, by matching the vascular bifurcation points in the surgical video stream with the vascular bifurcation points in the 3D vascular model, achieves the localization of the blood vessels in the surgical video stream, thereby effectively realizing surgical navigation, improving navigation accuracy, and contributing to improved surgical precision.

[0056] Compared to existing magnetic navigation solutions that rely on electromagnetic sensors and optical tracking solutions that rely on optical markers, this solution is based on pure vision. It does not rely on additional sensors, markers, or external camera equipment, and therefore does not require the placement of external markers or the installation of additional equipment. This can significantly shorten the preoperative preparation time and is not limited by electromagnetic interference or marker obstruction. It can work stably in the surgical environment, improving the system's versatility and applicability. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0058] Figure 1 This is a system architecture diagram applicable to the embodiments of this application;

[0059] Figure 2 A schematic flowchart illustrating the surgical navigation method provided in this application embodiment;

[0060] Figure 3 A schematic diagram of the first vascular bifurcation point provided in an embodiment of this application;

[0061] Figure 4 This is a schematic diagram of the second vascular bifurcation point provided in an embodiment of this application;

[0062] Figure 5a , Figure 5b , Figure 5c and Figure 5d A schematic diagram of a three-dimensional model of the blood vessels in lung tissue from different spatial angles;

[0063] Figure 6 A schematic diagram of a matching process for vascular bifurcation points is provided for an embodiment of this application;

[0064] Figure 7 An overall flowchart of one specific implementation of the method provided in this application embodiment;

[0065] Figure 8 This is a schematic diagram of the surgical navigation device provided in the embodiments of this application;

[0066] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0068] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0069] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0070] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0071] To facilitate understanding of this application, the system architecture on which this application is based will first be described. For example... Figure 1 The diagram shows an exemplary system architecture that can be applied to embodiments of this application, such as... Figure 1 As shown, the system architecture may include: a terminal, a network, and a server.

[0072] A network is a medium used to provide a communication link between a server and a terminal. Networks can include various connection types, such as wired and wireless communication links or fiber optic cables, etc.

[0073] The terminal may include, but is not limited to, mobile phones, tablets, and smart wearable devices. Smart wearable devices may include smartwatches, smart glasses, smart bracelets, VR (Virtual Reality) devices, AR (Augmented Reality) devices, mixed reality devices (i.e., devices that support both virtual and augmented reality), and so on. In this embodiment, the user equipment typically has information display functionality, such as a display screen. In addition, it may also have operation command input functionality, such as inputting operation commands via touchscreen, keyboard, or voice.

[0074] The server can be a dedicated server, a server cluster, or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a hosting product within the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPS) services, such as high management difficulty and weak service scalability.

[0075] In one embodiment, the surgical navigation method provided in this application can be executed by a terminal, or the server can obtain a surgical video stream, determine the vascular information in the surgical video stream based on the surgical video stream, and return the vascular information in the surgical video stream to the terminal, so that the terminal can overlay the vascular information in the surgical video stream when displaying the surgical video stream, thereby realizing surgical navigation.

[0076] It should be understood that Figure 1 The number of servers and terminals shown is merely illustrative. Depending on implementation needs, there can be any number of servers and terminals.

[0077] Figure 2 This illustration shows a flowchart of a surgical navigation method provided in an embodiment of this application. The method can be performed by… Figure 1 This is executed on the terminal or server in the system shown. For example... Figure 2 As shown, this method mainly includes:

[0078] Step S210: Acquire the surgical video stream for the target tissue and determine the first vascular bifurcation point in the surgical video stream;

[0079] Step S220: Match the first vascular bifurcation point with the second vascular bifurcation point and determine the matching result. The second vascular bifurcation point is the vascular bifurcation point in the 3D vascular model. The 3D vascular model is used to characterize the vascular vessels in the target tissue.

[0080] Step S230: Determine the information of blood vessels in the surgical video stream based on the matching results;

[0081] Step S240: When displaying the surgical video stream, overlay the information of the blood vessels in the surgical video stream to perform surgical navigation.

[0082] In this embodiment of the application, during surgery on the target tissue, a surgical video stream can be acquired using an image acquisition device. The surgical video stream can display local tissues, blood vessels, and surgical instruments within the target tissue.

[0083] Vascular vessels refer to arteries, veins, and trachea. Taking thoracoscopic surgery as an example, the target tissue is lung tissue, and the vascular vessels specifically refer to the pulmonary artery, pulmonary vein, and bronchi.

[0084] A vascular bifurcation point is the junction where multiple different blood vessels connect, and it can characterize the connection relationship between these vessels. Vascular bifurcation points can serve as key points in the target tissue for locating blood vessels displayed in the surgical video stream.

[0085] It is understandable that the blood vessels in this case are actually distinguished in the form of blood vessel segments or blood vessel subsegments, such as bronchial segments or bronchial subsegments, and the blood vessel bifurcation points are actually the connection points of different blood vessel segments or blood vessel subsegments.

[0086] The first vascular bifurcation point is the vascular bifurcation point identified from the video frames of the surgical video stream.

[0087] As an example, Figure 3 This is a schematic diagram of the first vascular bifurcation point provided in an embodiment of this application.

[0088] like Figure 3 As shown, points A and B are both the first bifurcation points of the blood vessels. Point A is located at the junction of blood vessels a0, a1, and a2. Point B is located at the junction of blood vessels b0, b1, and b2.

[0089] A 3D model of blood vessels is a three-dimensional model constructed for the blood vessels in a target tissue. As an example, a 3D model of the target tissue can be constructed based on a medical image, and then the 3D model of blood vessels can be segmented from the 3D model of the target tissue.

[0090] In this embodiment of the application, the bifurcation point of the blood vessel, namely the second bifurcation point, can be marked in the three-dimensional model of the blood vessel.

[0091] As an example, Figure 4 This is a schematic diagram of the second vascular bifurcation point provided in an embodiment of this application.

[0092] like Figure 4 As shown in the diagram, point C is the second bifurcation point of the blood vessel. Point C is located at the junction of blood vessels c0, c1, and c2.

[0093] In this embodiment, matching the first and second vascular bifurcation points yields a matching result. The matched first and second vascular bifurcation points correspond to the same actual vascular bifurcation point. By identifying the second vascular bifurcation point that matches the first vascular bifurcation point, the information of the vascular connected to the matched second vascular bifurcation point can be used as the vascular information in the surgical video stream.

[0094] Reference Figure 3 , Figure 4 In the example shown, if the first vascular bifurcation point A and the second vascular bifurcation point C match, it means that the first vascular bifurcation point A and the second vascular bifurcation point C correspond to the same actual vascular bifurcation point. The vessels c0, c1, and c2 connected to the second vascular bifurcation point C should match the vessels connected to the first vascular bifurcation point A, so that the information of vessels c0, c1, and c2 can be used as the vascular information in the surgical video stream.

[0095] As an example, vascular information can include at least the vascular name.

[0096] In this embodiment of the application, after determining the vascular information in the surgical video stream, the vascular information in the surgical video stream can be overlaid and displayed on the surgical video stream. For example, the text of the vascular information in the surgical video stream can be overlaid and displayed at the position of the vascular in the surgical video stream.

[0097] The method provided in this application involves acquiring a surgical video stream targeting a target tissue, determining a first vascular bifurcation point in the surgical video stream, matching the first vascular bifurcation point with a second vascular bifurcation point in a 3D vascular model, determining the matching result, determining vascular information in the surgical video stream based on the matching result, and overlaying the vascular information in the surgical video stream when displaying the surgical video stream for surgical navigation. In this solution, by matching the vascular bifurcation points in the surgical video stream with the vascular bifurcation points in the 3D vascular model, the location of the vascular in the surgical video stream is achieved, thereby effectively realizing surgical navigation, improving navigation accuracy, and contributing to improved surgical precision.

[0098] Compared to existing magnetic navigation solutions that rely on electromagnetic sensors and optical tracking solutions that rely on optical markers, this solution is based on pure vision. It does not rely on additional sensors, markers, or external camera equipment, and therefore does not require the placement of external markers or the installation of additional equipment. This can significantly shorten the preoperative preparation time and is not limited by electromagnetic interference or marker obstruction. It can work stably in the surgical environment, improving the system's versatility and applicability.

[0099] In one optional embodiment of this application, determining the information of blood vessels in the surgical video stream based on the matching result includes:

[0100] Determine the spatial angle of the target tissue in the surgical video stream;

[0101] Based on spatial angles and matching results, information about blood vessels in the surgical video stream is determined.

[0102] In this embodiment, multiple blood vessels connect to a vascular bifurcation point. After determining a second vascular bifurcation point that matches the first vascular bifurcation point, the blood vessel connected to the matching second vascular bifurcation point is the same blood vessel connected to the first vascular bifurcation point, thus enabling the acquisition of information about the blood vessels connected to the first vascular bifurcation point. However, since the actual spatial angle of the target tissue during surgery may differ from the spatial angle of the vascular 3D model, directly matching the information of the blood vessels connected to the second vascular bifurcation point to the blood vessels connected to the first vascular bifurcation point could lead to information mismatch. For example, it could cause a misalignment between the displayed blood vessel information and the actual blood vessels displayed in the surgical video stream.

[0103] During surgery, the spatial angle of the target tissue may change in real time. Taking lung surgery as an example, the surgeon can adjust the spatial angle of the lung using instruments during the operation. Therefore, it is necessary to detect the spatial angle of the target tissue in the surgical video stream in real time to match vascular information.

[0104] As an example, a default spatial angle can be specified for the target tissue, and the spatial angles determined based on the surgical video stream are all based on this default spatial angle.

[0105] As an example, the target tissue is lung tissue. Figure 5a , Figure 5b , Figure 5c and Figure 5d A schematic diagram of the vascular system of lung tissue from different spatial angles.

[0106] In this example, you can Figure 5a The image shown is a 3D model of a blood vessel from the default spatial angle. Figure 5a , Figure 5b , Figure 5c and Figure 5d As shown, the 3D model of the blood vessel is in different poses under different spatial angles.

[0107] In this embodiment of the application, after determining the spatial angle of the target tissue, the three-dimensional model of the blood vessels can be rotated based on the spatial angle. Then, the blood vessels in the rotated three-dimensional model of the blood vessels are matched with the blood vessels in the surgical video stream, thereby determining the matching relationship between the blood vessels in the three-dimensional model of the blood vessels and the blood vessels in the surgical video stream, and thus determining the information of the blood vessels in the surgical video stream.

[0108] In one optional embodiment of this application, determining the spatial angle of the target tissue in the surgical video stream includes:

[0109] Obtain the video processing model, which includes a backbone network and fully connected layers;

[0110] The video frames of the surgical video stream are input into the backbone network for feature extraction to obtain frame image features;

[0111] The frame image features are input into the fully connected layer to obtain the spatial angle of the target tissue.

[0112] In this embodiment of the application, the spatial angle of the target tissue can be determined by a video processing model.

[0113] Specifically, the video processing model can include a backbone network and fully connected layers. The backbone network is used to extract frame image features from the surgical video stream. The prediction of spatial angles can be viewed as a regression task; by inputting the frame image features into the fully connected layers, the predicted spatial angles can be obtained.

[0114] In one optional embodiment of this application, the video processing model further includes a blood vessel segmentation network, and the above method further includes:

[0115] The frame image features are input into the blood vessel segmentation network to obtain the blood vessel segmentation results.

[0116] In this embodiment of the application, the video processing model may further include a vascular segmentation network, which obtains vascular segmentation results by inputting frame image features into the vascular segmentation network.

[0117] As an example, the segmentation of the surgical video stream in this case can be achieved using a masked-attention mask transformer (mask2former) for video. Taking lung surgery as an example, the video segmentation results can include not only the segmentation results of blood vessels, but also the segmentation results of surgical instruments, chest wall lymph nodes, lungs, etc.

[0118] In one optional embodiment of this application, the method further includes at least one of the following:

[0119] The 3D model of the blood vessels is rotated based on spatial angles, and the rotated 3D model of the blood vessels is overlaid and displayed when the surgical video stream is shown.

[0120] Display a 3D model of blood vessels and label the target blood vessels in the 3D model. The target blood vessels are the blood vessels that correspond to the blood vessels in the surgical video stream.

[0121] When displaying the surgical video stream, the image regions corresponding to the blood vessel segmentation results are labeled and displayed.

[0122] Obtain the preoperative planning file, generate a navigation path based on the preoperative planning file, and overlay the navigation path when displaying the surgical video stream.

[0123] In this embodiment, in addition to displaying vascular information, various display functions can be provided to improve the effectiveness of surgical navigation.

[0124] As an alternative, the 3D model of the blood vessels can be rotated based on spatial angles, and the rotated 3D model of the blood vessels can be overlaid and displayed when the surgical video stream is shown.

[0125] Among them, after rotating the 3D model of the vascular vessel based on spatial angle, the 3D model of the vascular vessel can be registered with the vascular vessel in the surgical video stream. By superimposing the rotated 3D model of the vascular vessel with the surgical video stream, it is convenient to use the overall 3D model of the vascular vessel as a reference, thereby achieving better surgical navigation results.

[0126] As an optional approach, a 3D model of the blood vessels can be displayed, and the target blood vessels in the 3D model can be labeled and displayed. The target blood vessels are those corresponding to the blood vessels in the surgical video stream.

[0127] In particular, when displaying the 3D model of blood vessels, the blood vessels currently displayed in the surgical video stream can be marked and displayed in the 3D model of blood vessels, that is, the target blood vessels can be marked and displayed, so as to facilitate understanding of the position of the blood vessels in the surgical video stream in the 3D model of blood vessels, thereby achieving better surgical navigation results.

[0128] As an optional approach, when displaying the surgical video stream, the image regions corresponding to the blood vessel segmentation results are labeled and displayed.

[0129] In this way, the image regions of the blood vessel segmentation results can be labeled, and combined with the information of the displayed blood vessels, it is easier to understand the current blood vessel status in the surgical video stream, thereby achieving better surgical navigation results.

[0130] As an optional approach, the preoperative planning file is obtained, a navigation path is generated based on the preoperative planning file, and the navigation path is overlaid and displayed when the surgical video stream is shown.

[0131] The preoperative planning document includes the surgical path. After locating the blood vessels in the surgical video stream, a navigation path can be generated based on the location of the blood vessels and the surgical path, and the navigation path can be overlaid and displayed on the surgical video stream.

[0132] The solution provided in this application can be applied to an intraoperative laparoscopic navigation system. This system can generate a navigation path based on a preoperative planning file, thus providing intraoperative navigation. During laparoscopic navigation, the system can also provide real-time identification of various anatomical structures and surgical instruments in the surgical video stream and register them with the preoperative 3D reconstruction. This enables simultaneous comparison and display of laparoscopic images and 3D reconstructions, providing location information and real-time path navigation, and mitigating the risk of misoperation.

[0133] In this example, the intraoperative laparoscopic navigation system is a standalone workstation that can be pushed into the operating room and is connected to the laparoscopic video for analysis. It provides a high-definition display screen, spatially positioned alongside the laparoscopic display screen, offering synchronized display of laparoscopic video and AI-generated navigation information.

[0134] Taking thoracic surgery as an example, a 3D model can be obtained by reconstructing the medical images of the tissue to be operated on, and intraoperative laparoscopic video images can be acquired. The 3D model is then registered and fused with the intraoperative laparoscopic video images, and anatomical structures are identified in real time within the intraoperative laparoscopic video, such as identifying blood vessel names or corresponding relationships in the 3D reconstruction results. This allows surgeons to obtain real-time name indications or 3D spatial positioning of the anatomical tissues currently in the laparoscopy, thereby effectively achieving surgical navigation. Furthermore, this system can also indicate the surgical path and issue prompts when deviations from the actual surgical path are detected.

[0135] In one optional embodiment of this application, obtaining the preoperative planning document includes:

[0136] Retrieve preoperative planning documents from a removable storage device, which has pre-stored the preoperative planning documents.

[0137] Preoperative planning documents are obtained via a wired or wireless network connection to the management system for preoperative planning documents.

[0138] In this embodiment of the application, the intraoperative laparoscopic navigation system needs to acquire the preoperative planning file for intraoperative navigation.

[0139] Specifically, a removable storage device, such as a USB flash disk, can be connected to the preoperative planning file management system beforehand to download the preoperative planning files. The removable storage device is then connected to the intraoperative laparoscopic navigation system, allowing the system to retrieve the preoperative planning files from it. A wired or wireless connection can also be established between the intraoperative laparoscopic navigation system and the preoperative planning file management system, enabling the system to retrieve the preoperative planning files via either connection.

[0140] In one optional embodiment of this application, matching the first vascular bifurcation point with the second vascular bifurcation point and determining the matching result includes:

[0141] The video frames of the surgical video stream, the first vascular bifurcation point, the points in the 3D model of the vascular system, and the second vascular bifurcation point are input into a pre-trained vascular bifurcation point matching model, and the similarity between the first vascular bifurcation point and the second vascular bifurcation point is output.

[0142] Matching results are determined based on similarity.

[0143] In this embodiment of the application, the matching result between the first and second vascular bifurcation points can be determined based on a pre-trained vascular bifurcation point matching model.

[0144] Specifically, video frames from the surgical video stream, the first vascular bifurcation point, points in the 3D model of the vascular system, and the second vascular bifurcation point can be input into a vascular bifurcation point matching model. This model includes a first backbone network for extracting 3D point features from the 3D model of the vascular system. A second backbone network is also included to extract 2D point features from the video frames of the surgical video stream. The 2D point features extracted by the second backbone network are linearly transformed to obtain 2D point features with the same feature dimensions as the 3D point features. Then, the similarity between the 2D point features of the first vascular bifurcation point and the 3D point features of the second vascular bifurcation point is calculated, yielding the output of the vascular bifurcation point matching model. This output is the similarity between the first and second vascular bifurcation points. Based on this similarity, a matching result is determined; for example, a first vascular bifurcation point and a second vascular bifurcation point with a similarity higher than a preset similarity threshold are considered a successful match.

[0145] As an example, the first backbone network is used to perform sparse convolution processing on points in the 3D model of the blood vessels to reduce the amount of data, avoid occupying too much GPU memory, and at the same time ensure high resolution.

[0146] As an example, the second backbone network can adopt a backbone network based on a shifted window transformer (swin transformer) model.

[0147] In one optional embodiment of this application, the vascular bifurcation point matching model is trained in the following manner:

[0148] The training samples include video frames of the sample surgical video stream, the third vascular bifurcation point in the video frames of the sample surgical video stream, points in the sample vascular 3D model, and the fourth vascular bifurcation point in the sample vascular 3D model.

[0149] The model for matching vascular bifurcation points is trained using training samples. The training objectives include minimizing the difference between the matching third and fourth vascular bifurcation points and maximizing the difference between the mismatched third and fourth vascular bifurcation points.

[0150] In this embodiment of the application, training samples can be constructed based on video frames of the sample surgical video stream, the third vascular bifurcation point in the video frames of the sample surgical video stream, points in the sample vascular 3D model, and the fourth vascular bifurcation point in the sample vascular 3D model. The vascular bifurcation point matching model is trained at least once based on the training samples until the training is completed, and the trained vascular bifurcation point matching model is obtained.

[0151] During training, points from the sample vascular 3D model are input into the first backbone network to extract 3D point features. Video frames from the sample surgical video stream are input into the second backbone network to obtain 2D point features. The 2D and 3D point features of the third and fourth vascular bifurcation points are extracted. A linear transformation is performed on the 2D point features of the third vascular bifurcation point to make its feature dimension the same as that of the fourth vascular bifurcation point. Then, a similarity score is calculated between the 2D and 3D point features of the third and fourth vascular bifurcation points. A first contrastive loss is calculated based on this similarity score, and the vascular bifurcation point matching model is trained using this first contrastive loss.

[0152] Specifically, the objectives of model training may include minimizing the difference between matching third and fourth vascular bifurcation points and maximizing the difference between mismatched third and fourth vascular bifurcation points.

[0153] As an example, the fourth vascular bifurcation point specifically includes four points: x0, x1, x2, and x3, while the third vascular bifurcation point specifically includes two points: y0 and y1. x1 matches y0, x3 matches y1, and the remaining vascular bifurcation points do not match, thus obtaining the first label matrix, which can be represented by matrix 1 below.

[0154]

[0155] In this example, a first similarity matrix is ​​constructed between each fourth vascular bifurcation point and each third vascular bifurcation point. Then, a first contrast loss is calculated based on the first similarity matrix and the first label matrix, and the model is trained based on the first contrast loss.

[0156] In one optional embodiment of this application, the objective of model training further includes at least one of the following:

[0157] Maximize the differences between different third vascular bifurcation points;

[0158] Maximize the differences between different fourth vascular bifurcation points;

[0159] Minimize the first vessel segmentation loss determined based on video frames from the sample surgical video stream;

[0160] Minimize the second vessel segmentation loss determined based on points in the sample vessel 3D model.

[0161] In this embodiment of the application, the similarity between each two third vascular bifurcation points can be calculated based on the two-dimensional point features of each two third vascular bifurcation points, and a second contrast loss can be calculated based on the similarity between each two third vascular bifurcation points, thereby training the model based on the second contrast loss.

[0162] Continuing with the previous example, the third vascular bifurcation point specifically includes two points, y0 and y1. The second label matrix can be represented by matrix 2 below.

[0163]

[0164] A second similarity matrix is ​​constructed based on the similarity between every two third vascular bifurcation points. Then, a second contrast loss is calculated based on the second similarity matrix and the second label matrix, and the model is trained based on this second contrast loss.

[0165] In this embodiment of the application, the similarity between each two fourth vascular bifurcation points can be calculated based on the three-dimensional point features of each two fourth vascular bifurcation points, and a third contrast loss can be calculated based on the similarity between each two fourth vascular bifurcation points, thereby training the model based on the third contrast loss.

[0166] Continuing with the previous example, the fourth vascular bifurcation point specifically includes four points, namely x0, x1, x2, and x3. The third label matrix can be represented by matrix 3 below.

[0167]

[0168] A third similarity matrix is ​​constructed based on the similarity between every two fourth vascular bifurcation points. Then, a third contrast loss is calculated based on the third similarity matrix and the third label matrix, and the model is trained based on this third contrast loss.

[0169] Understandably, when calculating the second and third contrast loss, positive samples cannot be provided in the training samples. In this scheme, negative sample pairs composed of different vascular bifurcation points can be used to maximize the difference between the vascular bifurcation points in the negative sample pairs, thereby obtaining the contrast loss.

[0170] In this embodiment, vascular segmentation prediction can also be performed based on the 3D model of the vascular system or video frames of the surgical video stream, enabling the model to learn reasonable semantic features better and accelerating the convergence of the model.

[0171] Specifically, the first blood vessel segmentation prediction result can be obtained by performing blood vessel segmentation based on the two-dimensional point features of points in the video frame. The first blood vessel segmentation loss is calculated based on the first blood vessel segmentation prediction result and the first blood vessel segmentation label, and the first blood vessel segmentation loss is minimized during the model training process.

[0172] The blood vessels can be segmented based on the three-dimensional point features of each point in the sample blood vessel three-dimensional model to obtain the second blood vessel segmentation prediction result. The second blood vessel segmentation loss is calculated based on the second blood vessel segmentation prediction result and the second blood vessel segmentation label, and the second blood vessel segmentation loss is minimized during the model training process.

[0173] As an example, Figure 6 This is a schematic diagram of a matching process for a blood vessel bifurcation point provided in an embodiment of this application.

[0174] like Figure 6 As shown, points from the 3D model of the sample blood vessels are input into the first backbone network to extract the 3D point features of the points in the 3D model of the sample blood vessels. In this example, sparse convolution can be used for feature extraction in the first backbone network.

[0175] After extracting the three-dimensional point features of the points in the sample vascular three-dimensional model, vascular segmentation can be performed based on the three-dimensional point features of each point to obtain the second vascular segmentation prediction result. The local loss (focal loss) is calculated by combining the second vascular segmentation prediction result with the second vascular segmentation label and recorded as the first local loss (equivalent to the aforementioned first vascular segmentation loss).

[0176] In this example, the three-dimensional point features of the fourth vascular bifurcation point can be extracted from the three-dimensional point features of the points in the sample vascular three-dimensional model, and then the third contrast loss can be calculated based on the similarity between the three-dimensional point features of every two fourth vascular bifurcation points.

[0177] The video frames of the sample surgical video stream are input into the second backbone network to obtain the two-dimensional point features of the points in the video frames. In this example, the second backbone network can be a Swin transformer backbone network.

[0178] After extracting the two-dimensional point features of points in the video frames of the sample surgical video stream, the blood vessels can be segmented based on the two-dimensional point features of each point in the video frame to obtain the first blood vessel segmentation prediction result. The first blood vessel segmentation prediction result and the first blood vessel segmentation label are then used to calculate the second local loss (equivalent to the aforementioned second blood vessel segmentation loss).

[0179] In this example, the two-dimensional point features of the third vascular bifurcation point can be extracted from the two-dimensional point features of the points in the video frame, and then the second contrast loss can be calculated based on the similarity between the two-dimensional point features of every two third vascular bifurcation points.

[0180] In this example, in addition to calculating the contrast loss for 2D and 3D point features within the same modality, a first contrast loss between 2D and 3D point features can also be calculated across modalities. Specifically, the 2D point features of the third vascular bifurcation point can be linearly transformed to make their feature dimensions the same as the 3D point features of the fourth vascular bifurcation point. Then, the first contrast loss can be calculated based on the similarity between the 2D point features of the third vascular bifurcation point and the 3D point features of the fourth vascular bifurcation point.

[0181] In this example, a comprehensive loss can be calculated based on the first local loss, the second local loss, the first contrast loss, the second contrast loss, and the third contrast loss, and the model can be trained on the vascular bifurcation point matching model based on the comprehensive loss.

[0182] In one optional embodiment of this application, determining the first blood vessel bifurcation point in the surgical video stream includes:

[0183] Based on the surgical video stream, or based on the surgical video stream and the blood vessel segmentation results, determine the first blood vessel bifurcation point in the surgical video stream.

[0184] In this embodiment of the application, the first blood vessel bifurcation point can be extracted from the surgical video stream based on a key point detection algorithm.

[0185] As an alternative approach, after extracting the first vascular bifurcation point from the surgical video stream based on the key point detection algorithm, the vascular segmentation results can be used for filtering to exclude the first vascular bifurcation point that obviously does not conform to the vascular segmentation results.

[0186] In one optional embodiment of this application, the above method further includes:

[0187] Obtain the surgical path and identify the vascular bifurcation points along the surgical path as the second vascular bifurcation point.

[0188] In this embodiment of the application, the second vascular bifurcation point can be marked manually based on experience or automatically based on the surgical path.

[0189] Specifically, the surgical path can reflect the blood vessels traversed during the operation, and can be used to identify the possible blood vessel bifurcation points during the operation, marking these blood vessel bifurcation points as the second blood vessel bifurcation points.

[0190] As an example, the surgical path can be obtained from the preoperative planning document, or it can be determined based on the location of the lesion.

[0191] As an example, Figure 7 This is an overall flowchart of one specific implementation of the method provided in this application.

[0192] like Figure 7 As shown, surgical video streams can be acquired during surgery, and video segmentation and spatial angle determination can be performed on the surgical video streams to obtain the video segmentation results of blood vessels and the spatial angle of the target tissue.

[0193] The first vascular bifurcation point was obtained by detecting the vascular bifurcation point based on the surgical video stream.

[0194] The preoperative planning document can provide the second vascular bifurcation point marked in the 3D model of the blood vessels.

[0195] The first and second vascular bifurcation points are matched to obtain the matching results. Then, based on the matching results, vascular segmentation and pairing are performed to determine the vascular information in the surgical video stream, thereby superimposing and displaying the vascular information with the surgical video stream.

[0196] Based on and Figure 2 The method shown follows the same principle. Figure 8 This application provides a schematic diagram of the structure of a surgical navigation device according to an embodiment of the present application. Figure 8 As shown, the surgical navigation device 80 may include:

[0197] The blood vessel bifurcation point determination module 810 is used to acquire the surgical video stream for the target tissue and determine the first blood vessel bifurcation point in the surgical video stream.

[0198] The vessel bifurcation point matching module 820 is used to match the first vessel bifurcation point with the second vessel bifurcation point and determine the matching result. The second vessel bifurcation point is the vessel bifurcation point in the three-dimensional vessel model. The three-dimensional vessel model is used to characterize the vessels in the target tissue.

[0199] The vascular information determination module 830 is used to determine the vascular information in the surgical video stream based on the matching results;

[0200] The surgical navigation module is used to overlay and display vascular information 840 in the surgical video stream for surgical navigation.

[0201] As an optional approach, the vascular information determination module is specifically used for:

[0202] Determine the spatial angle of the target tissue in the surgical video stream;

[0203] Based on spatial angles and matching results, information about blood vessels in the surgical video stream is determined.

[0204] As an optional approach, the vascular information determination module 830, when determining the spatial angle of the target tissue in the surgical video stream, is specifically used for:

[0205] Obtain the video processing model, which includes a backbone network and fully connected layers;

[0206] The video frames of the surgical video stream are input into the backbone network for feature extraction to obtain frame image features;

[0207] The frame image features are input into the fully connected layer to obtain the spatial angle of the target tissue.

[0208] As an alternative, the video processing model also includes a blood vessel segmentation network, and the aforementioned device further includes:

[0209] The vessel segmentation module (not shown in the figure) is used to input frame image features into the vessel segmentation network to obtain vessel segmentation results.

[0210] As an optional approach, the surgical navigation module described above is also used for at least one of the following:

[0211] The 3D model of the blood vessels is rotated based on spatial angles, and the rotated 3D model of the blood vessels is overlaid and displayed when the surgical video stream is shown.

[0212] Display a 3D model of blood vessels and label the target blood vessels in the 3D model. The target blood vessels are the blood vessels that correspond to the blood vessels in the surgical video stream.

[0213] When displaying the surgical video stream, the image regions corresponding to the blood vessel segmentation results are labeled and displayed.

[0214] Obtain the preoperative planning file, generate a navigation path based on the preoperative planning file, and overlay the navigation path when displaying the surgical video stream.

[0215] As an optional method, obtain preoperative planning documents, including:

[0216] Retrieve preoperative planning documents from a removable storage device, which has pre-stored the preoperative planning documents.

[0217] Preoperative planning documents are obtained via a wired or wireless network connection to the management system for preoperative planning documents.

[0218] As an optional approach, the vascular bifurcation point matching module 820 is specifically used for:

[0219] The video frames of the surgical video stream, the first vascular bifurcation point, the points in the 3D model of the vascular system, and the second vascular bifurcation point are input into a pre-trained vascular bifurcation point matching model, and the similarity between the first vascular bifurcation point and the second vascular bifurcation point is output.

[0220] Matching results are determined based on similarity.

[0221] As an alternative approach, the vascular bifurcation point matching model is trained as follows:

[0222] The training samples include video frames of the sample surgical video stream, the third vascular bifurcation point in the video frames of the sample surgical video stream, points in the sample vascular 3D model, and the fourth vascular bifurcation point in the sample vascular 3D model.

[0223] The model for matching vascular bifurcation points is trained using training samples. The training objectives include minimizing the difference between the matching third and fourth vascular bifurcation points and maximizing the difference between the mismatched third and fourth vascular bifurcation points.

[0224] As an optional approach, the objectives of model training may also include at least one of the following:

[0225] Maximize the differences between different third vascular bifurcation points;

[0226] Maximize the differences between different fourth vascular bifurcation points;

[0227] Minimize the first vessel segmentation loss determined based on video frames from the sample surgical video stream;

[0228] Minimize the second vessel segmentation loss determined based on points in the sample vessel 3D model.

[0229] As an optional approach, the blood vessel bifurcation point determination module 810, when determining the first blood vessel bifurcation point in the surgical video stream, is specifically used for:

[0230] Based on the surgical video stream, or based on the surgical video stream and the blood vessel segmentation results, determine the first blood vessel bifurcation point in the surgical video stream.

[0231] As an optional approach, the aforementioned vascular bifurcation point determination module 810 is also used for:

[0232] Obtain the surgical path and identify the vascular bifurcation points along the surgical path as the second vascular bifurcation point.

[0233] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0234] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0235] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the methods in the foregoing method embodiments.

[0236] And an electronic device, comprising:

[0237] One or more processors; and

[0238] A memory associated with one or more processors, the memory being used to store program instructions that, when read and executed by one or more processors, perform the steps of any of the methods in the foregoing method embodiments.

[0239] In this embodiment, the electronic device can be an independent workstation that can be pushed into the operating room, providing a high-definition display screen. It can be positioned alongside the endoscope display screen in space to provide endoscope video and surgical navigation.

[0240] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the methods in the foregoing method embodiments.

[0241] in, Figure 9 An exemplary architecture of an electronic device is shown, which may include a processor 910, a video display adapter 911, a disk drive 912, an input / output interface 913, a network interface 914, and a memory 920. The processor 910, video display adapter 911, disk drive 912, input / output interface 913, network interface 914, and memory 920 can communicate with each other via a communication bus 930.

[0242] The processor 910 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0243] The memory 920 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 920 can store the operating system 921 for controlling the operation of the electronic device 900, and the basic input / output system (BIOS) 922 for controlling the low-level operations of the electronic device 900. Additionally, it can store a web browser 923, a data storage management system 924, and a surgical navigation device 925, etc. The aforementioned surgical navigation device 925 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 920 and is called and executed by the processor 910.

[0244] Input / output interface 913 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0245] Network interface 914 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0246] Bus 930 includes a pathway for transmitting information between various components of the device, such as processor 910, video display adapter 911, disk drive 912, input / output interface 913, network interface 914, and memory 920.

[0247] It should be noted that although the above-described device only shows the processor 910, video display adapter 911, disk drive 912, input / output interface 913, network interface 914, memory 920, bus 930, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0248] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of the embodiments of this application.

[0249] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A surgical navigation method, characterized in that, include: Acquire a surgical video stream targeting the target tissue and determine the first vascular bifurcation point in the surgical video stream; The first vascular bifurcation point and the second vascular bifurcation point are matched to determine the matching result. The second vascular bifurcation point is the vascular bifurcation point in the 3D model of the vascular system. The 3D model of the vascular system is used to characterize the vascular system in the target tissue. Based on the matching results, information about the blood vessels in the surgical video stream is determined; When displaying the surgical video stream, information about the blood vessels in the surgical video stream is overlaid to facilitate surgical navigation.

2. The method according to claim 1, characterized in that, Determining the information of blood vessels in the surgical video stream based on the matching result includes: Determine the spatial angle of the target tissue in the surgical video stream; Based on the spatial angle and the matching result, the information of the blood vessels in the surgical video stream is determined.

3. The method according to claim 2, characterized in that, Determining the spatial angle of the target tissue in the surgical video stream includes: Obtain a video processing model, which includes a backbone network and a fully connected layer; The video frames of the surgical video stream are input into the backbone network for feature extraction to obtain frame image features; The frame image features are input into a fully connected layer to obtain the spatial angle of the target tissue.

4. The method according to claim 3, characterized in that, The video processing model further includes a blood vessel segmentation network, and the method further includes: The frame image features are input into the blood vessel segmentation network to obtain the blood vessel segmentation results.

5. The method according to claim 2, characterized in that, It also includes at least one of the following: The 3D model of the blood vessel is rotated based on the spatial angle, and the rotated 3D model of the blood vessel is superimposed and displayed when the surgical video stream is shown. The three-dimensional model of the blood vessel is displayed, and the target blood vessels in the three-dimensional model are labeled and displayed. The target blood vessels are the blood vessels corresponding to the blood vessels in the surgical video stream. When displaying the surgical video stream, the image regions corresponding to the blood vessel segmentation results are labeled and displayed. Obtain the preoperative planning file, generate a navigation path based on the preoperative planning file, and overlay the navigation path when displaying the surgical video stream.

6. The method according to claim 5, characterized in that, The acquisition of preoperative planning documents includes: The preoperative planning file is retrieved from a removable storage device, which pre-stores the preoperative planning file. The preoperative planning document is obtained through a wired or wireless network connection to the management system of the preoperative planning document.

7. The method according to any one of claims 1-6, characterized in that, The step of matching the first vascular bifurcation point with the second vascular bifurcation point and determining the matching result includes: The video frames of the surgical video stream, the first vascular bifurcation point, the points in the 3D model of the vascular system, and the second vascular bifurcation point are input into a pre-trained vascular bifurcation point matching model, and the similarity between the first vascular bifurcation point and the second vascular bifurcation point is output. The matching result is determined based on the similarity.

8. The method according to claim 7, characterized in that, The blood vessel bifurcation point matching model is trained in the following manner: Acquire training samples, which include video frames of the sample surgical video stream, the third vascular bifurcation point in the video frames of the sample surgical video stream, points in the sample vascular 3D model, and the fourth vascular bifurcation point in the sample vascular 3D model. The model is trained using the training samples. The training objectives include minimizing the difference between the matching third and fourth vascular bifurcation points and maximizing the difference between the mismatched third and fourth vascular bifurcation points.

9. The method according to claim 8, characterized in that, The objective of the model training also includes at least one of the following: Maximize the differences between the different bifurcation points of the third vascular vessel; Maximize the differences between the different bifurcation points of the fourth vascular vessel; Minimize the first vessel segmentation loss determined based on video frames of the sample surgical video stream; Minimize the second vessel segmentation loss determined based on points in the sample vessel 3D model.

10. The method according to any one of claims 1-6, characterized in that, Determining the first blood vessel bifurcation point in the surgical video stream includes: Based on the surgical video stream, or based on the surgical video stream and the blood vessel segmentation results, the first blood vessel bifurcation point in the surgical video stream is determined.

11. The method according to any one of claims 1-6, characterized in that, The method further includes: Obtain the surgical path and determine the vascular bifurcation points passed through in the surgical path as the second vascular bifurcation point.

12. A surgical navigation device, characterized in that, include: The blood vessel bifurcation point determination module is used to acquire a surgical video stream targeting a target tissue and determine the first blood vessel bifurcation point in the surgical video stream. The vessel bifurcation point matching module is used to match the first vessel bifurcation point with the second vessel bifurcation point and determine the matching result. The second vessel bifurcation point is a vessel bifurcation point in the three-dimensional vessel model, and the three-dimensional vessel model is used to characterize the vessels in the target tissue. A vascular information determination module is used to determine the vascular information in the surgical video stream based on the matching result; The surgical navigation module is used to overlay information about blood vessels in the surgical video stream while displaying the surgical video stream, in order to perform surgical navigation.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1-11.

14. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1-11.

15. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program performs the steps of the method described in any one of claims 1-11.

Images