A surgical navigation method based on a hybrid knowledge model and related devices

Abstract

The application discloses a surgical navigation method based on a hybrid knowledge model and related devices. The method includes: obtaining a hybrid knowledge model that combines formalized surgical specifications and probabilistic anatomical mapping; based on the model and preoperative patient data, constructing a patient-specific knowledge representation; acquiring multi-modal perception data of the surgical scene in real time during surgery, and using the hybrid knowledge model for semantic understanding to generate a semantic representation of the intraoperative scene; spatially correlating the preoperative knowledge representation and the intraoperative semantic representation; and finally generating and outputting augmented reality navigation information based on the correlation result and the knowledge model. The application can deeply combine structured medical knowledge, individual anatomical features of patients, and real-time surgical scenes, and realize intelligent surgical navigation that is more accurate, semantically rich, and has clinical context perception ability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and medical technology, specifically a surgical navigation method and related device based on a hybrid knowledge model. Background Technology

[0002] Current surgical navigation systems primarily follow a "geometric mapping" paradigm, with the core objective of accurately registering preoperative 3D image models to the patient's intraoperative anatomical location. This paradigm has been successful in rigid structure surgeries such as neurosurgery and orthopedics. However, applying this paradigm to soft tissue surgeries such as those in the abdominal and pelvic cavities faces fundamental challenges: tissues undergo continuous, nonlinear, and extensive deformations due to respiration, heartbeat, and instrument manipulation, making it difficult for rigid or linear registration models based on static images to maintain intraoperative accuracy, leading to navigation deviations or even failure. Existing technologies, such as 3D laparoscopy, fluorescence navigation, and surgical robot systems, mainly focus on enhancing the operational dimension (e.g., restoring stereoscopic vision, improving operational stability) or providing additional sensory information channels. However, the systems themselves lack a deep semantic understanding of the surgical scenario and fail to digitally integrate standardized surgical procedure knowledge with real-time anatomical cognition. Therefore, surgeons still need to rely entirely on their own experience to complete the "visual-cognitive conversion" from 2D images to 3D anatomical structures on a 2D display interface. This results in a high cognitive load, a long learning curve, and surgical quality that is highly dependent on individual experience, making standardization and homogenization difficult. Summary of the Invention

[0003] To address the above technical problems, the present invention relates to a surgical navigation method and related apparatus based on a hybrid knowledge model, including but not limited to a surgical navigation device, electronic device, computer-readable storage medium, and computer program product based on a hybrid knowledge model.

[0004] Firstly, a surgical navigation method based on a hybrid knowledge model is provided, the method comprising: S1. Obtain a hybrid knowledge model, which integrates formal surgical normative knowledge units and probabilistic anatomical mapping knowledge units; S2. Based on the hybrid knowledge model and the preoperative medical data of the target object, generate a patient-specific knowledge representation; S3. During the surgery, acquire multimodal perception data of the surgical scene of the target object in real time; S4. Based on the hybrid knowledge model, perform semantic understanding on the multimodal perception data to generate a semantic representation of the intraoperative scene; S5. Spatially associate the patient-specific knowledge representation with the semantic representation of the intraoperative scene; S6. Based on the spatial association results and the hybrid knowledge model, generate augmented reality navigation information and output it.

[0005] In any embodiment of this application, the generation of patient-specific knowledge representation in S2 includes: inputting the preoperative medical data into a knowledge graph construction model; extracting anatomical entities contained in the preoperative medical data and semantic relationships between the anatomical entities through the knowledge graph construction model; and constructing the patient-specific knowledge representation based on the extracted anatomical entities and semantic relationships.

[0006] In any embodiment of this application, the knowledge graph construction model includes a shared encoder, an entity decoder, and a relation decoder; the extraction of anatomical entities and semantic relationships between the anatomical entities contained in the preoperative medical data includes: processing the preoperative medical data through the shared encoder to obtain feature representations; outputting segmentation information of the anatomical entities based on the feature representations through the entity decoder; and predicting the semantic relationship type between the anatomical entities based on the feature representations and the segmentation information through the relation decoder.

[0007] In any embodiment of this application, the shared encoder is a three-dimensional vision Transformer encoder.

[0008] In any embodiment of this application, the spatial association described in S5 includes: extracting a first set of feature points from the patient-specific knowledge representation; extracting a second set of feature points from the intraoperative scene semantic representation; performing initial alignment based on the first set of feature points and the second set of feature points; and estimating the deformation field between the intraoperative scene semantic representation and the patient-specific knowledge representation based on the result of the initial alignment.

[0009] In any embodiment of this application, the estimation of the deformation field is performed by a neural network.

[0010] In any embodiment of this application, generating augmented reality navigation information in S6 includes: determining at least one context parameter among the current surgical stage, surgical instrument position, and anatomical risk structure distance; determining the type and priority of the navigation content to be displayed according to the rules corresponding to the context parameter in the hybrid knowledge model; and generating corresponding visual enhancement information as the augmented reality navigation information based on the type and priority.

[0011] Secondly, a surgical navigation device based on a hybrid knowledge model is provided, comprising: The knowledge acquisition unit is used to initialize a hybrid knowledge model, which includes a formal surgical specification knowledge unit and a probabilistic anatomical mapping knowledge unit; it is also used to acquire the patient's multimodal perception data during surgery. The knowledge representation generation unit is used to generate patient-specific knowledge representations based on the hybrid knowledge model and the patient's preoperative medical data. The scene understanding unit is used to perform semantic understanding on the multimodal perception data based on the hybrid knowledge model, and generate a semantic representation of the intraoperative scene; A spatial association unit is used to spatially associate the patient-specific knowledge representation with the semantic representation of the intraoperative scene. The navigation output unit is used to generate and output augmented reality navigation information based on the spatial association results and the hybrid knowledge model.

[0012] Thirdly, an electronic device is provided, comprising: a processor, a communication module, a sensor, a user interface, and a storage unit, wherein the storage unit is used to store computer program code, the program code including computer instructions. When the processor executes these instructions, the electronic device performs the methods described in the first aspect and any of its embodiments above.

[0013] Fourthly, an electronic device is provided, comprising: a processor, a wireless communication module, a touchscreen, a speaker, and a storage unit, wherein the storage unit is used to store computer program code, the program code including computer instructions. When the processor executes these instructions, the electronic device performs the methods described in the first aspect and any of its embodiments above.

[0014] Fifthly, a computer-readable storage medium is provided, wherein a computer program is stored, the program comprising program instructions. When these instructions are executed by a processor, the processor performs the methods as described in the first aspect and any of its embodiments.

[0015] In a sixth aspect, a computer program product is provided, the computer program product comprising a computer program or instructions. When the computer program or instructions are executed on a computer, the computer will perform the methods as described in the first aspect and any of its embodiments above.

[0016] It should be understood that the above general descriptions and subsequent specific descriptions are for illustrative and explanatory purposes only and do not impose any limitations on this application.

[0017] In this application, compared with the prior art, the present invention provides a surgical navigation system and method based on a hybrid knowledge model. The method includes: constructing and fusing formal surgical normative knowledge units and probabilistic anatomical mapping knowledge units to form a hybrid knowledge model; generating a patient-specific knowledge graph based on the model and the patient's preoperative medical images; reconstructing a semantically labeled intraoperative 3D scene in real time using multimodal perception data during surgery; performing non-rigid registration between the patient-specific knowledge graph and the intraoperative semantic scene; and finally, dynamically generating and presenting augmented reality navigation information based on the registration results and real-time reasoning of the hybrid knowledge model. The innovations of this invention lie in: firstly, the digital integration of deterministic international surgical operating procedures with probabilistic surgical theories describing the anatomical patterns of populations, constructing a computable surgical knowledge engine and achieving a paradigm shift from "geometric mapping" to "knowledge-driven"; secondly, by constructing a patient-specific knowledge graph and employing hierarchical non-rigid registration, effectively solving the registration problem caused by large-scale deformation during soft tissue surgery; and thirdly, through a context-aware augmented reality navigation interface, transforming the implicit experience and explicit norms of experts into intuitive visual guidance, significantly reducing the cognitive load and learning curve for surgeons, and improving the safety, standardization, and homogenization of surgery. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments consistent with this application and, together with the specification, serve to illustrate the technical solutions of this application.

[0019] Figure 1 This is a flowchart illustrating a surgical navigation method based on a hybrid knowledge model, provided as an embodiment of this application.

[0020] Figure 2 This is a schematic diagram of the structure of a surgical navigation device based on a hybrid knowledge model, provided in an embodiment of this application.

[0021] Figure 3 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to more fully understand the technical solutions of this application, the technical solutions of this application will be explained in detail and clearly with reference to the accompanying drawings. It should be particularly noted that the described embodiments are only some examples of this application and do not represent all of them. Based on these embodiments, those skilled in the art can directly deduce all other possible implementation schemes without creative thinking, and these are also included within the protection scope of this application.

[0023] In the specification, claims, and related drawings of this application, the terms "first," "second," etc., are used only to distinguish different elements and do not imply any specific order. Furthermore, the use of "comprising" and "having," and their variations, indicates non-exclusive inclusion. This means that if a process, method, system, product, or device comprises a series of steps or components, it indicates that the process, method, system, product, or device is not limited to the listed steps or components and may also include other steps or components not listed, or other inherent steps or units thereof.

[0024] The term "embodiment" as used herein refers to any instance combining a particular feature, structure, or characteristic, which may be at least one embodiment of this application. The "embodiments" mentioned herein do not necessarily refer to the same specific case, nor do they imply that they are independent or exclusive alternatives. Those skilled in the art will understand that the embodiments described herein can be used with other embodiments. It should be clarified that in this application, "at least one" includes one or more instances, "multiple" means two or more instances, and "at least two" refers to two or more instances.

[0025] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The embodiments of this application will now be described with reference to the accompanying drawings.

[0026] Please see Figure 1 , Figure 1 This is a flowchart illustrating a surgical navigation method based on a hybrid knowledge model, provided as an embodiment of this application.

[0027] 101. Knowledge Model Acquisition: Acquire a hybrid knowledge model that integrates formal surgical normative knowledge units and probabilistic anatomical mapping knowledge units.

[0028] In another possible implementation, the hybrid knowledge model is dynamically acquired and updated through a cloud-based knowledge service interface, and the formal surgical specification knowledge unit and the probabilistic anatomical mapping knowledge unit can be remotely maintained and iterated based on the latest clinical guidelines and evidence-based medicine.

[0029] In another possible implementation, the construction of the hybrid knowledge model also includes an interactive knowledge editor, which allows domain experts to modify or supplement the rules and probabilistic relationships in a specific surgical scenario through a graphical interface, thereby achieving semi-automatic injection of expert experience and personalized customization of the model.

[0030] 102. Knowledge Representation: Based on the hybrid knowledge model and the preoperative medical data of the target object, generate a patient-specific knowledge representation.

[0031] In this embodiment, the generation of patient-specific knowledge representation includes: The preoperative medical data is input into a knowledge graph construction model; A model is constructed using the knowledge graph to extract anatomical entities and semantic relationships between them from the preoperative medical data. Based on the extracted anatomical entities and semantic relationships, the patient-specific knowledge representation is constructed.

[0032] In this embodiment, the knowledge graph construction model includes a shared encoder, an entity decoder, and a relation decoder. The extraction of anatomical entities and semantic relationships between the anatomical entities contained in the preoperative medical data includes: The preoperative medical data is processed by the shared encoder to obtain feature representations; The entity decoding head outputs segmentation information of the anatomical entity based on the feature representation. Based on the feature representation and the segmentation information, the semantic relationship type between the anatomical entities is predicted using the relationship decoding head.

[0033] In this embodiment, the shared encoder is a 3D vision Transformer encoder.

[0034] In another possible implementation, the generation of the patient-specific knowledge representation is not fully automated, but rather adopts a human-machine collaborative approach: after the system automatically generates the initial knowledge graph, radiologists or surgeons review, correct, or supplement key anatomical relationships on a three-dimensional visualization interface, and the corrected results are then fed back to the knowledge graph construction model to optimize its parameters.

[0035] In another possible implementation, the preoperative medical data, in addition to imaging data, also integrates the patient's electronic medical record text information. Clinical entities and events are extracted from these data using natural language processing technology and added as attributes or relation nodes to the patient-specific knowledge representation, forming a more comprehensive preoperative file.

[0036] 103. Perception Data Acquisition: During the surgery, multimodal perception data of the surgical scene of the target object is acquired in real time.

[0037] In another possible implementation, the multimodal sensing data acquisition module is integrated with the existing equipment bus in the operating room, enabling it to synchronously acquire and correlate real-time status data from different brands of surgical equipment. This data is used as part of the context parameters to more accurately understand the surgical stage and the patient's condition.

[0038] In another possible implementation, the multimodal sensing data also includes audio signals from the surgical field, which are used to identify typical sound features of specific surgical instrument operations through sound event detection and are used as auxiliary timing information to infer the currently performed surgical step.

[0039] 104. Semantic understanding: Based on the hybrid knowledge model, perform semantic understanding on the multimodal perception data to generate a semantic representation of the intraoperative scene.

[0040] In another possible implementation, the semantic understanding of the multimodal perception data employs an incremental learning mechanism. When the system has low confidence in recognizing a certain intraoperative structure, the current image can be frozen in real time, and the surgeon can be asked to quickly annotate it through an interactive interface. The annotation result is immediately used to update the semantic representation of the current surgical scene and can be used as feedback data to optimize the understanding model for subsequent scenes.

[0041] In another possible implementation, the semantic understanding process not only identifies static anatomical structures, but also dynamically tracks and identifies surgically triggered "events," such as the formation of tissue separation planes, closure of blood vessels, and local bleeding areas. These events are then added as dynamic nodes to the semantic representation of the intraoperative scene to reflect the dynamic changes in the surgical process.

[0042] 105. Spatial association: Spatially associate the patient-specific knowledge representation with the semantic representation of the intraoperative scene.

[0043] In this embodiment, the spatial association includes: Extract the first set of feature points from the patient-specific knowledge representation; Extract the second set of feature points from the semantic representation of the intraoperative scene; Initial alignment is performed based on the first set of feature points and the second set of feature points; Based on the initial alignment results, the deformation field between the intraoperative scene semantic representation and the patient-specific knowledge representation is estimated.

[0044] In this embodiment, the estimation of the deformation field between the semantic representation of the intraoperative scene and the patient-specific knowledge representation is performed by a neural network.

[0045] In another possible implementation, the spatial association process incorporates physical deformation simulation as a constraint. A simplified biomechanical model based on the mechanical properties of patient tissue is constructed using the finite element method. Intraoperative instrument manipulation forces and tissue traction forces are used as inputs to predict tissue deformation trends. This prediction is then incorporated as a priori constraint into the estimation of the deformation field, making the registration results more consistent with physical laws.

[0046] In another possible implementation, the spatial association employs a hierarchical, multi-resolution strategy. First, low-resolution coarse registration based on contours is performed at the organ scale. Then, high-resolution fine registration based on feature points is performed in the region of interest. Finally, super-resolution local deformation optimization based on depth features is performed near key risk structures to balance overall efficiency and local accuracy.

[0047] 106. Information Output: Based on the spatial association results and the hybrid knowledge model, generate augmented reality navigation information and output it.

[0048] In this embodiment, generating augmented reality navigation information includes: Determine at least one of the following contextual parameters: current surgical stage, surgical instrument location, and distance to anatomical risk structures; Based on the rules corresponding to the context parameters in the hybrid knowledge model, determine the type and priority of the navigation content to be displayed; Based on the type and priority, corresponding visual enhancement information is generated as the augmented reality navigation information.

[0049] In another possible implementation, the presentation of the augmented reality navigation information is configurable. Surgeons can select different information presentation modes or customize warning visual styles for different risk levels before or during surgery, based on personal preferences and the type of surgery.

[0050] In another possible implementation, the output of the augmented reality navigation information is not limited to visual overlay. The system can drive multi-sensory channels to provide coordinated prompts based on the priority and type of the navigation information. For example, for the highest level of risk warning, while visually highlighting and flashing, a haptic feedback device can be triggered to cause micro-vibrations in surgical instruments, supplemented by a brief prompt from synthesized voice, forming a multi-dimensional warning loop.

[0051] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0052] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0053] Please see Figure 2 , Figure 2This is a schematic diagram of a surgical navigation device based on a hybrid knowledge model, provided in an embodiment of this application. The surgical navigation device 1 based on the hybrid knowledge model includes: a knowledge acquisition unit 11, a knowledge representation generation unit 12, a scene understanding unit 13, a spatial association unit 14, and a navigation output unit 15. Specifically: The knowledge acquisition unit 11 is used to initialize a hybrid knowledge model, which includes a formal surgical specification knowledge unit and a probabilistic anatomical mapping knowledge unit; it is also used to acquire the patient's multimodal perception data during surgery. The knowledge representation generation unit 12 is used to generate patient-specific knowledge representations based on the hybrid knowledge model and the patient's preoperative medical data. Scene understanding unit 13 is used to perform semantic understanding on the multimodal perception data based on the hybrid knowledge model, and generate a semantic representation of the intraoperative scene; Spatial association unit 14 is used to spatially associate the patient-specific knowledge representation with the semantic representation of the intraoperative scene; The navigation output unit 15 is used to generate augmented reality navigation information and output the augmented reality navigation information based on the spatial association results and the hybrid knowledge model.

[0054] Please see Figure 3 , Figure 3 A schematic diagram of the hardware architecture of an electronic device according to an embodiment of this application is shown. The electronic device 2 mainly consists of a processor 21 and a memory 22. In addition, the device may also include an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are interconnected via connecting components, which can be various interfaces, data lines, or communication buses, etc., and are not specifically specified in this embodiment.

[0055] Processor 21 may be one or more graphics processing units (GPUs). If processor 21 is a GPU, the GPU may be single-core or multi-core. Alternatively, processor 21 may also be a processor group consisting of multiple GPUs interconnected via one or more buses. Furthermore, the processor may be other types of processors, which are not specifically limited in this embodiment.

[0056] In another possible implementation, the processor 21 employs a heterogeneous computing architecture. This architecture includes a general-purpose central processing unit (CPU) as the control core and at least one field-programmable gate array (FPGA) as a dedicated acceleration unit. The FPGA is pre-configured to perform latency-sensitive critical tasks with fixed computational patterns in the method, such as real-time extraction of specific multi-scale image features from an endoscopic video stream, or performing specific convolution operations for deformation field prediction in the spatial correlation unit. The CPU is responsible for task scheduling, logic control, and execution of the remaining algorithmic steps. This architecture achieves extreme computational efficiency for the critical path while maintaining flexibility.

[0057] In another possible implementation, the processor 21's computing resources support dynamic virtualization and elastic allocation. Within a single physical electronic device, independent virtual processing environments can be partitioned for tasks with different security levels and real-time requirements. For example, real-time data acquisition and scene understanding tasks can be run in a high-priority real-time operating system partition to ensure strictly controllable computation cycles; while tasks with lower real-time requirements, such as knowledge reasoning and user interface management, can be run in another general-purpose operating system partition. The two parts exchange data through a secure internal communication mechanism, thereby achieving functional and security isolation on a single hardware platform.

[0058] The memory 22 is designed to store the instructions of a computer program and various program codes required to execute the present application. Optionally, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or optical disc read-only memory (CD-ROM), which are used to store related instructions and data.

[0059] In another possible implementation, the memory 22 integrates non-volatile memory (NVM) as a new storage layer. The NVM combines read / write speeds close to dynamic random access memory with the non-volatile characteristics of flash memory. The processor 21 can directly store frequently updated and rapidly persistent intermediate state data, such as real-time registration parameters of the current surgery and dynamically updated intraoperative scene semantic representations, in the NVM. This reduces frequent read / write operations to traditional storage media, improves the speed of system state saving and recovery, and enhances data security in the event of unexpected power outages.

[0060] In another possible implementation, the memory 22 employs a compute-memory integrated design. Simple computational logic units are integrated within a portion of the memory chip. For preprocessing or post-processing tasks exhibiting data locality characteristics, such as preliminary filtering of continuously arriving sensor data or batch coordinate transformation of pre-rendering 3D vertex data, these tasks can be performed locally by the computational units within the memory chip, and the results can then be returned to the processor 21. This design reduces the overhead of data transfer between the memory and the processor, lowering the overall system power consumption and bus load.

[0061] Input device 23 is used to input data and / or signals, and output device 24 is used to output data and / or signals. Input device 23 and output device 24 can be independent devices or an integrated device.

[0062] In another possible implementation, the input device 23 possesses intent understanding and proactive perception capabilities. The device not only passively receives instructions but also infers the surgeon's potential intentions or state of confusion through multimodal fusion. For example, when the system detects that the surgeon's gaze is focused on a structure not highlighted by the system for an extended period, it can proactively inquire through the output device 24 whether the structure needs to be labeled, or automatically call the knowledge acquisition unit 11 to query relevant anatomical knowledge of the area, achieving a proactive, context-driven interaction mode.

[0063] In another possible implementation, the output device 24 supports adaptive channels and cross-modal substitution. The device continuously monitors the availability of the main output channel. When the main channel is detected to be potentially overloaded or blocked, the system automatically and dynamically adjusts the priority and presentation of critical information. For example, the highest level of risk warning may be automatically switched from visual highlighting to, or superimposed with, tactile pulses (transmitted via networked smart surgical instruments or wearable devices) and brief synthesized voice prompts, ensuring that critical information is delivered to the surgeon without omission through the most efficient available channel.

[0064] Those skilled in the art will recognize that the components and algorithm steps of the various examples described in the embodiments disclosed herein can be implemented by electronic hardware or by a combination of computer software and electronic hardware. Whether these functions are implemented through hardware or software will be determined based on the specific application requirements and design constraints of the technical solution. Those skilled in the art can adopt different implementation methods according to the needs of each specific application, but such implementation should not be considered as exceeding the scope of protection of this application.

[0065] Those skilled in the art should understand that, for ease of description and simplification, the specific operational procedures of the aforementioned systems, devices, and components can be referred to the corresponding steps in the preceding method embodiments, and will not be repeated here. Furthermore, those skilled in the art should also understand that each embodiment in this application has its own focus, and for ease of description and simplification, the same or similar content may not be repeated in different embodiments. Therefore, if a part is not mentioned or not described in detail in a certain embodiment, it can be referred to the relevant description in other embodiments.

[0066] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can also be implemented through other means. For example, the described apparatus embodiments are merely exemplary, and the division of the units therein is only a logical functional division; different division methods may exist in actual implementation. For example, multiple units or components may be merged or integrated into another system, or certain features may be omitted, or certain steps may not be performed. Furthermore, the interconnections shown or discussed, whether direct or indirect, whether coupling or communication connections, may be implemented electrically, mechanically, or otherwise through interfaces, devices, or units.

[0067] Units described as independent components may or may not actually be physically separate; parts presented as units may or may not be physical entities, meaning they may be concentrated in one location or distributed across multiple network nodes. Depending on the actual needs, some or all of these units can be selected to achieve the objectives of this embodiment.

[0068] Furthermore, in the various embodiments of this application, each functional unit can be integrated into a single processing unit, exist independently, or two or more units can be merged into one unit. In the foregoing embodiments, the relevant functions can be fully or partially implemented through software, hardware, firmware, or any combination thereof. If software implementation is chosen, it can be implemented entirely or partially in the form of a computer program product. This computer program product contains one or more computer instructions. When these instructions are loaded and executed on a computer, they will produce all or part of the processes or functions described in the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. These computer instructions can be stored in computer-readable storage media or transmitted through such media. Computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, DSL) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any computer-accessible available medium, or a data storage facility such as a server or data center that integrates one or more available media. These available media may include magnetic media (such as floppy disks, hard disks, and magnetic tapes), optical media (such as DVDs), semiconductor media (such as SSDs), etc. Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by computer program instructions and related hardware, and these programs can be stored in computer-readable storage media. When these programs are executed, they will contain the processes of the above method embodiments. The aforementioned storage media include, but are not limited to, various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A surgical navigation method based on a hybrid knowledge model, characterized in that, The method includes: S1. Initialize the hybrid knowledge model, which includes formal surgical normative knowledge units and probabilistic anatomical mapping knowledge units; S2. Based on the hybrid knowledge model and the patient's preoperative medical data, generate a patient-specific knowledge representation; the generation of the patient-specific knowledge representation includes: inputting the preoperative medical data into a knowledge graph construction model; extracting anatomical entities contained in the preoperative medical data and semantic relationships between the anatomical entities through the knowledge graph construction model; and constructing the patient-specific knowledge representation based on the extracted anatomical entities and semantic relationships. S3. Acquire the patient's multimodal perception data during the operation; S4. Based on the hybrid knowledge model, perform semantic understanding on the multimodal perception data to generate a semantic representation of the intraoperative scene; S5. Spatially associate the patient-specific knowledge representation with the semantic representation of the intraoperative scene; S6. Based on the spatial association results and the hybrid knowledge model, generate augmented reality navigation information and output the augmented reality navigation information; the generation of augmented reality navigation information includes: determining at least one context parameter among the current surgical stage, surgical instrument position, and anatomical risk structure distance; determining the type and priority of the navigation content to be displayed according to the rules in the hybrid knowledge model corresponding to the context parameter; and generating corresponding visual enhancement information as the augmented reality navigation information based on the type and priority.

2. The method according to claim 1, characterized in that, The knowledge graph construction model includes a shared encoder, an entity decoder, and a relation decoder. The extraction of anatomical entities and semantic relationships between the anatomical entities contained in the preoperative medical data includes: The preoperative medical data is processed by the shared encoder to obtain feature representations; The entity decoding head outputs segmentation information of the anatomical entity based on the feature representation. Based on the feature representation and the segmentation information, the semantic relationship type between the anatomical entities is predicted using the relationship decoding head.

3. The method according to claim 2, characterized in that, The shared encoder is a 3D vision Transformer encoder.

4. The method according to claim 1, characterized in that, The spatial association includes: Extract the first set of feature points from the patient-specific knowledge representation; Extract the second set of feature points from the semantic representation of the intraoperative scene; Initial alignment is performed based on the first feature point group and the second feature point group; Based on the initial alignment results, the deformation field between the semantic representation of the intraoperative scene and the patient-specific knowledge representation is estimated using a neural network.

5. A surgical navigation device based on a hybrid knowledge model, characterized in that, The device includes: The knowledge acquisition unit is used to initialize a hybrid knowledge model, which includes a formal surgical specification knowledge unit and a probabilistic anatomical mapping knowledge unit; it is also used to acquire the patient's multimodal perception data during surgery. A knowledge representation generation unit is used to generate patient-specific knowledge representations based on the hybrid knowledge model and the patient's preoperative medical data. The generation of patient-specific knowledge representations includes: inputting the preoperative medical data into a knowledge graph construction model; extracting anatomical entities and semantic relationships between the anatomical entities contained in the preoperative medical data through the knowledge graph construction model; and constructing the patient-specific knowledge representations based on the extracted anatomical entities and semantic relationships. The scene understanding unit is used to perform semantic understanding on the multimodal perception data based on the hybrid knowledge model, and generate a semantic representation of the intraoperative scene; A spatial association unit is used to spatially associate the patient-specific knowledge representation with the semantic representation of the intraoperative scene. A navigation output unit is configured to generate and output augmented reality navigation information based on the spatial association results and the hybrid knowledge model. The generation of augmented reality navigation information includes: determining at least one context parameter among the current surgical stage, surgical instrument location, and distance to anatomical risk structures; determining the type and priority of the navigation content to be displayed according to the rules corresponding to the context parameters in the hybrid knowledge model; and generating corresponding visual enhancement information as the augmented reality navigation information based on the type and priority.

6. An electronic device, characterized in that, include: A processor and a storage unit for storing computer program code, the computer program code containing computer instructions, wherein when the processor executes the computer instructions, the electronic device performs the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program containing program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1 to 4.

Images