A multi-modal fusion-based intraoperative organ displacement real-time perception method and related device

Abstract

The application discloses a kind of intraoperative organ displacement real-time perception methods and related devices based on multi-modal fusion.The method comprises: first, synchronously acquire optical, electromagnetic and blood flow signal data;Then the magnetic field intensity characteristics of electromagnetic data are extracted, and selectively based on Kalman filtering and neural network weighted fusion filtering processing is carried out on electromagnetic data;Then, according to real-time signal quality, historical accuracy and the anatomical region attribute of target organ, dynamically allocate confidence weight for the three kinds of data;Finally, based on weight, spatial registration is carried out on multi-modal data, and the final pose data of organ is calculated.The application effectively improves the robustness, anti-interference ability and positioning accuracy of organ displacement perception in complex surgical environment through intelligent filtering and dynamic weight allocation.

Description

Technical Field

[0001] This invention relates to the fields of medicine, artificial intelligence and robotics, specifically to a method and related device for real-time intraoperative organ displacement sensing based on multimodal fusion. Background Technology

[0002] In image-guided interventional and surgical procedures, real-time and accurate sensing of the displacement of organs or lesions within the body is crucial, especially in scenarios where breathing, heartbeat, or instrument contact cause significant tissue deformation. Currently, intraoperative navigation primarily relies on technologies such as optical tracking and electromagnetic sensing. Optical tracking offers high accuracy but is susceptible to line-of-sight obstruction; electromagnetic sensing does not require direct visualization, but its signals are easily interfered with by the field strength generated by metal instruments or equipment in the operating room, leading to data drift and distortion. Existing solutions often employ single technologies or simple multi-sensor data overlay, lacking effective signal quality assessment and dynamic response mechanisms. For example, under strong magnetic field interference, the reliability of electromagnetic data drops sharply, yet the system may still fuse it with high-precision optical data with equal weight, introducing significant errors. Furthermore, the physiological characteristics of different anatomical regions (such as the highly vascularized porta hepatis) are not fully utilized to optimize sensing strategies. Therefore, current methods suffer from insufficient robustness, poor anti-interference capabilities, and decreased positioning accuracy in complex and dynamic surgical environments, making it difficult to meet the stringent requirements of real-time displacement sensing in high-precision surgery. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention relates to a method and related apparatus for real-time intraoperative organ displacement sensing based on multimodal fusion, including but not limited to a real-time intraoperative organ displacement sensing device, electronic device, computer-readable storage medium, and computer program product based on multimodal fusion.

[0004] Firstly, a method for real-time intraoperative organ displacement sensing based on multimodal fusion is provided, including:

[0005] S1. Acquire optical data, electromagnetic data, and blood flow signal data;

[0006] S2. Time synchronization is performed on the optical data, the electromagnetic data, and the blood flow signal data, and magnetic field strength features are extracted from the electromagnetic data; the time synchronization is achieved by sending a unified hardware clock synchronization signal through the central processing module;

[0007] S3. Compare the magnetic field strength characteristic with a preset threshold:

[0008] When the magnetic field strength characteristic exceeds the preset threshold, the electromagnetic data is filtered to obtain filtered electromagnetic data.

[0009] When the magnetic field strength characteristic does not exceed the preset threshold, the electromagnetic data is directly used as the filtered electromagnetic data.

[0010] S4. Based on real-time signal quality, historical accuracy, and the anatomical region attributes of the target organ, dynamically assign confidence weights to the optical data, the filtered electromagnetic data, and the blood flow signal data; the real-time signal quality is calculated based on the normalized signal-to-noise ratio of each of the optical data, electromagnetic data, and blood flow signal data; the historical accuracy is obtained based on the historical signal quality of each of the optical data, electromagnetic data, and blood flow signal data; the anatomical region attributes include the patient's organ and lesion category;

[0011] S5. Based on the confidence weight, spatial registration is performed on the optical data, the filtered electromagnetic data, and the blood flow signal data to calculate the final pose data.

[0012] In any embodiment of this application, the filtering process includes:

[0013] Use the following formula:

[0014] ,

[0015] The filtered electromagnetic data is calculated.

[0016] Where KF represents Kalman filtering and NN represents neural network prediction. As a weighting factor, For current electromagnetic data, For current optical data, This is historical electromagnetic data.

[0017] In any embodiment of this application, the weighting factor is calculated using a piecewise function: when the magnetic field strength feature is less than a first threshold (20 Gauss), the weighting factor is 0; when the magnetic field strength feature is between the first threshold and a second threshold (215 Gauss), the weighting factor increases linearly from 0 to 1 as the magnetic field strength increases; when the magnetic field strength feature exceeds the second threshold, the weighting factor is 1.

[0018] In conjunction with any embodiment of this application, when the anatomical region attribute is the porta hepatis, the confidence weight assigned to the blood flow signal data is dynamically increased.

[0019] In any embodiment of this application, the optical data comes from the identification of marker points; when some marker points are occluded, the position of the occluded marker points is predicted based on the remaining unoccluded marker points and their historical pose data, and pose calculation is performed.

[0020] In any embodiment of this application, the dynamic allocation of confidence weights is implemented through an adaptive decision model. The training and optimization of this model are based on a reinforcement learning framework, specifically including:

[0021] A reward function is constructed to evaluate the merits of the weight allocation strategy in real time; the reward function incorporates the pose estimation error generated by the current weight allocation.

[0022] Based on the reward function, the agent learns and dynamically updates the confidence weight allocation strategy online through continuous interaction with the surgical environment, with the goal of maximizing long-term cumulative rewards.

[0023] Secondly, a real-time intraoperative organ displacement sensing device based on multimodal fusion is provided, comprising:

[0024] Sensing unit: Acquires optical data, electromagnetic data, and blood flow signal data;

[0025] Synchronization unit: performs time synchronization on the optical data, the electromagnetic data, and the blood flow signal data, and extracts magnetic field strength features from the electromagnetic data; the time synchronization is achieved by sending a unified hardware clock synchronization signal through the central processing module;

[0026] Filtering unit: compares the magnetic field strength characteristics with a preset threshold.

[0027] When the magnetic field strength feature exceeds the preset threshold, the electromagnetic data is filtered to obtain filtered electromagnetic data; the filtering process is based on a weighted fusion of Kalman filtering and neural network; the weighting factor of the weighted fusion is calculated based on the magnetic field strength feature.

[0028] When the magnetic field strength characteristic does not exceed the preset threshold, the electromagnetic data is directly used as the filtered electromagnetic data.

[0029] Weighting unit: Based on real-time signal quality, historical accuracy, and the anatomical region attributes of the target organ, it dynamically assigns confidence weights to the optical data, the filtered electromagnetic data, and the blood flow signal data; the real-time signal quality is calculated based on the normalized signal-to-noise ratio of each of the optical data, electromagnetic data, and blood flow signal data; the historical accuracy is obtained based on the historical signal quality of each of the optical data, electromagnetic data, and blood flow signal data; the anatomical region attributes include the patient's organ and lesion category;

[0030] Fusion unit: Based on the confidence weight, spatially registers the optical data, the filtered electromagnetic data, and the blood flow signal data to calculate the final pose data.

[0031] Thirdly, an electronic device is provided, comprising: a processor, a communication module, a sensor, a user interface, and a storage unit, wherein the storage unit is used to store computer program code, the program code including computer instructions. When the processor executes these instructions, the electronic device performs the methods described in the second aspect above and any of its embodiments.

[0032] Fourthly, another electronic device is provided, comprising: a processor, a wireless communication module, a touchscreen, a speaker, and a storage unit, wherein the storage unit is used to store computer program code, the program code including computer instructions. When the processor executes these instructions, the electronic device performs the methods described in the second aspect above and any of its embodiments.

[0033] Fifthly, a computer-readable storage medium is provided, wherein a computer program is stored, the program comprising program instructions. When these instructions are executed by a processor, the processor performs the methods described in the second aspect above and any of its embodiments.

[0034] In a sixth aspect, a computer program product is provided, the computer program product comprising a computer program or instructions. When the computer program or instructions are executed on a computer, the computer will perform the methods described in the second aspect above and any of its embodiments.

[0035] In this application, compared with the prior art, the present invention provides a method and related device for real-time intraoperative organ displacement sensing based on multimodal fusion. The present invention simultaneously acquires optical, electromagnetic, and blood flow signal data, and performs intelligent filtering on the electromagnetic data based on magnetic field strength characteristics. Subsequently, it dynamically assigns confidence weights to each modality of data according to real-time signal quality, historical accuracy, and the anatomical region attributes of the target organ. Finally, it performs spatial registration of the multimodal data based on the weights to calculate the final pose data of the organ. By introducing an adaptive filtering and dynamic weight allocation mechanism, the present invention effectively solves the problems of poor anti-interference ability and insufficient robustness of existing technologies in complex surgical environments, significantly improving the real-time performance and positioning accuracy of organ displacement sensing. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to illustrate the technical solutions of this application.

[0038] Figure 1A flowchart illustrating a method for real-time intraoperative organ displacement sensing based on multimodal fusion, provided in an embodiment of this application.

[0039] Figure 2 This is a schematic diagram of a real-time intraoperative organ displacement sensing device based on multimodal fusion, provided as an embodiment of this application.

[0040] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] To enable those skilled in the art to more fully understand the technical solutions of this application, the technical solutions of this application will be explained in detail and clearly with reference to the accompanying drawings. It should be particularly noted that the described embodiments are only some examples of this application and do not represent all of them. Based on these embodiments, those skilled in the art can directly deduce all other possible implementation schemes without creative thinking, and these are also included within the protection scope of this application.

[0042] In the specification, claims, and related drawings of this application, the terms "first," "second," etc., are used only to distinguish different elements and do not imply any specific order. Furthermore, the use of "comprising" and "having," and their variations, indicates non-exclusive inclusion. This means that if a process, method, system, product, or device comprises a series of steps or components, it indicates that the process, method, system, product, or device is not limited to the listed steps or components and may also include other steps or components not listed, or other inherent steps or units thereof.

[0043] The term "embodiment" as used herein refers to any instance combining a particular feature, structure, or characteristic, which may be at least one embodiment of this application. The "embodiments" mentioned herein do not necessarily refer to the same specific case, nor do they imply that they are independent or exclusive alternatives. Those skilled in the art will understand that the embodiments described herein can be used with other embodiments. It should be clarified that in this application, "at least one" includes one or more instances, "multiple" means two or more instances, and "at least two" refers to two or more instances.

[0044] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The embodiments of this application will now be described with reference to the accompanying drawings.

[0045] Please see Figure 1 , Figure 1 A flowchart illustrating a method for real-time intraoperative organ displacement sensing based on multimodal fusion, provided in an embodiment of this application.

[0046] 101. Synchronous sensing: Acquiring optical data, electromagnetic data, and blood flow signal data.

[0047] In this embodiment, the time synchronization is achieved by sending a unified hardware clock synchronization signal through the central processing module.

[0048] In this embodiment, the optical data comes from the identification of marker points; when some marker points are occluded, the position of the occluded marker points is predicted based on the remaining unoccluded marker points and their historical pose data, and pose calculation is performed.

[0049] 102. Time synchronization: The optical data, the electromagnetic data, and the blood flow signal data are synchronized in time.

[0050] In this embodiment, a dual synchronization mechanism based on hardware triggering and software interpolation compensation is adopted to ensure that the timestamp alignment accuracy of the optical data, the electromagnetic data and the blood flow signal data meets the millisecond-level requirement.

[0051] 103. Anti-interference filtering: Extract magnetic field strength features from the electromagnetic data and compare the magnetic field strength features with a preset threshold.

[0052] In this embodiment, when the magnetic field strength characteristic exceeds the preset threshold, the electromagnetic data is filtered to obtain filtered electromagnetic data.

[0053] In this embodiment, when the magnetic field strength characteristic does not exceed the preset threshold, the electromagnetic data is directly used as the filtered electromagnetic data.

[0054] In this embodiment, the filtering process includes:

[0055] Use the following formula:

[0056] ,

[0057] The filtered electromagnetic data is calculated.

[0058] Where KF represents Kalman filtering and NN represents neural network prediction. As a weighting factor, For current electromagnetic data, For current optical data, This is historical electromagnetic data.

[0059] In this embodiment, when the magnetic field strength feature is less than the first threshold, the weighting factor is 0; when the magnetic field strength feature is between the first threshold and the second threshold, the weighting factor increases linearly from 0 to 1 as the magnetic field strength increases; when the magnetic field strength feature exceeds the second threshold, the weighting factor is 1.

[0060] In this embodiment, the first threshold is 20 Gauss and the second threshold is 215 Gauss.

[0061] 104. Weighting: Based on real-time signal quality, historical accuracy, and anatomical region attributes of the target organ, reliability weights are dynamically assigned to the optical data, the filtered electromagnetic data, and the blood flow signal data.

[0062] In this embodiment, the real-time signal quality is calculated based on the normalized signal-to-noise ratio of each of the optical data, the electromagnetic data, and the blood flow signal data.

[0063] In this embodiment, the historical accuracy is obtained based on the historical signal quality of the optical data, the electromagnetic data, and the blood flow signal data, respectively.

[0064] In this embodiment, the anatomical region attributes include the patient's organs and the categories of lesions.

[0065] In this embodiment, when the anatomical region attribute is the porta hepatis, the confidence weight assigned to the blood flow signal data is dynamically increased.

[0066] In this embodiment, the dynamic allocation of confidence weights is implemented through an adaptive decision model. The training and optimization of this model are based on a reinforcement learning framework, specifically including:

[0067] A reward function is constructed to evaluate the merits of the weight allocation strategy in real time; the reward function incorporates the pose estimation error generated by the current weight allocation.

[0068] Based on the reward function, the agent learns and dynamically updates the confidence weight allocation strategy online through continuous interaction with the surgical environment, with the goal of maximizing long-term cumulative rewards.

[0069] 105. Weighted Fusion: Based on the confidence weights, spatial registration is performed on the optical data, the filtered electromagnetic data, and the blood flow signal data to calculate the final pose data.

[0070] In this embodiment, a quaternion-based spherical linear interpolation method is used to spatially register data from different coordinate systems.

[0071] In this embodiment, the final pose data is obtained by solving the generalized inverse matrix weighted by the confidence weight, and the pose information of six degrees of freedom (including the linear velocity of the x, y, and z axes and the angular velocity of rotation about the x, y, and z axes) and its corresponding covariance matrix are output for uncertainty assessment.

[0072] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0073] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0074] Please see Figure 2 , Figure 2 This application provides a schematic diagram of a real-time intraoperative organ displacement sensing device based on multimodal fusion. The sensing device 1 includes: a sensing unit 11, a synchronization unit 12, a filtering unit 13, a weighting unit 14, and a fusion unit 15. Specifically:

[0075] Sensing unit 11: acquires optical data, electromagnetic data, and blood flow signal data; the time synchronization is achieved by sending a unified hardware clock synchronization signal through the central processing module;

[0076] Synchronization unit 12: performs time synchronization on the optical data, the electromagnetic data and the blood flow signal data, and extracts magnetic field strength features from the electromagnetic data;

[0077] Filtering unit 13: compares the magnetic field strength characteristics with a preset threshold.

[0078] When the magnetic field strength feature exceeds the preset threshold, the electromagnetic data is filtered to obtain filtered electromagnetic data; the filtering process is based on a weighted fusion of Kalman filtering and neural network; the weighting factor of the weighted fusion is calculated based on the magnetic field strength feature.

[0079] When the magnetic field strength characteristic does not exceed the preset threshold, the electromagnetic data is directly used as the filtered electromagnetic data.

[0080] Weighting unit 14: Based on real-time signal quality, historical accuracy, and the anatomical region attributes of the target organ, dynamically assigns confidence weights to the optical data, the filtered electromagnetic data, and the blood flow signal data; the real-time signal quality is calculated based on the normalized signal-to-noise ratio of each of the optical data, electromagnetic data, and blood flow signal data; the historical accuracy is obtained based on the historical signal quality of each of the optical data, electromagnetic data, and blood flow signal data; the anatomical region attributes include the patient's organ and lesion category;

[0081] Fusion unit 15: Based on the confidence weight, spatially registers the optical data, the filtered electromagnetic data, and the blood flow signal data to calculate the final pose data.

[0082] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0083] Please see Figure 3 , Figure 3 A schematic diagram of the hardware architecture of an electronic device according to an embodiment of this application is shown. The electronic device 2 mainly consists of a processor 21 and a memory 22. In addition, the device may also include an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are interconnected via connecting components, which can be various interfaces, data lines, or communication buses, etc., and are not specifically specified in this embodiment.

[0084] Processor 21 may be one or more graphics processing units (GPUs). If processor 21 is a GPU, the GPU may be single-core or multi-core. Alternatively, processor 21 may also be a processor group consisting of multiple GPUs interconnected via one or more buses. Furthermore, the processor may be other types of processors, which are not specifically limited in this embodiment.

[0085] The memory 22 is designed to store the instructions of a computer program and various program codes required to execute the present application. Optionally, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or optical disc read-only memory (CD-ROM), which are used to store related instructions and data.

[0086] Input device 23 is used to input data and / or signals, and output device 24 is used to output data and / or signals. Input device 23 and output device 24 can be independent devices or an integrated device.

[0087] It should be understood that, in the embodiments of this application, the memory 22 can store not only related instructions but also related data. The embodiments of this application do not specify the specific data content stored in the memory.

[0088] It should be understood that Figure 3This illustration only shows a simplified design of an electronic device. In actual use, the electronic device may also include other necessary components, such as different numbers of input / output devices, processors, memory, etc. All electronic devices capable of implementing the embodiments of this application are within the protection scope of this application.

[0089] Those skilled in the art will recognize that the components and algorithm steps of the various examples described in the embodiments disclosed herein can be implemented by electronic hardware or by a combination of computer software and electronic hardware. Whether these functions are implemented through hardware or software will be determined based on the specific application requirements and design constraints of the technical solution. Those skilled in the art can adopt different implementation methods according to the needs of each specific application, but such implementation should not be considered as exceeding the scope of protection of this application.

[0090] Those skilled in the art should understand that, for ease of description and simplification, the specific operational procedures of the aforementioned systems, devices, and components can be referred to the corresponding steps in the preceding method embodiments, and will not be repeated here. Furthermore, those skilled in the art should also understand that each embodiment in this application has its own focus, and for ease of description and simplification, the same or similar content may not be repeated in different embodiments. Therefore, if a part is not mentioned or not described in detail in a certain embodiment, it can be referred to the relevant description in other embodiments.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can also be implemented through other means. For example, the described apparatus embodiments are merely exemplary, and the division of the units therein is only a logical functional division; different division methods may exist in actual implementation. For example, multiple units or components may be merged or integrated into another system, or certain features may be omitted, or certain steps may not be performed. Furthermore, the interconnections shown or discussed, whether direct or indirect, whether coupling or communication connections, may be implemented electrically, mechanically, or otherwise through interfaces, devices, or units.

[0092] Units described as independent components may or may not actually be physically separate; parts presented as units may or may not be physical entities, meaning they may be concentrated in one location or distributed across multiple network nodes. Depending on the actual needs, some or all of these units can be selected to achieve the objectives of this embodiment.

[0093] Furthermore, in the various embodiments of this application, each functional unit can be integrated into a single processing unit, exist independently, or two or more units can be merged into one unit. In the foregoing embodiments, the relevant functions can be fully or partially implemented through software, hardware, firmware, or any combination thereof. If software implementation is chosen, it can be implemented entirely or partially in the form of a computer program product. This computer program product contains one or more computer instructions. When these instructions are loaded and executed on a computer, they will produce all or part of the processes or functions described in the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. These computer instructions can be stored in computer-readable storage media or transmitted through such media. Computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, DSL) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any computer-accessible available medium, or a data storage facility such as a server or data center that integrates one or more available media. These available media may include magnetic media (such as floppy disks, hard disks, and magnetic tapes), optical media (such as DVDs), semiconductor media (such as SSDs), etc. Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by computer program instructions and related hardware, and these programs can be stored in computer-readable storage media. When these programs are executed, they will contain the processes of the above method embodiments. The aforementioned storage media include, but are not limited to, various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A computer program product, characterized in that, The computer program product comprises a computer program or instructions which, when executed on a computer, cause the computer to perform a multi-modal fusion-based intraoperative organ displacement real-time perception method, the method comprising: S1. acquiring optical data, electromagnetic data, and blood flow signal data; S2. time-synchronizing the optical data, the electromagnetic data, and the blood flow signal data, and extracting magnetic field intensity features from the electromagnetic data; the time synchronization is achieved by sending a unified hardware clock synchronization signal by a central processing module; S3. comparing the magnetic field intensity features with a preset threshold value: when the magnetic field intensity features exceed the preset threshold value, filtering the electromagnetic data to obtain filtered electromagnetic data; when the magnetic field intensity features do not exceed the preset threshold value, directly using the electromagnetic data as the filtered electromagnetic data; S4. dynamically assigning confidence weights to the optical data, the filtered electromagnetic data, and the blood flow signal data based on real-time signal quality, historical accuracy, and anatomical region attributes of the target organ; the real-time signal quality is calculated based on the normalized signal-to-noise ratio of each of the optical data, the electromagnetic data, and the blood flow signal data; the historical accuracy is obtained based on the historical signal quality of each of the optical data, the electromagnetic data, and the blood flow signal data; the anatomical region attributes include the categories of patient organs and lesions; S5. spatially registering the optical data, the filtered electromagnetic data, and the blood flow signal data according to the confidence weights, and calculating the final pose data.

2. The computer program product of claim 1, wherein, The filtering process includes: using the following formula: ， to calculate the filtered electromagnetic data; wherein KF represents a Kalman filter processing, and NN represents a neural network prediction processing, is a weight factor, is current electromagnetic data, is current optical data, is historical electromagnetic data.

3. The computer program product of claim 2, wherein, The calculation of the weight factor uses a piecewise function: when the magnetic field intensity features are less than a first threshold value, the weight factor is 0; when the magnetic field intensity features are between the first threshold value and a second threshold value, the weight factor linearly increases from 0 to 1 as the magnetic field intensity increases; when the magnetic field intensity features exceed the second threshold value, the weight factor is 1; the first threshold value is 20 Gauss, and the second threshold value is 215 Gauss.

4. The computer program product of claim 1, wherein, When the anatomical region attribute is the hepatic portal area, the confidence weight assigned to the blood flow signal data is dynamically increased.

5. The computer program product of claim 1, wherein, The optical data comes from the identification of marker points; when some marker points are blocked, the positions of the blocked marker points are predicted based on the remaining unblocked marker points and their historical pose data for pose solving.

6. The computer program product of claim 1, wherein, The dynamic assignment of confidence weights is achieved through an adaptive decision-making model, the training and optimization of which is based on a reinforcement learning framework, specifically including: constructing a reward function for real-time evaluation of the pros and cons of the weight allocation strategy; the reward function incorporates the pose estimation error resulting from the current weight allocation; based on the reward function, through continuous interaction between the agent and the surgical environment, aiming to maximize the long-term cumulative reward, the allocation strategy of the confidence weight is learned and dynamically updated online.

7. A multi-modal fusion based intraoperative organ displacement real-time perception device, characterized in that, including: perception unit: acquiring optical data, electromagnetic data, and blood flow signal data; The synchronization unit: time synchronization is performed on the optical data, the electromagnetic data and the blood flow signal data, and a magnetic field intensity feature is extracted from the electromagnetic data; the time synchronization is realized by sending a unified hardware clock synchronization signal by a central processing module; The filtering unit: the magnetic field intensity feature is compared with a preset threshold value: When the magnetic field intensity feature exceeds the preset threshold value, the electromagnetic data is filtered to obtain filtered electromagnetic data; the filtering is based on a weighted fusion of Kalman filtering and a neural network; a weight factor of the weighted fusion is calculated based on the magnetic field intensity feature; When the magnetic field intensity feature does not exceed the preset threshold value, the electromagnetic data is directly used as the filtered electromagnetic data; The weighting unit: a confidence weight is dynamically assigned to the optical data, the filtered electromagnetic data and the blood flow signal data based on real-time signal quality, historical accuracy and anatomical region attributes of a target organ; The real-time signal quality is calculated based on a normalized signal-to-noise ratio of each of the optical data, the electromagnetic data and the blood flow signal data; the historical accuracy is obtained based on historical signal quality of each of the optical data, the electromagnetic data and the blood flow signal data; and the anatomical region attributes include a category of a patient organ and a lesion; The fusion unit: according to the confidence weight, spatial registration is performed on the optical data, the filtered electromagnetic data and the blood flow signal data to calculate final pose data.

8. An electronic device, comprising: The processor and a storage unit for storing computer program code, the code containing computer instructions, when the processor executes the computer instructions, the electronic device executes the method executed by the computer program product of any one of claims 1 to 6. The computer readable storage medium stores a computer program, and the computer program contains program instructions. When the program instructions are executed by the processor, the processor executes the method executed by the computer program product of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, ​

Images