Flexible surgical instrument-oriented multi-source sensing data adaptive fusion method and related device

Abstract

The invention discloses a flexible surgical instrument-oriented multi-source sensing data adaptive fusion method and a related device. The method comprises the following steps: acquiring optical, mechanical and pose multi-mode sensing data based on a timestamp synchronization technology in a working environment; clock compensation and space coordinate transformation based on an instrument kinematics model are carried out on the multi-modal data, and space-time registration is completed; inputting the registered data into a neural radiation field model to generate continuous space implicit scene representation; inputting the implicit representation into a pre-trained RWKV fusion network to obtain a fusion feature; and superposing the fusion feature and the original low-frequency component through residual connection, and integrating to generate three-dimensional voxelization environment state data. According to the method, multi-source information high-fidelity fusion is achieved through a single path, data redundancy is remarkably reduced, and the real-time performance and precision of intra-operative environment perception are improved.

Description

Technical Field

[0001] The present invention relates to the field of medical devices and signal processing, and specifically to a multi-source perception data adaptive fusion method and related devices for flexible surgical instruments. Background Art

[0002] Currently, flexible surgical instruments are increasingly used in delicate operations such as minimally invasive, laparoscopic, and vascular intervention. However, due to the flexible structure of the instruments themselves and the complex anatomical environment, intraoperative perception still relies on single-modality imaging or force feedback, resulting in information fragmentation and high latency, making it difficult for doctors to grasp the real spatial relationship and interactive mechanical state between the instrument end and the surrounding tissue in real time. Existing methods often process optical, force, and posture data separately and then simply splice them together. This lacks a unified spatiotemporal reference and is prone to registration errors during rapid movement or tissue deformation. At the same time, traditional explicit reconstruction requires dense point clouds, which are computationally intensive and memory intensive, making it difficult to implement on surgical-level low-latency, low-power terminals. In addition, there is a lack of large-scale simulation pre-training data for surgical scenarios, and the model is not generalizable enough, further restricting accurate and safe intraoperative navigation and intelligent control. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention relates to a multi-source perception data adaptive fusion method and related devices for flexible surgical instruments, which include but are not limited to multi-source perception data adaptive fusion equipment, electronic devices, computer-readable storage media and computer program products for flexible surgical instruments.

[0004] In a first aspect, a method for adaptively fusing multi-source perception data for flexible surgical instruments is provided, comprising: a. In a working environment, collect multimodal perception data based on timestamp synchronization technology; b. registering the multimodal perception data to obtain registration data; c. generating an implicit scene representation of the registration data using neural radiance field reconstruction technology; d. Input the implicit scene representation into the pre-trained RWKV fusion network to obtain fusion features; e. Perform residual connection post-processing on the fusion features and integrate them with the multimodal perception data to obtain environmental state data.

[0005] In combination with any embodiment of the present application, the acquisition of multimodal perception data is based on optical sensors, mechanical sensors and posture sensors; the timestamp synchronization uses hardware clock signals to align the acquisition cycles of each sensor.

[0006] In combination with any embodiment of the present application, the registration of the multimodal sensing data is spatiotemporal registration, including clock compensation and spatial coordinate transformation; the spatial coordinate transformation is based on a kinematic model of the instrument.

[0007] In conjunction with any embodiment of the present application, step c includes: inputting the registered data into a neural radiation field model; The neural radiance field model outputs a continuous spatial field representation as an implicit scene representation.

[0008] In combination with any embodiment of the present application, the pre-training is completed using a surgical scene simulation data set.

[0009] In combination with any embodiment of the present application, the residual connection post-processing superimposes the fusion feature with the low-frequency component of the multimodal perception data; and the integration includes generating three-dimensional voxelized environmental state data.

[0010] In a second aspect, a multi-source perception data adaptive fusion device for flexible surgical instruments is provided, the device comprising: Perception unit: used to collect multimodal perception data based on timestamp synchronization technology in the working environment; A registration unit is configured to register the multimodal perception data to obtain registration data; and is further configured to generate an implicit scene representation for the registration data using a neural radiation field reconstruction technique; Fusion unit: used for inputting the implicit scene representation into the pre-trained RWKV fusion network to obtain fusion features; Output unit: used to perform residual connection post-processing on the fusion feature and integrate it with the multimodal perception data to obtain environmental state data.

[0011] In a third aspect, an electronic device is provided, comprising: a processor, a communication module, a sensor, a user interface, and a storage unit, wherein the storage unit is configured to store computer program code, wherein the program code comprises computer instructions. When the processor executes these instructions, the electronic device performs the method described in the second aspect and any embodiment thereof.

[0012] In a fourth aspect, another electronic device is provided, comprising: a processor, a wireless communication module, a touch screen, a speaker, and a storage unit, wherein the storage unit is configured to store computer program code, wherein the program code comprises computer instructions. When the processor executes these instructions, the electronic device performs the method described in the second aspect and any embodiment thereof.

[0013] In a fifth aspect, a computer-readable storage medium is provided, wherein a computer program is stored, wherein the program includes program instructions. When these instructions are executed by a processor, the processor will perform the method described in the second aspect and any embodiment thereof.

[0014] In a sixth aspect, a computer program product is provided, wherein the computer program product comprises a computer program or instructions. When the computer program or instructions are run on a computer, the computer will execute the method described in the second aspect and any embodiment thereof.

[0015] It should be understood that the above general description and the following detailed description are only used as examples and explanations and do not limit the present application in any way.

[0016] Compared to the prior art, this invention provides a method and related device for adaptive fusion of multi-source sensor data for flexible surgical instruments. This method uses timestamp synchronization technology to simultaneously collect heterogeneous optical, mechanical, and pose data from multiple sources. It then performs spatiotemporal registration using spatial coordinate transformation and clock compensation based on the instrument's kinematic model, addressing the geometric drift problem caused by inconsistent sensor time bases in traditional methods. The registered data is then fed into a neural radiation field model to generate a continuous implicit spatial representation, replacing traditional explicit point clouds or voxels and significantly compressing the data volume. This implicit representation is then inferred using a RWKV fusion network pre-trained on a surgical scene simulation dataset to generate high-dimensional fused features. These features are then superimposed with the original low-frequency components through residual connections to ultimately output three-dimensional voxelized environmental state data. Compared to the prior art, which suffers from slow surface rendering, high computational power consumption for volume rendering, and large splicing errors, this method achieves high-fidelity fusion of multimodal information using a single path, improving rendering speed and reducing hardware resource usage. It can run in real time on low-power intraoperative terminals, providing a reliable basis for the safe navigation and precise control of flexible instruments within complex cavities. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0018] The drawings herein are incorporated into the specification and constitute a part of this specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0019] Figure 1 A flowchart of a method for adaptively fusion of multi-source perception data for flexible surgical instruments proposed in an embodiment of the present application.

[0020] Figure 2Schematic diagram of a multi-source perception data adaptive fusion device for flexible surgical instruments proposed in an embodiment of the present application.

[0021] Figure 3 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In order to allow professionals in this technical field to more fully understand the technical solution of the present application, the technical solution of the present application will be explained in detail and clearly with the help of the accompanying drawings. It should be noted that the described embodiments are only some examples of the present application and do not represent all. Based on these embodiments, those skilled in the art can directly deduce all other possible implementation plans without engaging in creative thinking, and these are also included in the scope of protection of the present application.

[0023] In the specification, claims, and related drawings of this application, the terms "first," "second," and the like are used solely to distinguish between different elements and do not imply any particular order. Furthermore, the use of "including," "having," and their variations denotes non-exclusive inclusion. This means that if a process, method, system, product, or apparatus includes a series of steps or components, the process, method, system, product, or apparatus is not limited to the enumerated steps or components and may include other steps or components not listed, or other steps or units inherent to the process, method, system, product, or apparatus.

[0024] The “embodiment” mentioned in this document refers to any instance in which a particular feature, structure or characteristic is combined, and these instances may belong to at least one embodiment of the present application. The “embodiment” mentioned in this document does not necessarily refer to the same specific case, nor does it mean that they are independent or exclusive alternatives. It should be understood by those skilled in the art that the embodiments described herein can be used in conjunction with other embodiments. It should be understood that in this application, “at least one” includes one or more instances, “a plurality” means two or more instances, and “at least two” means two or more instances.

[0025] It should be understood that the method embodiment of the present application can also be implemented by a processor executing computer program code. The embodiment of the present application is described below in conjunction with the drawings in the embodiment of the present application.

[0026] See also Figure 1 , Figure 1 A flowchart of a method for adaptively fusion of multi-source perception data for flexible surgical instruments provided in an embodiment of the present application.

[0027] 101. Synchronous acquisition of intraoperative data: In the working environment, multimodal perception data is collected based on timestamp synchronization technology.

[0028] In this embodiment, the IEEE 1588 protocol is used to lock the sampling clocks of the optical camera, MEMS force sensor, and IMU pose sensor, ensuring that cross-modal timestamp errors are controlled within the same period. The clock signal is divided by the FPGA to trigger the synchronous exposure of each sensor, thereby outputting data frames with global time stamps.

[0029] In another possible implementation, the multimodal sensing data is synchronously collected through a hardware trigger signal.

[0030] In another possible implementation, a GNSS timing module is used to send PPS pulses to each sensor, which is then combined with a local clock counter to achieve soft synchronization. Alternatively, a PTP hardware clock is embedded in the sensor end, and a transparent clock mechanism is used to compensate for link delays, thereby obtaining a synchronized timestamp.

[0031] 102. Multi-source perception registration: registering the multimodal perception data to obtain registration data.

[0032] In this embodiment, the local coordinates of the force sensor are mapped to the world coordinates of the optical camera using the forward kinematic model of the instrument. The spatial deviation is then compensated using an iterative closest point algorithm. The clock drift is then corrected frame by frame through linear interpolation to achieve spatiotemporal consistency alignment.

[0033] In another possible implementation, the external parameters of the optical camera and the end of the instrument are first solved through hand-eye calibration, and then the extended Kalman filter is used to fuse the IMU data and the force sensor readings to achieve online estimation and compensation of the kinematic model error and complete the alignment.

[0034] 103. Scene representation generation: Generate an implicit scene representation for the registration data using neural radiation field reconstruction technology.

[0035] In this embodiment, the registration data is input into the neural radiance field, and the density and color are integrated in the continuous space using position encoding and learnable implicit functions to output a gridless implicit scene representation to support arbitrary viewpoint queries.

[0036] In another possible implementation, other technologies are used to perform implicit scene representation on the registration data.

[0037] In another possible implementation, the truncated signed distance function is used to voxelize the registered point cloud to construct a discrete TSDF volume; or the point cloud is directly converted into a triangular mesh using differentiable mesh reconstruction as an explicit scene representation.

[0038] 104. Multi-source feature fusion: The implicit scene representation is input into the pre-trained RWKV fusion network to obtain fusion features.

[0039] In this embodiment, the implicit scene representation is flattened into a sequence and then fed into the RWKV network. The RWKV network integrates cross-modal context based on time mixing and channel mixing mechanisms, and outputs fusion features of unified dimensions. The fusion features are then residually superimposed with the original low-frequency components to preserve details.

[0040] In another possible implementation, the implicit scene representation is projected into a two-dimensional feature map according to the perspective. After extracting local features through a convolutional neural network, a Transformer encoder is used to perform inter-modal attention interaction and output fused features. Alternatively, a bidirectional LSTM can be used to process serialized features to achieve information integration.

[0041] 105. Environmental state output: Perform residual connection post-processing on the fused features and integrate them with the multimodal perception data to obtain environmental state data.

[0042] In this embodiment, the fused features are resampled into regular voxels through trilinear interpolation to generate a three-dimensional voxel body containing density, color and force information; the three-dimensional voxel body is directly used as environmental state data for subsequent navigation or control modules to call.

[0043] In another possible implementation, the fused features are mapped to sparse octree nodes along a spatial hash function, retaining non-empty leaf node data to form a sparse body structure; or the environmental state data is encoded as a sign-distance field tensor to support subsequent field-based queries and operations.

[0044] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0045] The above describes in detail the method of the embodiment of the present application, and the following provides an apparatus of the embodiment of the present application.

[0046] See also Figure 2 , Figure 2 This is a schematic diagram of a multi-source perception data adaptive fusion device for flexible surgical instruments proposed in an embodiment of the present application. The multi-source perception fusion device 1 includes: a perception unit 11, a registration unit 12, a fusion unit 13, and an output unit 14. Specifically: Perception unit 11: used to collect multimodal perception data based on timestamp synchronization technology in a working environment; Registration unit 12: used to register the multimodal perception data to obtain registration data; and further used to generate implicit scene representation from the registration data using neural radiation field reconstruction technology; Fusion unit 13: used for inputting the implicit scene representation into the pre-trained RWKV fusion network to obtain fusion features; Output unit 14: used to perform residual connection post-processing on the fusion features and integrate them with the multimodal perception data to obtain environmental status data.

[0047] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0048] See also Figure 3 , Figure 3 The following is a schematic diagram of the hardware architecture of an electronic device described in an embodiment of the present application. The electronic device 2 is primarily composed of a processor 21 and a memory 22. In addition, the device may also include an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are interconnected via connecting components, which may be various interfaces, data cables, or communication buses, and are not specifically specified in the present embodiment.

[0049] Processor 21 may be one or more graphics processing units (GPUs). If processor 21 is a GPU, the GPU may be single-core or multi-core. Optionally, processor 21 may comprise a processor group consisting of multiple GPUs, interconnected via one or more buses. Furthermore, the processor may be other types of processors, which are not specifically limited in this embodiment of the present application.

[0050] Memory 22 is designed to store computer program instructions and various program codes required to execute the present invention. Optionally, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which are used to store relevant instructions and data.

[0051] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0052] It should be appreciated that in the embodiment of the present application, the memory 22 can store not only relevant instructions but also relevant data. The embodiment of the present application does not specify the specific data content stored in the memory.

[0053] You should understand that Figure 3Only a simplified design of an electronic device is shown. In actual use, the electronic device may also include other necessary components, such as different numbers of input / output devices, processors, memories, etc. All electronic devices that can implement the embodiments of this application are within the scope of protection of this application.

[0054] Those skilled in the art will recognize that, according to the components and algorithm steps of each example described in the embodiments disclosed herein, these functions can be implemented by electronic hardware or by combining computer software and electronic hardware. Whether these functions are performed by hardware or software will be determined based on the specific application requirements and design limitations of the technical solution. Technicians can adopt different implementation methods according to the requirements of each specific application, but such implementation methods should not be considered to exceed the scope of protection of this application.

[0055] Professionals should understand that, for the sake of ease of description and simplification, the specific operating procedures of the above-mentioned systems, devices, and components can refer to the corresponding steps in the previous method embodiments and will not be repeated here. At the same time, professionals should also understand that each embodiment in this application has its own focus. For the sake of ease of description and simplification, the same or similar content may not be repeated in different embodiments. Therefore, if a part is not mentioned or not explained in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0056] In the several embodiments provided in this application, it should be recognized that the disclosed systems, devices and methods can also be implemented in other ways. For example, the device embodiments described are only exemplary, in which the division of the units is only a division of logical functions, and there may be different division methods in actual implementation. For example, multiple units or components may be merged or integrated into another system, or certain features may be omitted, or certain steps may not be performed. In addition, the connections between each other shown or discussed, whether direct or indirect, whether coupling or communication connection, may be implemented in electrical, mechanical or other forms through interfaces, devices or units.

[0057] Units described as independent components may or may not actually be physically separate; parts presented as units may or may not be physical entities; that is, they may be centralized in one location or distributed across multiple network nodes. Depending on actual needs, some or all of these units may be selected to achieve the objectives of this embodiment.

[0058] Furthermore, in the various embodiments of the present application, the various functional units may be integrated into a single processing unit, physically exist independently, or two or more units may be combined into a single unit. In the aforementioned embodiments, the relevant functions may be implemented in whole or in part through software, hardware, firmware, or any combination thereof. If software implementation is chosen, it may be implemented in whole or in part in the form of a computer program product. This computer program product comprises one or more computer instructions. When these instructions are loaded and executed on a computer, they will generate, in whole or in part, the processes or functions described in the embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. These computer instructions may be stored in a computer-readable storage medium or transmitted via such a medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic cable, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium may be any computer-accessible, usable medium, or a data storage facility such as a server or data center that integrates one or more usable media. These available media may include magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), semiconductor media (e.g., SSDs), etc. Those skilled in the art will appreciate that all or part of the process steps for implementing the above-described method embodiments can be accomplished through hardware associated with computer program instructions. These programs can be stored on computer-readable storage media. When executed, these programs will contain the processes for each of the above-described method embodiments. These storage media include, but are not limited to, various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A multi-source perception data adaptive fusion method for flexible surgical instruments, characterized by: include: a. In a working environment, collect multimodal perception data based on timestamp synchronization technology; b. registering the multimodal perception data to obtain registration data; c. generating an implicit scene representation of the registration data using neural radiance field reconstruction technology; d. Input the implicit scene representation into the pre-trained RWKV fusion network to obtain fusion features; e. Perform residual connection post-processing on the fusion features and integrate them with the multimodal perception data to obtain environmental state data.

2. The method according to claim 1, characterized in that The multimodal perception data is collected based on optical sensors, mechanical sensors and posture sensors; the timestamp synchronization uses hardware clock signals to align the collection cycles of each sensor.

3. The method according to claim 1, characterized in that The registration of the multimodal sensing data is spatiotemporal registration, including clock compensation and spatial coordinate transformation; the spatial coordinate transformation is based on a kinematic model of the instrument.

4. The method according to claim 1, wherein Step c includes: inputting the registered data into a neural radiation field model; The neural radiance field model outputs a continuous spatial field representation as an implicit scene representation.

5. The method according to claim 1, wherein The pre-training is completed using a surgical scene simulation dataset.

6. The method according to claim 1, characterized in that The residual connection post-processing superimposes the fusion feature with the low-frequency component of the multimodal perception data; and the integration includes generating three-dimensional voxelized environmental state data.

7. A multi-source perception data adaptive fusion device for flexible surgical instruments, characterized by: include: Perception unit: used to collect multimodal perception data based on timestamp synchronization technology in the working environment; A registration unit is configured to register the multimodal perception data to obtain registration data; and is further configured to generate an implicit scene representation for the registration data using a neural radiation field reconstruction technique; Fusion unit: used for inputting the implicit scene representation into the pre-trained RWKV fusion network to obtain fusion features; Output unit: used to perform residual connection post-processing on the fusion feature and integrate it with the multimodal perception data to obtain environmental state data.

8. An electronic device, characterized in that: include: A processor and a storage unit, the storage unit is used to store computer program code, the code includes computer instructions, when the processor executes these instructions, the electronic device performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is caused to perform the method according to any one of claims 1 to 6.

Images