An adaptive fusion method and related device for multi-source sensing data of flexible surgical instruments

Abstract

This application discloses an adaptive fusion method and related device for multi-source sensing data of flexible surgical instruments. The method includes: acquiring optical, mechanical, and pose multimodal sensing data in the working environment based on timestamp synchronization technology; performing clock compensation and spatial coordinate transformation based on the instrument's kinematic model on the multimodal data to complete spatiotemporal registration; inputting the registered data into a neural radiation field model to generate a continuous spatial implicit scene representation; inputting the implicit representation into a pre-trained RWKV fusion network to obtain fusion features; and superimposing the fusion features with the original low-frequency components through residual connections to integrate and generate three-dimensional voxelized environmental state data. This invention achieves high-fidelity fusion of multi-source information through a single path, significantly reducing data redundancy and improving the real-time performance and accuracy of intraoperative environmental perception.

Description

Technical Field

[0001] This invention relates to the field of medical devices and signal processing, specifically to an adaptive fusion method and related device for multi-source sensing data of flexible surgical instruments. Background Technology

[0002] Flexible surgical instruments are increasingly used in delicate procedures such as minimally invasive surgery, laparoscopy, and vascular intervention. However, due to the flexible structure of the instruments and the complex anatomical environment, intraoperative perception still relies on single-modal imaging or force feedback, resulting in fragmented information and high latency. This makes it difficult for surgeons to grasp the real spatial relationship and interactive mechanical state between the instrument tip and surrounding tissues in real time. Existing methods often process optical, force, and pose data separately and then simply stitch them together, lacking a unified spatiotemporal reference. This makes them prone to registration errors during rapid movement or tissue deformation. At the same time, traditional explicit reconstruction requires dense point clouds, resulting in large computational loads and high memory consumption, making it difficult to implement on surgical-grade low-latency, low-power terminals. In addition, the lack of large-scale simulation pre-training data for surgical scenarios leads to insufficient model generalization, further restricting accurate and safe intraoperative navigation and intelligent control. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention relates to a method and related apparatus for adaptive fusion of multi-source sensing data for flexible surgical instruments, including but not limited to a device for adaptive fusion of multi-source sensing data for flexible surgical instruments, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] Firstly, an adaptive fusion method for multi-source sensing data of flexible surgical instruments is provided, including:

[0005] a. In the work environment, collect multimodal sensing data based on timestamp synchronization technology;

[0006] b. Register the multimodal sensing data to obtain registered data;

[0007] c. Generate implicit scene representations from the registration data using neural radiation field reconstruction technology;

[0008] d. Input the implicit scene representation into a pre-trained RWKV fusion network to obtain fused features;

[0009] e. Perform residual connection post-processing on the fused features and integrate them with the multimodal sensing data to obtain environmental state data.

[0010] In any embodiment of this application, the acquisition of multimodal sensing data is based on optical sensors, mechanical sensors, and pose sensors; the timestamp synchronization uses a hardware clock signal to align the acquisition cycles of each sensor.

[0011] In any embodiment of this application, the registration of the multimodal sensing data is spatiotemporal registration, including clock compensation and spatial coordinate transformation; the spatial coordinate transformation is based on the kinematic model of the device.

[0012] In any embodiment of this application, step c includes:

[0013] Input the registration data into the neural radiation field model;

[0014] The neural radiation field model outputs a continuous spatial field representation, which serves as an implicit scene representation.

[0015] In any embodiment of this application, the pre-training is performed using a surgical scenario simulation dataset.

[0016] In any embodiment of this application, the residual connection post-processing superimposes the fused features with the low-frequency components of the multimodal sensing data; the integration includes generating three-dimensional voxelized environmental state data.

[0017] Secondly, a multi-source sensing data adaptive fusion device for flexible surgical instruments is provided, the device comprising:

[0018] Sensing unit: Used to collect multimodal sensing data in the working environment based on timestamp synchronization technology;

[0019] The registration unit is used to register the multimodal sensing data to obtain registration data; it is also used to generate implicit scene representations from the registration data using neural radiation field reconstruction technology.

[0020] Fusion unit: used to input the implicit scene representation into a pre-trained RWKV fusion network to obtain fused features;

[0021] Output unit: used to perform residual connection post-processing on the fused features and integrate them with the multimodal sensing data to obtain environmental state data.

[0022] Thirdly, an electronic device is provided, comprising: a processor, a communication module, a sensor, a user interface, and a storage unit, wherein the storage unit is used to store computer program code, the program code including computer instructions. When the processor executes these instructions, the electronic device performs the methods described in the second aspect above and any of its embodiments.

[0023] Fourthly, another electronic device is provided, comprising: a processor, a wireless communication module, a touchscreen, a speaker, and a storage unit, wherein the storage unit is used to store computer program code, the program code including computer instructions. When the processor executes these instructions, the electronic device performs the methods described in the second aspect above and any of its embodiments.

[0024] Fifthly, a computer-readable storage medium is provided, wherein a computer program is stored, the program comprising program instructions. When these instructions are executed by a processor, the processor performs the methods described in the second aspect above and any of its embodiments.

[0025] In a sixth aspect, a computer program product is provided, the computer program product comprising a computer program or instructions. When the computer program or instructions are executed on a computer, the computer will perform the methods described in the second aspect above and any of its embodiments.

[0026] It should be understood that the above general descriptions and subsequent specific descriptions are for illustrative and explanatory purposes only and do not impose any limitations on this application.

[0027] In this application, compared with the prior art, the present invention provides an adaptive fusion method and related device for multi-source sensing data of flexible surgical instruments. It acquires multi-source heterogeneous data (optical, mechanical, and pose) simultaneously using timestamp synchronization technology, and completes spatiotemporal registration using spatial coordinate transformation and clock compensation based on the instrument's kinematic model, solving the geometric drift problem caused by inconsistent sensor time bases in traditional methods. The registered data is then fed into a neural radiation field model to generate a continuous implicit spatial representation, replacing traditional explicit point clouds or voxels, significantly compressing the data volume. This implicit representation is then inferred through an RWKV fusion network pre-trained on a surgical scenario simulation dataset to obtain high-dimensional fusion features, which are then superimposed with the original low-frequency components through residual connections, ultimately outputting three-dimensional voxelized environmental state data. Compared with the shortcomings of existing technologies such as slow surface rendering, high volume rendering computational cost, and large stitching errors, this method achieves high-fidelity fusion of multimodal information through a single path, improving rendering speed, reducing hardware resource consumption, and can run in real-time on a low-power intraoperative terminal, providing a reliable basis for the safe navigation and precise control of flexible instruments in complex cavities. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to illustrate the technical solutions of this application.

[0030] Figure 1 This is a schematic diagram of a multi-source sensing data adaptive fusion method for flexible surgical instruments proposed in an embodiment of this application.

[0031] Figure 2This is a schematic diagram of a multi-source sensing data adaptive fusion device for flexible surgical instruments proposed in an embodiment of this application.

[0032] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0033] To enable those skilled in the art to more fully understand the technical solutions of this application, the technical solutions of this application will be explained in detail and clearly with reference to the accompanying drawings. It should be particularly noted that the described embodiments are only some examples of this application and do not represent all of them. Based on these embodiments, those skilled in the art can directly deduce all other possible implementation schemes without creative thinking, and these are also included within the protection scope of this application.

[0034] In the specification, claims, and related drawings of this application, the terms "first," "second," etc., are used only to distinguish different elements and do not imply any specific order. Furthermore, the use of "comprising" and "having," and their variations, indicates non-exclusive inclusion. This means that if a process, method, system, product, or device comprises a series of steps or components, it indicates that the process, method, system, product, or device is not limited to the listed steps or components and may also include other steps or components not listed, or other inherent steps or units thereof.

[0035] The term "embodiment" as used herein refers to any instance combining a particular feature, structure, or characteristic, which may be at least one embodiment of this application. The "embodiments" mentioned herein do not necessarily refer to the same specific case, nor do they imply that they are independent or exclusive alternatives. Those skilled in the art will understand that the embodiments described herein can be used with other embodiments. It should be clarified that in this application, "at least one" includes one or more instances, "multiple" means two or more instances, and "at least two" refers to two or more instances.

[0036] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The embodiments of this application will now be described with reference to the accompanying drawings.

[0037] Please see Figure 1 , Figure 1 This is a schematic diagram of a multi-source sensing data adaptive fusion method for flexible surgical instruments provided in an embodiment of this application.

[0038] 101. Intraoperative data synchronous acquisition: In the working environment, multimodal sensing data is acquired based on timestamp synchronization technology.

[0039] In this embodiment, the sampling clocks of the optical camera, MEMS force sensor, and IMU pose sensor are locked using the IEEE 1588 protocol to ensure that cross-modal timestamp errors are controlled within the same period; the clock signal is divided by the FPGA to trigger synchronous exposure of each sensor, thereby outputting a data frame with a global time stamp.

[0040] In another possible implementation, the multimodal sensing data is acquired synchronously via a hardware trigger signal.

[0041] In another possible implementation, a GNSS timing module is used to send PPS pulses to each sensor, which is combined with a local clock counter to achieve soft synchronization; or a PTP hardware clock is embedded at the sensor end, and a transparent clock mechanism is used to compensate for link delay, thereby obtaining a synchronization timestamp.

[0042] 102. Multi-source sensing registration: Register the multi-modal sensing data to obtain registered data.

[0043] In this embodiment, a forward kinematics model of the device is used to map the local coordinates of the force sensor to the world coordinates of the optical camera, and then the spatial deviation is compensated by the iterative nearest point algorithm. Subsequently, the clock drift is corrected frame by frame by linear interpolation to complete the spatiotemporal consistency alignment.

[0044] In another possible implementation, the extrinsic parameters of the optical camera and the instrument end are first solved through hand-eye calibration, and then the extended Kalman filter is used to fuse IMU data and force sensor readings to achieve online estimation and compensation of kinematic model errors, thus completing the registration.

[0045] 103. Scene representation generation: Implicit scene representations are generated from the registration data using neural radiation field reconstruction technology.

[0046] In this embodiment, registration data is input into the neural radiation field, and position encoding and learnable implicit functions are used to integrate density and color in continuous space to output a gridless implicit scene representation to support queries from any viewpoint.

[0047] In another possible implementation, other techniques are used to implicitly represent the registration data.

[0048] In another possible implementation, the registered point cloud is voxelized using a truncated symbolic distance function to construct a discrete TSDF volume; or the point cloud is directly converted into a triangular mesh using differentiable mesh reconstruction as an explicit scene representation.

[0049] 104. Multi-source feature fusion: Input the implicit scene representation into the pre-trained RWKV fusion network to obtain fused features.

[0050] In this embodiment, the implicit scene representation is flattened into a sequence and then fed into the RWKV network. The RWKV network integrates cross-modal context based on temporal mixing and channel mixing mechanisms, and outputs a unified dimension fusion feature. The fusion feature is then residually superimposed with the original low-frequency component to preserve details.

[0051] In another possible implementation, the implicit scene representation is projected into a two-dimensional feature map according to the viewpoint. After extracting local features through a convolutional neural network, a Transformer encoder is used to perform intermodal attention interaction and output fused features. Alternatively, bidirectional LSTM can be used to process serialized features to achieve information integration.

[0052] 105. Environmental Status Output: Perform residual connection post-processing on the fused features and integrate them with the multimodal sensing data to obtain environmental status data.

[0053] In this embodiment, the fused features are resampled into regular voxels through trilinear interpolation to generate a three-dimensional voxel volume containing density, color, and force information; the three-dimensional voxel volume is directly used as environmental state data for subsequent navigation or control modules to call.

[0054] In another possible implementation, the fused features are mapped to sparse octree nodes along a spatial hash function, retaining non-empty leaf node data to form a sparse volume structure; or the environmental state data is encoded as a symbolic-distance field tensor to support subsequent field-based queries and operations.

[0055] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0056] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0057] Please see Figure 2 , Figure 2 This is a schematic diagram of a multi-source sensing data adaptive fusion device for flexible surgical instruments according to an embodiment of this application. The multi-source sensing fusion device 1 includes: a sensing unit 11, a registration unit 12, a fusion unit 13, and an output unit 14, specifically:

[0058] Sensing unit 11: Used to collect multimodal sensing data in the working environment based on timestamp synchronization technology;

[0059] Registration unit 12: used to register the multimodal sensing data to obtain registration data; also used to generate implicit scene representations from the registration data using neural radiation field reconstruction technology;

[0060] Fusion unit 13: used to input the implicit scene representation into the pre-trained RWKV fusion network to obtain fused features;

[0061] Output unit 14: used to perform residual connection post-processing on the fused features and integrate them with the multimodal sensing data to obtain environmental state data.

[0062] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0063] Please see Figure 3 , Figure 3 A schematic diagram of the hardware architecture of an electronic device according to an embodiment of this application is shown. The electronic device 2 mainly consists of a processor 21 and a memory 22. In addition, the device may also include an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are interconnected via connecting components, which can be various interfaces, data lines, or communication buses, etc., and are not specifically specified in this embodiment.

[0064] Processor 21 may be one or more graphics processing units (GPUs). If processor 21 is a GPU, the GPU may be single-core or multi-core. Alternatively, processor 21 may also be a processor group consisting of multiple GPUs interconnected via one or more buses. Furthermore, the processor may be other types of processors, which are not specifically limited in this embodiment.

[0065] The memory 22 is designed to store the instructions of a computer program and various program codes required to execute the present application. Optionally, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or optical disc read-only memory (CD-ROM), which are used to store related instructions and data.

[0066] Input device 23 is used to input data and / or signals, and output device 24 is used to output data and / or signals. Input device 23 and output device 24 can be independent devices or an integrated device.

[0067] It should be understood that, in the embodiments of this application, the memory 22 can store not only related instructions but also related data. The embodiments of this application do not specify the specific data content stored in the memory.

[0068] It should be understood that Figure 3This illustration only shows a simplified design of an electronic device. In actual use, the electronic device may also include other necessary components, such as different numbers of input / output devices, processors, memory, etc. All electronic devices capable of implementing the embodiments of this application are within the protection scope of this application.

[0069] Those skilled in the art will recognize that the components and algorithm steps of the various examples described in the embodiments disclosed herein can be implemented by electronic hardware or by a combination of computer software and electronic hardware. Whether these functions are implemented through hardware or software will be determined based on the specific application requirements and design constraints of the technical solution. Those skilled in the art can adopt different implementation methods according to the needs of each specific application, but such implementation should not be considered as exceeding the scope of protection of this application.

[0070] Those skilled in the art should understand that, for ease of description and simplification, the specific operational procedures of the aforementioned systems, devices, and components can be referred to the corresponding steps in the preceding method embodiments, and will not be repeated here. Furthermore, those skilled in the art should also understand that each embodiment in this application has its own focus, and for ease of description and simplification, the same or similar content may not be repeated in different embodiments. Therefore, if a part is not mentioned or not described in detail in a certain embodiment, it can be referred to the relevant description in other embodiments.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can also be implemented through other means. For example, the described apparatus embodiments are merely exemplary, and the division of the units therein is only a logical functional division; different division methods may exist in actual implementation. For example, multiple units or components may be merged or integrated into another system, or certain features may be omitted, or certain steps may not be performed. Furthermore, the interconnections shown or discussed, whether direct or indirect, whether coupling or communication connections, may be implemented electrically, mechanically, or otherwise through interfaces, devices, or units.

[0072] Units described as independent components may or may not actually be physically separate; parts presented as units may or may not be physical entities, meaning they may be concentrated in one location or distributed across multiple network nodes. Depending on the actual needs, some or all of these units can be selected to achieve the objectives of this embodiment.

[0073] Furthermore, in the various embodiments of this application, each functional unit can be integrated into a single processing unit, exist independently, or two or more units can be merged into one unit. In the foregoing embodiments, the relevant functions can be fully or partially implemented through software, hardware, firmware, or any combination thereof. If software implementation is chosen, it can be implemented entirely or partially in the form of a computer program product. This computer program product contains one or more computer instructions. When these instructions are loaded and executed on a computer, they will produce all or part of the processes or functions described in the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. These computer instructions can be stored in computer-readable storage media or transmitted through such media. Computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, DSL) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any computer-accessible available medium, or a data storage facility such as a server or data center that integrates one or more available media. These available media may include magnetic media (such as floppy disks, hard disks, and magnetic tapes), optical media (such as DVDs), semiconductor media (such as SSDs), etc. Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by computer program instructions and related hardware, and these programs can be stored in computer-readable storage media. When these programs are executed, they will contain the processes of the above method embodiments. The aforementioned storage media include, but are not limited to, various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An adaptive fusion method for multi-source sensing data of flexible surgical instruments, characterized in that, include: a. In the work environment, collect multimodal sensing data based on timestamp synchronization technology; The acquisition of multimodal sensing data is achieved based on optical sensors, mechanical sensors, and pose sensors; b. Register the multimodal sensing data to obtain registered data; c. Generate implicit scene representations from the registration data using neural radiation field reconstruction technology; d. Input the implicit scene representation into a pre-trained RWKV fusion network to obtain fused features; e. Perform residual connection post-processing on the fused features and integrate them with the multimodal sensing data to obtain environmental state data; the residual connection post-processing superimposes the fused features with the low-frequency components of the multimodal sensing data; the integration includes generating three-dimensional voxelized environmental state data.

2. The method according to claim 1, characterized in that, The timestamp synchronization uses a hardware clock signal to align the acquisition cycle of each sensor.

3. The method according to claim 1, characterized in that, The registration of the multimodal sensing data is a spatiotemporal registration, which includes clock compensation and spatial coordinate transformation; the spatial coordinate transformation is based on the kinematic model of the device.

4. The method according to claim 1, characterized in that, Step c includes: Input the registration data into the neural radiation field model; The neural radiation field model outputs a continuous spatial field representation, which serves as an implicit scene representation.

5. The method according to claim 1, characterized in that, The pre-training was performed using a surgical scenario simulation dataset.

6. A multi-source sensing data adaptive fusion device for flexible surgical instruments, characterized in that, include: Sensing unit: Used to collect multimodal sensing data in the working environment based on timestamp synchronization technology; The acquisition of multimodal sensing data is achieved based on optical sensors, mechanical sensors, and pose sensors; Registration unit: used to register the multimodal sensing data to obtain registration data; also used to generate implicit scene representations from the registration data using neural radiation field reconstruction technology; Fusion unit: used to input the implicit scene representation into a pre-trained RWKV fusion network to obtain fused features; Output unit: used to perform residual connection post-processing on the fused features and integrate them with the multimodal sensing data to obtain environmental state data; the residual connection post-processing superimposes the fused features with the low-frequency components of the multimodal sensing data; the integration includes generating three-dimensional voxelized environmental state data.

7. An electronic device, characterized in that, include: A processor and a storage unit for storing computer program code, the code containing computer instructions, wherein when the processor executes these instructions, the electronic device performs the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program containing program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1 to 5.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when run on a computer, causes the computer to perform the method described in any one of claims 1 to 5.

Images