Surgical robot dynamic compensation method based on multi-mode real-time 4D digital twinning and related device

Abstract

The invention discloses a surgical robot dynamic compensation method based on multi-mode real-time 4D digital twinning and a related device. The method comprises the following steps: acquiring preoperative three-dimensional and four-dimensional image data of a patient, synchronously acquiring optical body surface movement data and in-vivo ultrasonic data during an operation, inputting the data into an end-to-end modular network in combination with current state data of a surgical robot for feature extraction and fusion, generating a 4D twinborn space containing spatio-temporal information, and generating a compensation instruction based on the space to control the surgical robot. According to the method, real-time fusion and dynamic modeling of multi-modal data in an operation can be realized, the self-adaption and compensation capability of a surgical robot in a complex surgical environment is improved, and the surgical precision and safety are enhanced.

Description

TECHNICAL FIELD

[0001] The application relates to the field of artificial intelligence, medicine and robots, and particularly relates to a surgical robot dynamic compensation method based on multi-modal real-time 4D digital twinning and a related device. BACKGROUND

[0002] In the field of minimally invasive surgery, surgical robots have been widely used due to their advantages of accurate operation and filtering of tremor. However, the current surgical robot system still faces severe challenges in precision and safety when facing the deformation and displacement of tissues and organs caused by breathing, heartbeat or patient movement. In the prior art, a common solution is to rely on preoperative static three-dimensional images for surgical planning and navigation. However, such static images cannot reflect the dynamic physiological changes during surgery, resulting in a mismatch between the "blueprint" and the real-time situation, causing positioning errors. Another solution is to try to introduce intraoperative images such as ultrasound or optical tracking to capture dynamic information. However, these multi-modal data are often independent of each other and lack effective fusion, making it difficult to construct a unified and real-time dynamic model to guide robot actions. In addition, the architecture of the existing system is usually fixed and cannot flexibly adapt to the differences and changes in data modalities in different surgical scenarios, limiting its universality and optimization potential. Therefore, there is an urgent need in the field for a technical solution that can integrate multi-source information, real-time perceive dynamic anatomical structures, and intelligently drive surgical robots for adaptive compensation. SUMMARY

[0003] To solve the above technical problems, the application relates to a surgical robot dynamic compensation method based on multi-modal real-time 4D digital twinning and related devices, including but not limited to a surgical robot dynamic compensation device based on multi-modal real-time 4D digital twinning, an electronic device, a computer readable storage medium and a computer program product.

[0004] In a first aspect, a surgical robot dynamic compensation method based on multi-modal real-time 4D digital twinning is provided, comprising: S1: acquiring preoperative three-dimensional image data and preoperative four-dimensional image data of a patient; S2: based on a timestamp, synchronously collecting optical body surface motion data and in-vivo ultrasound data of the patient during surgery; S3: inputting the preoperative three-dimensional image data, the preoperative four-dimensional image data, the optical body surface motion data, the in-vivo ultrasound data and current state data of the surgical robot as input data into an end-to-end modular network; S4: The end-to-end modular network extracts features and fuses data from the preoperative three-dimensional image data, the preoperative four-dimensional image data, the optical body surface motion data, the intracorporeal ultrasound data, and the current state data, and generates a 4D twin space; the 4D twin space contains spatial three-dimensional coordinates and a time dimension; S5: Based on the 4D twin space, compensation instructions are generated and sent to the surgical robot.

[0005] In combination with any embodiment of the present application, the end-to-end modular network adopts a modular structure; the modular structure includes a fusion module and respective processing branches corresponding to the input data, and the processing branches and the fusion module support hot plugging and replacement.

[0006] In combination with any embodiment of the present application, the feature extraction includes processing the preoperative three-dimensional image data, the optical body surface motion data, and the intracorporeal ultrasound data using corresponding processing branches; the processing branches are based on convolutional neural networks.

[0007] In combination with any embodiment of the present application, the 4D twin space includes geometric representation and physical attributes.

[0008] In combination with any embodiment of the present application, the modular structure supports dynamic configuration of the processing branches and the fusion module according to intraoperative data characteristics.

[0009] In combination with any embodiment of the present application, the modular structure uses a standardized interface protocol.

[0010] In a second aspect, a surgical robot dynamic compensation device based on multi-modal real-time 4D perception is provided, comprising: A preoperative preparation unit is configured to obtain preoperative three-dimensional image data and preoperative four-dimensional image data of a patient; An intraoperative perception unit is configured to synchronously collect optical body surface motion data and intracorporeal ultrasound data of a patient based on timestamps during surgery; A 4D modeling unit is configured to input the preoperative three-dimensional image data, the preoperative four-dimensional image data, the optical body surface motion data, the intracorporeal ultrasound data, and current state data of a surgical robot as input data to an end-to-end modular network; the end-to-end modular network extracts features and fuses data from the preoperative three-dimensional image data, the preoperative four-dimensional image data, the optical body surface motion data, the intracorporeal ultrasound data, and the current state data, and generates a 4D twin space; the 4D twin space contains spatial three-dimensional coordinates and a time dimension; A pose adjustment unit is configured to generate compensation instructions based on the 4D twin space and send the compensation instructions to the surgical robot.

[0011] In a third aspect, an electronic device is provided, comprising: a processor, a communication module, a sensor, a user interface, and a storage unit, wherein the storage unit is configured to store computer program code, the program code comprising computer instructions. When the processor executes the instructions, the electronic device will perform the method described in the second aspect above and any of its implementation forms.

[0012] In a fourth aspect, another electronic device is provided, comprising: a processor, a wireless communication module, a touch screen, a speaker, and a storage unit, wherein the storage unit is configured to store computer program code, the program code comprising computer instructions. When the processor executes the instructions, the electronic device will perform the method described in the second aspect above and any of its implementation forms.

[0013] In a fifth aspect, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, the program comprising program instructions. When the instructions are executed by a processor, the processor will perform the method described in the second aspect above and any of its implementation forms.

[0014] In a sixth aspect, a computer program product is provided, the computer program product comprising a computer program or instructions. When the computer program or instructions are run on a computer, the computer will perform the method described in the second aspect above and any of its implementation forms.

[0015] In the present application, compared with the prior art, a surgical robot dynamic compensation method based on multi-modal real-time 4D perception is provided. The method constructs an end-to-end neural network, performs multi-modal feature extraction and fusion on preoperative three-dimensional image data, preoperative four-dimensional image data, intraoperative synchronously collected optical body surface motion data and in-vivo ultrasound data, and current state data of a surgical robot, generates a 4D twin space containing spatial three-dimensional coordinates and a time dimension, and generates compensation instructions for the surgical robot based on the 4D twin space, thereby realizing continuous spatiotemporal representation of intraoperative dynamic anatomical structures and real-time adaptive adjustment of the robot pose. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments or background of the present application, the drawings needed to be used in the embodiments or background of the present application will be described below.

[0017] The drawings herein are incorporated into the specification and form part of the specification, which illustrate embodiments consistent with the present application, and together with the specification serve to explain the technical solutions of the present application.

[0018] Figure 1 A surgical robot dynamic compensation method flowchart based on multi-modal real-time 4D digital twin is provided for the embodiments of the present application.

[0019] Figure 2 A surgical robot dynamic compensation device based on multi-modal real-time 4D digital twinning provided by an embodiment of the present application is shown in the schematic diagram.

[0020] Figure 3 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0021] In order to enable the person skilled in the art to have a more comprehensive understanding of the technical solutions of the present application, the technical solutions of the present application will be described in detail and clearly in the following with reference to the accompanying drawings. It should be particularly pointed out that the described embodiments are only part of the examples of the present application, and do not represent the whole. Based on these embodiments, the person skilled in the art can directly deduce all other possible embodiments without creative thinking, and these are also included in the protection scope of the present application.

[0022] In the specification, claims and related drawings of the present application, the terms "first", "second", etc. are only used to distinguish different elements, and do not imply any specific order. At the same time, the use of "include" and "have" and their variants means non-exclusive inclusion. This means that if a process, method, system, product or device includes a series of steps or components, it means that the process, method, system, product or device is not limited to the listed steps or components, and can also include other steps or components not listed, or other steps or units inherent to it.

[0023] In this paper, "embodiment" refers to any example that combines specific features, structures or properties, which may belong to at least one embodiment of the present application. The "embodiment" mentioned in this paper does not necessarily refer to the same specific case, nor does it mean that they are mutually independent or alternative solutions. The person skilled in the art should understand that the embodiments described in this paper can be used with other embodiments. It should be clear that in this application, "at least one" includes one or more instances, "multiple" means two or more instances, and "at least two" means two or more instances.

[0024] It should be understood that the method embodiments of the present application can also be realized by the processor executing computer program code. The embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0025] Please refer to Figure 1 , Figure 1 A surgical robot dynamic compensation method based on multi-modal real-time 4D digital twinning provided by an embodiment of the present application is shown in the flowchart.

[0026] 101、Preoperative preparation: obtain preoperative three-dimensional image data and preoperative four-dimensional image data of the patient.

[0027] In this embodiment, the preoperative four-dimensional image data can be at least dynamic CT data or dynamic MRI data.

[0028] In this embodiment, the preoperative three-dimensional image data and the preoperative four-dimensional image data contain image information of the same target area.

[0029] In this embodiment, the modalities of the preoperative three-dimensional image data and the preoperative four-dimensional image data can be different.

[0030] In another possible implementation, the preoperative three-dimensional image data and the preoperative four-dimensional image data are obtained through a picture archiving and communication system.

[0031] 102、Intraoperative perception: based on timestamps, synchronously collect optical surface motion data and in-vivo ultrasound data of the patient during surgery.

[0032] In this embodiment, the synchronous collection is realized based on a hardware clock signal.

[0033] In this embodiment, the optical surface motion data is collected through optical beads on the surface of the patient.

[0034] In this embodiment, the in-vivo ultrasound data is a two-dimensional ultrasound image stream.

[0035] In another possible implementation, the synchronous collection is realized through a time synchronization protocol at the software level.

[0036] In another possible implementation, the optical surface motion data is collected through a structured light or stereo vision camera.

[0037] 103、4D twin: input the preoperative three-dimensional image data, the preoperative four-dimensional image data, the optical surface motion data, the in-vivo ultrasound data, and current state data of the surgical robot as input data to an end-to-end modular network; the end-to-end modular network performs feature extraction and data fusion on the preoperative three-dimensional image data, the preoperative four-dimensional image data, the optical surface motion data, the in-vivo ultrasound data, and the current state data to generate a 4D twin space; the 4D twin space contains spatial three-dimensional coordinates and a time dimension.

[0038] In this embodiment, the end-to-end modular network adopts a modular structure; the modular structure contains a fusion module and respective processing branches corresponding to the input data, and the processing branches and the fusion module support hot plugging and replacement.

[0039] In the embodiment, the modular structure supports dynamic configuration of the processing branches and the fusion module according to intraoperative data characteristics.

[0040] In the embodiment, the modular structure uses a standardized interface protocol.

[0041] In the embodiment, the feature extraction includes processing the preoperative three-dimensional image data, the optical body surface motion data, and the intracorporeal ultrasound data using corresponding processing branches; the processing branches are based on convolutional neural networks.

[0042] In the embodiment, the 4D twin space includes geometric representation and physical attributes.

[0043] In the embodiment, the 4D twin space continuously spatiotemporally characterizes morphological changes and positional changes of anatomical structures in the surgical robot workspace.

[0044] In the embodiment, the data fusion is achieved through an attention architecture.

[0045] In another possible implementation, the data fusion is achieved through a graph neural network or an attention architecture.

[0046] In another possible implementation, the 4D twin space supports real-time updating and feedback of biomechanical attributes of anatomical structures in the surgical robot workspace.

[0047] 104. Dynamic compensation: based on the 4D twin space, generate compensation instructions and send the compensation instructions to the surgical robot.

[0048] In the embodiment, the generation of the compensation instructions is based on the spatiotemporal evolution state of the 4D twin space.

[0049] In the embodiment, the compensation instructions include a pose adjustment amount of an end effector of the surgical robot.

[0050] In the embodiment, the pose adjustment amount is calculated by a fully connected regressor.

[0051] In the embodiment, executing the compensation instructions includes sending the compensation instructions to a controller of the surgical robot, which can be achieved through wireless network signals or wired network signals.

[0052] In another possible implementation, the generation of the compensation instructions further introduces surgical task constraints or safety boundary constraints.

[0053] In another possible implementation, the sending of the compensation instructions is achieved through a real-time communication bus or an industrial Ethernet protocol.

[0054] In some embodiments, the device provided by the embodiments of the present application has functions or includes modules that can be used to perform the methods described in the above method embodiment descriptions, and specific implementations can refer to the descriptions of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0055] The above describes the method of the embodiments of the present application in detail. The device of the embodiments of the present application is provided below.

[0056] Please refer to Figure 2 , Figure 2 A schematic diagram of a surgical robot dynamic compensation device based on multi-modal real-time 4D digital twinning provided by the embodiments of the present application is shown. The twinning and compensation device 1 includes a preoperative preparation unit 11, an intraoperative perception unit 12, a 4D modeling unit 13, and a pose adjustment unit 14, specifically: The preoperative preparation unit 11 is configured to obtain preoperative three-dimensional image data and preoperative four-dimensional image data of a patient. The intraoperative perception unit 12 is configured to synchronously collect optical body surface motion data and in-vivo ultrasound data of the patient based on a timestamp during an operation. The 4D modeling unit 13 is configured to input the preoperative three-dimensional image data, the preoperative four-dimensional image data, the optical body surface motion data, the in-vivo ultrasound data, and current state data of a surgical robot as input data to an end-to-end modular network. The end-to-end modular network performs feature extraction and data fusion on the preoperative three-dimensional image data, the preoperative four-dimensional image data, the optical body surface motion data, the in-vivo ultrasound data, and the current state data to generate a 4D twinning space. The 4D twinning space includes spatial three-dimensional coordinates and a time dimension. The pose adjustment unit 14 is configured to generate compensation instructions based on the 4D twinning space and send the compensation instructions to the surgical robot.

[0057] In some embodiments, the device provided by the embodiments of the present application has functions or includes modules that can be used to perform the methods described in the above method embodiment descriptions, and specific implementations can refer to the descriptions of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0058] Please refer to Figure 3 , Figure 3 A hardware architecture schematic diagram of an electronic device described in the embodiments of the present application is shown. The electronic device 2 mainly consists of a processor 21 and a memory 22. In addition, the device can also include an input device 23 and an output device 24. The processor 21, the memory 22, the input device 23, and the output device 24 are connected to each other through connection components. These connection components can be various interfaces, data lines, or communication buses, etc., and the embodiments of the present application do not make specific provisions for this.

[0059] The processor 21 can be one or more graphic processors (GPUs). If the processor 21 is a GPU, the GPU can be single-core or multi-core. Alternatively, the processor 21 can also be a processor group composed of multiple GPUs, which are connected to each other through one or more buses. In addition, the processor can also be other types of processors, and the embodiments of the present application do not make specific limitations on this.

[0060] The memory 22 is designed to save the instructions of the computer program and various program codes required for the implementation of the scheme of the present application. Alternatively, the memory can include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or compact disc read-only memory (CD-ROM), which are used to store relevant instructions and data.

[0061] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices, or can be an integral device.

[0062] It should be recognized that, in the embodiments of the present application, the memory 22 can not only save relevant instructions, but also save relevant data. The embodiments of the present application do not make specific provisions for the specific data content stored in the memory.

[0063] It should be understood that, Figure 3 Only a simplified design of an electronic device is shown. In actual use, the electronic device can also include other necessary components, such as different numbers of input / output devices, processors, memories, etc. All electronic devices capable of implementing the embodiments of the present application fall within the protection scope of the present application.

[0064] Those skilled in the art should recognize that, according to the components and algorithm steps of various examples described in the embodiments disclosed herein, the functions can be realized by electronic hardware or in combination with computer software and electronic hardware. Specifically, whether the functions are executed by hardware or software depends on the specific application requirements and design constraints of the technical solutions. The skilled person can adopt different implementation methods according to the requirements of each specific application, but such implementation methods should not be regarded as beyond the protection scope of the present application.

[0065] It should be appreciated by those skilled in the art that, for convenience and simplicity, the specific operations of the system, device and component described above can refer to the corresponding steps in the foregoing method embodiments, which will not be repeated here. Meanwhile, it should be appreciated by those skilled in the art that each of the embodiments in the present application has its own emphasis, and for the convenience and simplicity, the same or similar content may not be described repeatedly in different embodiments, so if the part is not mentioned or not described in detail in an embodiment, it can be referred to the relevant description of other embodiments.

[0066] In several embodiments provided in the present application, it should be appreciated that the disclosed system, device and method can also be implemented by other means. For example, the described device embodiments are only exemplary, and the division of the units is only logical functional division, and different division methods may exist in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or some steps may not be performed. In addition, the connection between the units shown or discussed, whether direct or indirect, whether coupled or communicatively connected, can be realized by interfaces, devices or units in electrical, mechanical or other forms.

[0067] The units described as independent components may actually be physically separated or not; the parts presented as units may or may not be physical entities, that is, they can be concentrated in one location or dispersed on multiple network nodes. According to actual needs, part or all of these units can be selected to achieve the goal of the present embodiment.

[0068] Moreover, in each embodiment of the present application, each functional unit can be integrated into one processing unit, or each independent physical unit, or two or more units can be combined into one unit. In the foregoing embodiments, the related functions can be fully or partially implemented by software, hardware, firmware or any combination thereof. If software implementation is chosen, it can be implemented in whole or in part in the form of a computer program product. The computer program product contains one or more computer instructions. When the instructions are loaded and executed on a computer, they produce all or part of the processes or functions described in the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, DSL) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any computer-accessible available medium, or a data storage facility such as a server, data center, etc. integrated with one or more available media. These available media can include magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), semiconductor media (such as SSDs), etc. Those skilled in the art should understand that all or part of the processes of the above-mentioned embodiments can be completed by computer program instruction related hardware, and these programs can be stored in a computer-readable storage medium. When these programs are executed, they will contain the processes of the above-mentioned embodiments. The above-mentioned storage medium includes but is not limited to read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

Claims

1. A surgical robot dynamic compensation method based on multi-modal real-time 4D digital twinning, characterized in that, The method comprises the following steps: S1: obtaining preoperative three-dimensional image data and preoperative four-dimensional image data of a patient; S2: synchronously collecting optical body surface motion data and in-vivo ultrasound data of the patient in an operation based on a time stamp; S3: inputting the preoperative three-dimensional image data, the preoperative four-dimensional image data, the optical body surface motion data, the in-vivo ultrasound data and current state data of a surgical robot as input data into an end-to-end modular network; S4: the end-to-end modular network extracts features and fuses data of the preoperative three-dimensional image data, the preoperative four-dimensional image data, the optical body surface motion data, the in-vivo ultrasound data and the current state data to generate a 4D twin space; the 4D twin space comprises spatial three-dimensional coordinates and a time dimension; S5: generating compensation instructions based on the 4D twin space and sending the compensation instructions to the surgical robot.

2. The method of claim 1, wherein, The end-to-end modular network adopts a modular structure; the modular structure comprises a fusion module and respective processing branches corresponding to the input data, and the processing branches and the fusion module support hot plugging and replacement.

3. The method of claim 1, wherein, The feature extraction comprises processing the preoperative three-dimensional image data, the optical body surface motion data and the in-vivo ultrasound data using respective processing branches; the processing branches are based on convolutional neural networks.

4. The method of claim 1, wherein, The 4D twin space comprises geometric representation and physical attributes.

5. The method of claim 2, wherein, The modular structure supports dynamically configuring the processing branches and the fusion module according to in-operation data characteristics.

6. The method of claim 2, wherein, The modular structure uses a standardized interface protocol.

7. A surgical robot dynamic compensation device based on multi-modal real-time 4D perception, characterized in that, The method comprises the following steps: A preoperative preparation unit is configured to obtain preoperative three-dimensional image data and preoperative four-dimensional image data of a patient; An in-operation sensing unit is configured to synchronously collect optical body surface motion data and in-vivo ultrasound data of the patient in an operation based on a time stamp; A 4D modeling unit is configured to input the preoperative three-dimensional image data, the preoperative four-dimensional image data, the optical body surface motion data, the in-vivo ultrasound data and current state data of a surgical robot as input data into an end-to-end modular network; the end-to-end modular network extracts features and fuses data of the preoperative three-dimensional image data, the preoperative four-dimensional image data, the optical body surface motion data, the in-vivo ultrasound data and the current state data to generate a 4D twin space; the 4D twin space comprises spatial three-dimensional coordinates and a time dimension; A pose adjustment unit is configured to generate compensation instructions based on the 4D twin space and send the compensation instructions to the surgical robot.

8. An electronic device, comprising: The method comprises the following steps: A processor and a storage unit are configured to store computer program codes; the codes comprise computer instructions; when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer readable storage medium stores a computer program; the computer program comprises program instructions; when the program instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 6.

10. A computer program product, characterised in that, The computer program product comprises computer programs or instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 6.

Images