Puncture assistance system and device, and correction system and surgical robot

Abstract

Provided in the embodiments of the present description are a puncture assistance system and device, and a correction system and a surgical robot. The puncture assistance system comprises a processor, which is configured to: obtain a scanning image of a target object, wherein the scanning image comprises at least one of a preoperative image, a first image in a first time period and a second image in a second time period, and the first time period is earlier than the second time period; generate puncture guidance information on the basis of puncture state information and the scanning image; guide a puncture operation on the basis of the puncture guidance information; and display the puncture guidance information.

Description

Puncture assist system, device, correction system and surgical robot

[0001] Cross-references

[0002] This application claims priority to Chinese application No. 202310964477.0, filed on July 31, 2023, the entire contents of which are incorporated herein by reference. This application claims priority to Chinese application No. 202310707754.X, filed on June 14, 2023, the entire contents of which are incorporated herein by reference. This application claims priority to Chinese application No. 202311421935.2, filed on October 27, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This specification relates to the field of medical technology, and in particular to a puncture assistance system, equipment, correction system and surgical robot. Background Art

[0004] Percutaneous access to lesions or soft tissues using surgical needles is an important component of modern clinical medicine and has attracted considerable attention in recent years. Numerous clinical applications, such as biopsy, injections, neurosurgery, and cancer treatment, utilize similar minimally invasive procedures.

[0005] Current puncture surgical options primarily include manual blind puncture, real-time image-guided puncture, and master-slave real-time guided puncture. Master-slave real-time guided puncture involves the surgeon remotely controlling a surgical robot from outside the operating room (i.e., a CT-shielded room) to perform the puncture and control the computed tomography (CT) scanner to scan and display the patient's images in real time. This approach protects the surgeon from radiation exposure during surgery.

[0006] However, since the current master-slave real-time guided puncture scheme relies too much on the doctor's subjective medical experience, it is easy to cause the puncture to be in place, which in turn leads to prolonged operation time; and multiple posture adjustments, scanning and punctures will also increase the patient's pain and radiation dose.

[0007] Therefore, in order to reduce the radiation dose during surgery, shorten the imaging time and improve the accuracy of puncture, it is necessary to provide a puncture assistance system, equipment, correction system and surgical robot.

[0008] Summary of the Invention

[0009] One or more embodiments of the present specification provide a puncture assistance system, including a processor, wherein the processor is configured to: obtain a scanned image of a target object; the scanned image includes at least one of a preoperative image, a first image of a first time period, and a second image of a second time period; wherein the first time period is earlier than the second time period; generate puncture guidance information based on puncture status information and the scanned image; guide a puncture operation based on the puncture guidance information; and display the puncture guidance information.

[0010] One or more embodiments of the present specification provide a puncture needle position correction system, including a positioning correction module, wherein the positioning correction module is configured to: determine first position information of the puncture needle in an image coordinate system based on the positioning position information of the puncture needle in a robot coordinate system, wherein the first position information includes the needle tip position and the needle tail position of the puncture needle; identify the needle tip position in a puncture image based on the first position information to obtain second position information of the puncture needle in the puncture image; the puncture image includes a second image and / or a target guide image; verify the first position information based on the second position information to obtain a verification result, and determine target position information based on the verification result; the target position information includes position information of a first needle tip point.

[0011] One or more embodiments of the present specification provide a puncture assist device, comprising: a scanning module, a puncture assist module, and a display module; the scanning module is configured to obtain a scanned image of a target object; the scanned image includes at least one of a preoperative image, a first image in a first time period, and a second image in a second time period; wherein the first time period is earlier than the second time period; the puncture assist module is configured to: generate puncture guidance information based on puncture status information and the scanned image; guide a puncture operation based on the puncture guidance information; and the display module is configured to display the puncture guidance information.

[0012] One or more embodiments of the present specification provide a surgical robot comprising: a surgical robot master hand, a first processor and a surgical robot slave hand, the puncture needle being arranged on the surgical robot slave hand, and the surgical robot master hand being communicatively connected to a puncture auxiliary device; wherein: the surgical robot master hand is configured to, after receiving a first puncture operation instruction, send the first puncture operation instruction to the first processor, and send a motion instruction for performing the puncture operation to the puncture auxiliary device; the first processor is configured to, after receiving the first puncture operation instruction, send a fifth movement instruction corresponding to the first puncture operation instruction to the surgical robot slave hand; the puncture auxiliary device is configured to, after receiving the motion instruction, send a scanning instruction to a medical imaging device, so that the medical imaging device performs a medical image scan on a target object; the surgical robot slave hand is configured to move the puncture needle according to the fifth movement instruction, and perform a puncture operation corresponding to the first puncture operation instruction, wherein the puncture operation includes a posture adjustment operation or a needle insertion operation, the posture adjustment operation being used to adjust the posture of the puncture needle, and the needle insertion operation being used to adjust the puncture depth of the puncture needle. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] This specification will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, like numbers represent like structures, wherein:

[0014] FIG1 is a schematic structural diagram of a puncture assist system according to some embodiments of this specification;

[0015] FIG2 is a schematic structural diagram of a puncture assisting device according to some embodiments of this specification;

[0016] FIG3 is an exemplary interactive diagram of a puncture method according to some embodiments of the present specification;

[0017] FIG4 is an exemplary schematic diagram of a puncture process according to some embodiments of this specification;

[0018] FIG5 is an exemplary schematic diagram of tissue segmentation obtained during a preoperative image scanning stage according to some embodiments of the present specification;

[0019] FIG6 is an exemplary schematic diagram of an initial puncture path displayed on an image obtained by a scanning device according to some embodiments of the present specification;

[0020] FIG7 is an exemplary schematic diagram of an extracted needle track contour according to some embodiments of the present specification;

[0021] FIG8 is a schematic diagram of an interface in which a needle tip point and a predicted target point are located on the same target image according to some embodiments of this specification;

[0022] FIG9 is a schematic diagram of an interface in which a needle tip point and a predicted target point are located in different target images according to some embodiments of this specification;

[0023] FIG10 is a schematic diagram of an interface for adjusting posture on the same layer according to some embodiments of this specification;

[0024] FIG11 is a schematic diagram of an interface for cross-layer posture adjustment according to some embodiments of this specification;

[0025] FIG12 is an exemplary flowchart of determining a target image according to some embodiments of this specification;

[0026] FIG13 is an exemplary schematic diagram of target respiratory amplitude points according to some embodiments of the present specification;

[0027] FIG14 is an exemplary schematic diagram of target respiratory amplitude intervals according to some embodiments of the present specification;

[0028] FIG15 is an exemplary schematic diagram of real-time image update according to some embodiments of this specification;

[0029] FIG16 is an exemplary schematic diagram of ideal respiratory amplitude points of a target subject during a surgical procedure according to some embodiments of the present specification;

[0030] FIG17 is an exemplary flowchart of determining a first guide image according to some embodiments of this specification;

[0031] FIG18 is an exemplary flowchart of determining an updated target guidance image according to some embodiments of the present specification;

[0032] FIG19 is an exemplary schematic diagram of a puncture needle in a second image according to some embodiments of the present specification;

[0033] FIG20 is an exemplary schematic diagram of a first guide image according to some embodiments of the present specification;

[0034] FIG21 is an exemplary schematic diagram of a second guide image according to some embodiments of the present specification;

[0035] FIG. 22 is an exemplary flowchart of a method for correcting the position of a puncture needle according to some embodiments of the present specification.

[0036] FIG23 is an exemplary flow chart of a method for correcting the position of a puncture needle according to another embodiment of the present specification;

[0037] FIG24 is an exemplary flow chart of a method for correcting the position of a puncture needle according to another embodiment of the present specification;

[0038] FIG25 is an exemplary flow chart of a method for correcting the position of a puncture needle according to another embodiment of the present specification;

[0039] FIG26 is an exemplary flow chart of a method for correcting the position of a puncture needle according to another embodiment of the present specification;

[0040] FIG27 is an exemplary flow chart of a method for correcting the position of a puncture needle according to another embodiment of the present specification;

[0041] FIG28 is a schematic structural diagram of a puncture assisting device according to other embodiments of this specification;

[0042] FIG29 is a schematic structural diagram of a surgical robot according to some embodiments of the present specification;

[0043] FIG30 is a schematic structural diagram of a puncture needle position correction system according to some embodiments of this specification;

[0044] FIG31 is a schematic structural diagram of a device for correcting the position of a puncture needle according to some embodiments of this specification;

[0045] FIG32 is a schematic structural diagram of a puncture needle position correction device according to other embodiments of this specification;

[0046] FIG33 is a schematic structural diagram of a device for correcting the position of a puncture needle according to other embodiments of this specification;

[0047] FIG34 is a schematic structural diagram of a puncture needle position correction device according to other embodiments of this specification;

[0048] FIG35 is a diagram illustrating the internal structure of a computer device according to some embodiments of the present specification. DETAILED DESCRIPTION

[0049] To more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for describing the embodiments. Obviously, the drawings described below are merely examples or embodiments of this specification. Those skilled in the art can apply this specification to other similar scenarios based on these drawings without inventive effort. Unless otherwise apparent from the context or otherwise noted, the same reference numerals in the figures represent the same structure or operation.

[0050] It should be understood that the terms "system," "device," "unit," and / or "module" used herein are a method for distinguishing different components, elements, parts, portions, or assemblies at different levels. However, if other terms can achieve the same purpose, the terms may be replaced by other expressions.

[0051] Unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not refer to the singular but include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements.

[0052] Flowcharts are used throughout this specification to illustrate the operations performed by systems according to embodiments of this specification. It should be understood that preceding or following operations do not necessarily need to be performed in exact order. Instead, the steps may be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0053] In some embodiments of this specification, a puncture assistance system is provided, comprising a processor configured to: obtain a scanned image of a target object; the scanned image comprising at least one of a preoperative image, a first image in a first time period, and a second image in a second time period; wherein the first time period is earlier than the second time period; generate puncture guidance information based on puncture status information and the scanned image; guide a puncture operation based on the puncture guidance information; and display the puncture guidance information. For more information, please refer to the description related to FIG. 2. The processor involved in each embodiment provided in this specification may be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0054] FIG1 is a schematic structural diagram of a puncture assist system according to some embodiments of the present specification.

[0055] As shown in Figure 1, the puncture assist system in the embodiment of this specification includes: a puncture assist device, a surgical robot, and a medical imaging device. The puncture assist device is communicatively connected to the surgical robot and the medical imaging device respectively.

[0056] In some embodiments, the puncture assist device may be configured to obtain a scanned image of the target object; generate puncture guidance information based on the puncture status information and the scanned image; guide the puncture operation based on the puncture guidance information; and display the puncture guidance information. In some embodiments, the puncture assist device may include a scanning module 210, a puncture assist module 220, and a display module 230. For more information, see the relevant description of FIG. 2.

[0057] In some embodiments, the surgical robot is configured to perform a puncture operation. For example, the surgical robot is configured to send puncture status information of a puncture needle in the surgical robot to a puncture assist device.

[0058] The puncture assistance system in the embodiment of this specification generates an annotation object by marking the needle tip point and the predicted target point on the target image, and displays the annotation image, so that the doctor has a clear understanding of the position of the needle tip point of the puncture needle and the position that the needle tip point can reach (that is, the position of the predicted target point), assisting the doctor in judging the needle tip point of the puncture needle and the puncture point that the puncture needle can reach, reducing the dependence on the doctor's experience and improving the accuracy of puncture.

[0059] As shown in Figure 1, the surgical robot in the embodiments of this specification may include a master robot hand, a first processor, and a slave robot hand, which are communicatively connected in sequence. It is understood that in the surgical robot, the master robot hand is the operating end, and the slave robot hand is the execution end. The user operates the master robot hand to remotely control the slave robot hand to perform the puncture operation. In some embodiments, the puncture needle is mounted on the slave robot hand, and the master robot hand is communicatively connected to the puncture assist device.

[0060] As shown in Figure 1, in order to protect doctors from the effects of radiation during medical imaging scanning, the puncture assistance system is divided into two parts: outside the operating room and inside the operating room. The puncture assistance equipment operated by the doctor and the main hand of the surgical robot are set outside the operating room; inside the operating room are: medical imaging equipment for scanning the target object, and the slave hand of the surgical robot for puncturing the target object.

[0061] In some embodiments, the surgical robot's slave hand may include a robotic arm, a puncture tip connected to the end of the robotic arm, and a puncture needle mounted on the puncture tip. During the puncture operation, the robotic arm of the surgical robot's slave hand drives the puncture needle via the puncture tip, thereby achieving puncture of the target object with the puncture needle. Generally speaking, the puncture needle puncture operation may include posture adjustment and needle insertion, wherein the posture adjustment operation includes adjusting the posture of the puncture needle, and the needle insertion operation includes adjusting the puncture depth of the puncture needle.

[0062] Exemplarily, the first processor is used to convert the operation instructions applied by the doctor to the main hand of the surgical robot into movement instructions executable by the slave hand of the surgical robot, and send the movement instructions to the slave hand of the surgical robot. The slave hand of the surgical robot executes the movement instructions to drive the puncture needle to move, thereby completing the puncture operation.

[0063] It can be understood that in the embodiment shown in Figure 1, the first processor is arranged in the operating room. In other embodiments, the first processor can also be arranged outside the operating room, or the first processor can also be integrated with the surgical robot's slave hand or the surgical robot's main hand. This manual does not impose any specific restrictions on the specific setting position and form of the first processor.

[0064] It is understood that the communication connection in the embodiments of this specification may be a wireless communication connection, for example, the wireless communication connection may be implemented through wireless communication technologies such as Bluetooth (BT), wireless-fidelity (WiFi), or near-field communication (NFC). Of course, the communication connection in the embodiments of this specification may also be a wired communication connection, and this specification does not limit or elaborate on this.

[0065] The medical imaging device may be configured to perform a scanning operation on a target object to obtain a scanned image.

[0066] A medical imaging device is a medical scanning device capable of obtaining scanned images, and may be, for example, a CT scanner, an MR (Magnetic Resonance) scanner, a PET / MR (Positron Emission Tomography / Magnetic Resonance) scanner, a PET-CT (Positron Emission Tomography-Computed Tomography) scanner, or an ultrasound scanner. This specification does not limit the specific type of medical imaging device, as long as the medical imaging device can obtain a scanned image of a target object.

[0067] In some embodiments, the scanning operation of the medical imaging device on the target object can be manually controlled by the doctor. For example, while the doctor is operating the main hand of the surgical robot to perform a puncture operation, he or she can control the medical imaging device to perform a scanning operation by stepping on the control pedal of the medical imaging device.

[0068] In some other embodiments, the scanning operation of the medical imaging device can be automatically controlled by the puncture assist device.

[0069] For example, when a doctor is operating a surgical robot to perform a puncture operation, the main hand of the surgical robot sends the first puncture operation instruction to the first processor after receiving the first puncture operation instruction, and sends a motion instruction for the puncture operation to the puncture auxiliary device. After receiving the first puncture operation instruction, the first processor sends a fifth movement instruction corresponding to the first puncture operation instruction to the slave hand of the surgical robot. After receiving the motion instruction, the puncture auxiliary device sends a scanning instruction to the medical imaging device so that the medical imaging device performs a medical imaging scan on the target object (i.e., a real-time exposure scanning operation). After receiving the scanning instruction, the medical imaging device performs a scanning operation on the target object. The slave hand of the surgical robot moves the puncture needle according to the fifth movement instruction and performs the puncture operation corresponding to the first puncture operation instruction. The puncture operation includes a posture adjustment operation or a needle insertion operation. The posture adjustment operation is used to adjust the posture of the puncture needle, and the needle insertion operation is used to adjust the puncture depth of the puncture needle. It is understandable that the first puncture operation instruction can be the doctor's operation on the main hand of the surgical robot, the motion instruction can be the notification sent by the main hand of the surgical robot to the surgical robot of the puncture auxiliary device that the puncture operation is about to be performed, and the target object can be the patient. In this embodiment, when the main hand of the surgical robot receives the first puncture operation instruction, it prompts the puncture auxiliary device to control the medical imaging device to perform a scanning operation, so that the medical imaging device can be automatically controlled to perform real-time exposure scanning during the puncture operation, thereby ensuring that the medical imaging device will perform a scanning operation every time the main hand of the surgical robot receives a puncture operation instruction, thereby ensuring the timeliness of the medical image scan. In this embodiment, the medical image scan is performed using the time interval between the main hand of the surgical robot receiving the puncture operation instruction and the slave hand of the surgical robot executing the puncture operation instruction, so that the medical image scan can be performed before the puncture operation is performed, thereby avoiding blind puncture caused by the puncture operation having been performed but the medical image scan having not been performed.

[0070] It can be understood that the specific function of the fifth movement instruction is related to the puncture operation corresponding to the first puncture operation instruction. When the puncture operation corresponding to the first puncture operation instruction is a posture adjustment operation, the fifth movement instruction is used to control the surgical robot to move the puncture needle for posture adjustment; when the puncture operation corresponding to the first puncture operation instruction is a needle insertion operation, the fifth movement instruction is used to control the surgical robot to move the puncture needle for needle insertion.

[0071] For another example, when a doctor is operating a surgical robot to perform a puncture operation, the robot's main hand, after receiving a second puncture operation instruction, sends the second puncture operation instruction to the first processor. The first processor determines a sixth movement instruction corresponding to the second puncture instruction, then sends the sixth movement instruction to the robot's slave hand and a motion instruction to perform the puncture operation to the puncture assistance device. After receiving the motion instruction, the puncture assistance device sends a scanning instruction to the medical imaging device, causing the medical imaging device to perform a medical image scan of the target object. After receiving the scanning instruction, the medical imaging device performs a scan operation on the target object. The robot's slave hand moves the puncture needle according to the sixth movement instruction, performing the puncture operation corresponding to the second puncture operation instruction. The puncture operation includes a posture adjustment operation for adjusting the posture of the puncture needle and a needle insertion operation for adjusting the puncture depth of the puncture needle. It is understood that the second puncture operation instruction can be an operation performed by the doctor on the robot's main hand, and the motion instruction can be a notification from the first processor to the puncture assistance device that the robot is about to perform a puncture operation. The target object can be a patient. In this embodiment, after receiving the second puncture operation instruction, the surgical robot master hand sends the second puncture instruction to the first processor. Upon receiving the second puncture operation instruction, the first processor prompts the puncture auxiliary device to control the medical imaging device to perform a scanning operation. This enables automatic control of the medical imaging device for real-time exposure during the puncture operation, thereby ensuring that the medical imaging device performs a scanning operation each time the surgical robot master hand receives a puncture operation instruction, thereby ensuring the timeliness of the medical image scan. In this embodiment, the medical image scan is performed during the time interval between the first processor sending the sixth movement instruction corresponding to the second puncture operation instruction to the surgical robot slave hand and the surgical robot slave hand executing the sixth movement instruction. This allows the medical image scan to be performed before the puncture operation is performed, thereby avoiding blind punctures caused by the puncture operation having already been performed but the medical image scan has not yet been performed. Furthermore, the first processor sending the instruction information to the puncture auxiliary device shortens the scanning time compared to the surgical robot master hand sending the instruction information to the puncture auxiliary device, thereby reducing the radiation dose to the target object.

[0072] It can be understood that the specific function of the sixth movement instruction is related to the puncture operation corresponding to the second puncture operation instruction. When the puncture operation corresponding to the second puncture operation instruction is a posture adjustment operation, the sixth movement instruction is used to control the surgical robot to move the puncture needle for posture adjustment; when the puncture operation corresponding to the second puncture operation instruction is a needle insertion operation, the sixth movement instruction is used to control the surgical robot to move the puncture needle for needle insertion.

[0073] It should be understood that the first puncture operation instruction and the second puncture operation instruction may be the same or different, and this specification does not impose any limitation on this.

[0074] FIG2 is a schematic structural diagram of a puncture assisting device according to some embodiments of this specification.

[0075] As shown in FIG. 2 , the puncture assisting device in the embodiment of this specification includes a scanning module 210 , a puncture assisting module 220 and a display module 230 .

[0076] In some embodiments, the scanning module 210 may be configured to obtain a scanned image of the target object. For more information about the medical imaging device, please refer to the relevant description of FIG1 .

[0077] The target object refers to an object that needs to undergo a puncture interventional procedure. In some embodiments, the puncture interventional procedure includes but is not limited to at least one of percutaneous intervention, transvascular intervention, puncture biopsy, etc.

[0078] In some embodiments, the scanning operation may be a scanning operation performed before intraoperative puncture. In some embodiments, the scanning operation may be a scanning operation performed during a preoperative imaging scanning stage.

[0079] In some embodiments, the scanning operation can also be performed during posture adjustment or during needle insertion, where the posture adjustment process includes adjusting the posture of the puncture needle, and the needle insertion process includes adjusting the puncture depth of the puncture needle. Performing the scanning operation during posture adjustment or needle insertion ensures that the scanning operation is performed during the main stage of puncture, allowing the user to monitor the puncture process in real time through the scanned image, reducing the difficulty of puncture and improving puncture accuracy.

[0080] In some embodiments, during the posture adjustment process, the medical imaging device performs a scanning operation before the surgical robot performs the posture adjustment operation; during the needle insertion process, the medical imaging device performs a scanning operation before the surgical robot performs the needle insertion operation. By performing the scanning operation before the surgical robot performs the posture adjustment operation or the needle insertion operation, the scanning can ensure that the scanning guides the posture adjustment operation or the needle insertion operation, avoiding blind posture adjustment or needle insertion. For more information about the surgical robot, please refer to the relevant description of Figure 1.

[0081] A scanned image refers to an image obtained by scanning with a medical imaging device. In some embodiments, the scanned image may include at least one of a preoperative image, a first image taken during a first time period, and a second image taken during a second time period, wherein the first time period is earlier than the second time period.

[0082] Preoperative images refer to images of the target part (e.g., lesions and target organs) of the target object captured before the operation is performed when the user (e.g., the doctor performing the operation) is planning the operation (e.g., determining the surgical path, needle entry point, etc.). Preoperative images may include images of different slices of the target part of the target object. Preoperative images may include any time before the operation begins on the target object. For example, a few days before the operation, ten minutes before the operation, etc. The user can scan the target object before the operation (a few days before the operation, ten minutes before the operation, etc.) to obtain preoperative images, thereby facilitating surgical planning for the target object.

[0083] The scanning module 210 can use medical imaging equipment to perform a plain scan of the target area of ​​the target object before surgery to obtain a preoperative image. To determine the specific location of the lesion and the distribution of surrounding organs and tissues, the preoperative image can be scanned and reconstructed based on a larger scanning range (e.g., 200mm) to obtain an image that includes the target area and surrounding organs and tissues.

[0084] After obtaining the preoperative image, the scanning module 210 also needs to obtain the first image in the first time period to align the first image with the preoperative image, so as to obtain the deformation parameters under the corresponding respiratory amplitude parameters. For more information about the respiratory amplitude parameters and deformation parameters, please refer to the relevant description of Figure 12 of this manual. The first time period refers to a period of registration preparation time before the operation is performed, which may also be called the intraoperative preparation time period. For example, the first time period may be the first 3 to 7 seconds before the operation is performed. Among them, the time of operation refers to the time period from the formal start of the operation on the target object to the end of the operation. For example, the time of operation refers to the time period from the user formally starting to insert the surgical instruments (such as puncture needles, etc.) used in the interventional operation into the body of the target object to the removal of the aforementioned surgical instruments from the body of the target object. Surgical instruments refer to instruments used in interventional surgery. For example, surgical instruments may include but are not limited to puncture needles, drainage tubes, etc. The puncture needle may include multiple types of puncture needles.

[0085] The first image refers to an image obtained by real-time acquisition of the target part when the operation is about to be performed. For example, the first image can be a real-time scanning image obtained by acquiring the target part within a complete respiratory cycle (for example, 3 to 4 seconds) of the target object when the operation is about to be performed. For another example, the first image can be a real-time scanning image obtained by acquiring the target part when the target object is performing respiratory training when the operation is about to be performed. For another example, the first image can be a real-time scanning image obtained by acquiring the target part at any time when the target object has respiratory changes when the operation is about to be performed. In some embodiments, the first image is acquired after the preoperative image. For example, the preoperative image is acquired 2 to 3 days, 15 minutes, etc. before the operation is performed, and the first image is acquired 3 to 7 seconds before the operation is about to be performed.

[0086] The scanning module 210 can continuously scan the target object through a medical imaging device within a first time period, reconstruct in real time, and obtain a first image. In order to accurately reflect the situation of the lesion area, the first image can be an image obtained by scanning the target part based on a smaller scanning range (for example, 20mm) and reconstructing the image. In some embodiments, in order to subsequently determine multiple target guidance images to better guide the interventional surgery, the scanning module can collect multiple first images within the first time period. For the target guidance image, please refer to the relevant description of Figure 12.

[0087] In some embodiments, the scanning module 210 may also continuously acquire multiple scan images of the target object through a medical imaging device within a first time period, divide the multiple scan images into multiple intervals, and select a scan image in each interval as the first image. The deformation of the target part of the target object in each of the aforementioned intervals is less than a preset deformation threshold. For example, the scanning module 210 continuously acquires 30 scan images within the complete breathing interval of the target object in the first time period at a preset frequency. The scanning module 210 may evenly divide the aforementioned 30 scan images into 10 intervals, and select the middle scan image in each interval as the first image. For example, the second scan image, the fifth scan image, the seventh scan image, etc. are selected in sequence as the first image.

[0088] The second time period refers to the time period when the operation is performed, which may also be referred to as the intraoperative time period.

[0089] In some embodiments, the second image is an image acquired in real time from the target site during the execution of the surgery.

[0090] In some embodiments, the scanning module 210 may continuously scan the target part of the target object using a medical imaging device within the second time period and perform image reconstruction to obtain at least one second image. In some embodiments, the first image and the second image may be tomographic images.

[0091] In order to accurately reflect the deformation of the lesion area during the execution of the operation and to reduce the radiation dose of the target object during the operation, the second image can be an image obtained by scanning the target part based on a smaller scanning range (for example, 20mm) and reconstructing the image. It should be understood that in order to be able to determine multiple target guidance images later, so as to better guide the interventional surgery, the scanning module 210 can collect multiple second images within the second time period. It should be understood that the second image not only includes the target part, but also the puncture needle. However, since the second image is an image collected based on a smaller scanning range, the puncture needle may not be fully displayed in the second image, and the distribution of organ tissues around the lesion area cannot be reflected, so it cannot be directly used for interventional guidance of the surgery.

[0092] It should be understood that in order to perform real-time interventional guidance during an interventional procedure, the scanning module 210 needs to continuously scan and reconstruct the target area using medical imaging equipment at a rate greater than a rate threshold, which may be a default value (e.g., 10 fps).

[0093] Figure 15 is an exemplary diagram illustrating real-time image updating according to some embodiments of this specification. As an example only, as shown in Figure 15 , to enable real-time interventional guidance within time period T, the second image must be updated at a rate greater than a threshold. Image updating includes scanning the target area, image reconstruction, image registration of the second image with the preoperative image, and image transmission.

[0094] For more information about obtaining the scanned image, please refer to the relevant description of FIG. 12 .

[0095] In some embodiments, the puncture assisting module 220 may be configured to generate puncture guiding information based on the puncture status information and the scanned image; and guide the puncture operation based on the puncture guiding information.

[0096] The puncture state information refers to the state information of the puncture needle when performing a scanning operation on the target object. In some embodiments, the puncture state information may include puncture depth, puncture speed, and posture angle, etc. For relevant descriptions, please refer to the corresponding description of Figure 3.

[0097] In some embodiments, the processor can obtain puncture status information through the surgical robot. In some embodiments, the puncture assist module 220 can obtain puncture status information directly from the surgical robot, or it can obtain puncture status information indirectly from the surgical robot, and this specification does not limit this. For example, the surgical robot can directly report the puncture status information to the puncture assist module 220, or the puncture assist module 220 can actively obtain the puncture status information from the surgical robot. Alternatively, the surgical robot can also send the puncture status information to a transfer device, and the puncture assist module 220 obtains the puncture status information from the transfer device. For more information about the surgical robot, please refer to the relevant description of Figure 1.

[0098] The puncture guidance information is guidance information that helps perform a puncture operation on a target object.

[0099] In some embodiments, the puncture guidance information may include positioning information such as positioning position information, first position information, second position information, third position information and target position information, and may also include alarm information prompting needle withdrawal when the puncture needle is deformed, etc. For related instructions, please refer to the corresponding descriptions of Figures 22-27.

[0100] In some embodiments, the puncture guidance information may also include the surface puncture point, real-time needle tip position, predicted target point, lesion location, related information of the posture adjustment process, various annotated images, etc. For related explanations, please refer to the corresponding descriptions of Figures 3-11.

[0101] In some embodiments, the puncture guidance information may further include a target guidance image, etc. For related explanations, please refer to the corresponding descriptions of Figures 12 to 21.

[0102] Puncture operation refers to the related operations during puncture surgery.

[0103] In some embodiments, the display module 230 may be configured to display puncture guidance information.

[0104] In some embodiments, the display module 230 can display the puncture guidance information in various ways. For example, the display module 230 can control the medical imaging device to display the puncture guidance information.

[0105] In some embodiments, the puncture assist device may further include a positioning correction module. For more details, please refer to the relevant description of Figure 22.

[0106] In some embodiments of the present specification, puncture guidance information is generated based on puncture status information and scanned images to guide the puncture operation, which can more intuitively guide doctors to perform the puncture operation and improve the accuracy and safety of the puncture surgery.

[0107] FIG3 is an exemplary interactive diagram of a puncture method according to some embodiments of the present specification. As shown in FIG3 , the puncture method includes the following steps:

[0108] S301: A medical imaging device scans a target object to obtain a scanned image. For more details, please refer to the description of FIG2 .

[0109] S302: The medical imaging device sends the scanned image to the puncture assisting device.

[0110] S303: The surgical robot obtains puncture status information and sends it to the puncture auxiliary device.

[0111] For more information about how the surgical robot obtains puncture status information, please refer to the relevant description in Figure 2.

[0112] S304: Determine the position information of the first needle tip of the puncture needle and the position information of the predicted target point based on the puncture state information.

[0113] The position information of the first needle tip is calculated based on the relative positional relationship between the puncture needle and the surgical robot. That is, the position information of the first needle tip is a calculated theoretical value. When the puncture needle is in a normal state, the calculated position information of the first needle tip is accurate. In this case, the puncture assist device directly obtains the position information of the first needle tip from the puncture status information. It is understood that the puncture needle being in a normal state can mean that the puncture needle is not bent, broken, or otherwise abnormal.

[0114] Exemplarily, the position information of the first needle tip point may include the coordinates of the needle tip point, which can be obtained by calculating the joint angles of the robotic arm. The specific calculation process will not be described in detail here.

[0115] In some embodiments, in addition to directly obtaining the coordinates of the needle tip from the surgical robot, the puncture assist device may also obtain relevant parameters from the surgical robot and calculate the coordinates of the needle tip by calculating the joint angles of the robotic arm based on the relevant parameters. In some embodiments, when the robotic arm is a 5-axis joint robotic arm, the specific calculation process can be referred to the following formula (1):

[0116] in, is the spinor coordinate of each joint of the manipulator, θ i ,i=1,…,5 is the joint angle between each axis of the axial joint manipulator of the surgical robot, It represents the homogeneous transformation matrix of the control point coordinate system in the robot coordinate system when each joint of the manipulator is at zero position. T is the multi-axis robot pose matrix, which contains the robot's pose information and position information, where the position information is Indicates the initial coordinates of the needle tip in the robot coordinate system.

[0117] In some embodiments, when the needle is inserted, the real-time coordinates of the needle tip can be calculated by the following formula (2):

[0118] in, is the coordinate of the real-time needle tip point in the robot coordinate system, and (0,0,-depth) is the movement distance of the real-time needle tip point in the X, Y, and Z directions of the robot coordinate system.

[0119] In some embodiments, the position information of the first needle tip point may also be the position information of the needle tip point in the target position information obtained after correction. For more relevant content, please refer to the relevant description of Figure 22.

[0120] In some embodiments, the predicted target point refers to the spatial point reached by the needle tip of the puncture needle after the puncture needle completes the insertion according to the posture in the puncture state information.

[0121] Exemplarily, the position information of the predicted target includes the coordinates of the predicted target. The puncture assist device can obtain the coordinates of the predicted target based on the initial coordinates of the needle tip in the robot coordinate system and the length of the puncture needle. The specific calculation process can be seen in the following formula (3):

[0122] in, is the initial coordinate of the needle tip in the robot coordinate system, length is the length of the puncture needle, To predict the coordinates of the target in the robot coordinate system.

[0123] S305 : Generate a labeled image based on the position information of the first needle tip point and the position information of the predicted target point.

[0124] In some embodiments, the annotated image is an image in which the needle tip point, predicted target point, etc. are annotated on the target image.

[0125] In some embodiments, the target image may include a preoperative image, a first image, one or two images of a plurality of second images, and a target guide image. For more description of the second image, please refer to the relevant description of FIG. 2 .

[0126] In some embodiments, the target image may be determined based on the scanned image.

[0127] In some embodiments, the puncture assist device is further configured to determine a preoperative image and a first image in the scan image as the target image.

[0128] In some embodiments, the puncture assist device is further configured to determine a second image in the scan image as a target image.

[0129] In some embodiments, the puncture assist device can be further configured to determine a target image (ie, a target guide image) for the second time period based on the preoperative image, the first image, and the second image. For more information, please refer to the relevant description of FIG. 12 .

[0130] In some embodiments, the puncture assist device can generate a labeled image based on the position information of the first needle tip point and the position information of the predicted target point. For example, the puncture assist device can determine the corresponding second image based on the position information of the first needle tip point, and determine the second image as the target image. The puncture assist device can mark the needle tip point on the target image to generate a labeled image. Exemplarily, after determining the target image where the needle tip point is located, the puncture assist device can convert the coordinates of the needle tip point from the robot coordinate system to the image coordinate system where the second image obtained by the medical imaging device is located, and then mark the needle tip point on the corresponding target image to generate a labeled image. Similarly, in some embodiments, the puncture assist device can mark the predicted target point on the target image based on the position information of the predicted target point to generate a labeled image.

[0131] In some embodiments, the puncture assist device can mark the needle tip point and the predicted target point on the same target image or two different target images. The needle tip point and the predicted target point each occupy a certain three-dimensional space. The needle tip point can be located in one or more target images of the target object, so the target images involving the needle tip point can include one or more. Similarly, the predicted target point can also be located in one or more faults of the target object, so the target images involving the predicted target point can include one or more. In this specification, the quantifiers "one", "a plurality of", etc. related to the description of images can represent one, or more, etc. images or one or more layers of images, and this specification does not impose any restrictions.

[0132] For example, when the needle tip point and the predicted target point are located on the same slice of the target object, the needle tip point and the predicted target point can be annotated on the same target image. In this case, the target image includes one second image from the plurality of second images, and the annotated target image is referred to as the annotated image, which is 1 in number. When the needle tip point and the predicted target point are located on different slices of the target object, the needle tip point and the predicted target point are annotated on two different second images. In this case, the target image includes at least two second images from the plurality of second images, and the annotated target images are referred to as the annotated images, which are 2 in number.

[0133] S306: Display the annotated image.

[0134] It is understandable that the displayed annotated image, ie, the annotated target image, may be a partial tomographic image in the target image or may be all tomographic images in the target image, and this specification does not impose any limitation on this.

[0135] In some embodiments, the needle tip point and the predicted target point can be located on the target image of the same layer or on target images of different layers. When the needle tip point and the predicted target point are marked on the target image of the same layer, the connecting line between the needle tip point and the predicted target point is also displayed on the marked image. For more information, please refer to the relevant descriptions of Figures 8 and 9.

[0136] In some embodiments, the puncture assist device displays puncture status information via a display module. The displayed puncture status information may include at least one of puncture depth, puncture speed, and attitude angle. The puncture depth indicates the depth of the puncture needle tip into the skin. The attitude angle indicates the angle of the puncture needle relative to the target object. The puncture speed indicates the speed of the puncture needle moving along the insertion direction.

[0137] In some embodiments, when the puncture assist device determines that the puncture speed exceeds a certain threshold, the display module displays a warning message indicating that the puncture is too fast. By providing a warning regarding excessive puncture speed, the probability of excessive puncture can be reduced, thereby improving the safety of the puncture procedure. For example, when the puncture speed exceeds 30 mm / s, a warning message indicating that the puncture is too fast is displayed. The specific form of the warning message may be a pop-up prompt window, or a change in the display method of the puncture speed, such as displaying a warning icon along with the displayed puncture speed. This specification does not limit the form of the warning message.

[0138] In some embodiments, the puncture assist device can also mark the target lesion of the target object. Specifically, the puncture assist device marks the target lesion of the target object on a second image among the multiple tomographic images, obtaining a marked second image, which includes the tomographic image of the multiple tomographic images in which the target lesion is scanned; and displays the marked second image via a display module. In this embodiment, by marking the target lesion, the user can more clearly see the target lesion, thereby enabling more precise control of the puncture process and improving the user experience.

[0139] It is understandable that the lesion is generally a three-dimensional structure, so the target lesion of the target object may exist in multiple slices, that is, the target lesion may be confirmed by multiple slice images obtained by scanning.

[0140] In some embodiments, the second image includes only one tomographic image, then the second image can be a tomographic image scanned to the center of the target lesion, or the second image can be a tomographic image with the largest lesion area, or the second image can be a tomographic image that meets other conditions, and this specification does not limit this.

[0141] In some other embodiments, the second image may include multiple tomographic images, or the second image may include all tomographic images of the target lesion, which is not limited in this specification.

[0142] In some embodiments, the target lesion is also annotated on the tomographic image labeled with the predicted target point, i.e., the first image labeled with the predicted target point and the second image labeled with the target lesion can be the same tomographic image. By annotating the target lesion on the tomographic image where the predicted target point is located, it is convenient for the user to determine whether the target lesion can be punctured using the current puncture state of the puncture needle. For example, when the predicted target point coincides with the target lesion, or the predicted target point is within the outline of the target lesion, it can be determined that the target lesion can be punctured using the current puncture state of the puncture needle. Otherwise, it is determined that the target lesion cannot be punctured using the current puncture state of the puncture needle.

[0143] In some other embodiments, the needle tip point, predicted target point and target lesion can be marked on the same tomographic image, the first image marked with the needle tip point and predicted target point includes a tomographic image, the second image marked with the target lesion includes a tomographic image, and the first image and the second image are the same tomographic image.

[0144] Exemplarily, the puncture assist device displays the annotated second image through the display module and also displays adjacent scanned images of the second image through the display module. The display of adjacent scanned images of the second image in this embodiment is substantially the same as the display of adjacent scanned images of the first image described above in terms of method and technical effect, and is not further described here.

[0145] In some embodiments, the puncture assist device further displays a three-dimensional image of the puncture needle and a three-dimensional image of the target lesion via a display module. The three-dimensional image of the puncture needle and the three-dimensional image of the target lesion are each obtained using multiple tomographic images. The three-dimensional image of the puncture needle and the three-dimensional image of the target lesion can be obtained by three-dimensional reconstruction using multiple tomographic images, or can be generated using an image recognition algorithm, which is not limited in this specification. In this embodiment, the display of the three-dimensional image makes the presentation of the puncture needle and the target lesion more three-dimensional and vivid.

[0146] For example, the puncture needle and target lesion are displayed in the same 3D image, and the relative positional relationship between the puncture needle and the target lesion in the 3D image is the same as the relative positional relationship between the puncture needle and the target lesion in the image coordinate system, where the image coordinate system is the coordinate system in which the multiple tomographic images are located. By displaying the 3D images of the puncture needle and the target lesion in the same 3D image and faithfully presenting the relative positional relationship between the puncture needle and the target lesion in the image coordinate system, the puncture progress can be more intuitively viewed, making it easier for the user to determine whether the puncture is complete.

[0147] For example, in the same 3D image, the puncture needle and the target lesion are displayed in different colors. Displaying the puncture needle and the target lesion in different colors makes it easier for the user to distinguish the puncture needle and the target lesion, and the user can better judge the relative position relationship between the puncture needle and the target lesion.

[0148] In some embodiments, the processor can flexibly set and display the exposure range and exposure interval of the annotated image.

[0149] The exposure range refers to data reflecting the exposure level of the annotated image. For example, the exposure range may include the center of the annotated image, the layer thickness of the annotated image, and the number of layers radiating outward.

[0150] The exposure interval refers to the time interval between two adjacent exposures.

[0151] In some embodiments, the processor can flexibly set the exposure range and exposure interval of the annotated image in a variety of ways. For example, the processor can retrieve the doctor's historical settings within a preset historical time period from memory and set the exposure range and exposure interval that were used most frequently in the historical settings as the exposure range and exposure interval of the current annotated image.

[0152] In some embodiments, the processor can display the exposure range of the annotated image in various ways. For example, the processor can control the medical imaging device to display the exposure range and exposure interval of the annotated image while displaying the annotated image.

[0153] In some embodiments, the processor may determine the exposure range based on the puncture range and lesion characteristics.

[0154] The puncture range refers to the path of the puncture needle. The puncture range can be preset in advance.

[0155] Lesion characteristics refer to relevant features that reflect the characteristics of the lesion. For example, lesion characteristics may include the size, shape, and organ where the lesion is located.

[0156] In some embodiments, the processor may determine the exposure range based on a first preset relationship between the puncture range, lesion characteristics, and exposure range. The first preset relationship may be preset based on actual conditions and historical experience. For example, the first preset relationship may include a correlation between the puncture range and the exposure range. Exemplarily, if the current exposure range cannot cover the puncture range, the exposure range is increased. For another example, the first preset relationship may include a correlation between the lesion characteristics and the exposure range. Exemplarily, if the current exposure range cannot cover the lesion requiring puncture, the exposure range is increased. If the current exposure range is much larger than the lesion, making it impossible to clearly display the lesion, the exposure range is reduced.

[0157] In some embodiments, the processor may determine the exposure range based on the puncture range and lesion characteristics using a first vector database. For example, the processor may construct a first vector database based on historical puncture ranges, historical lesion characteristics, and actual exposure ranges in historical records, construct a first feature vector based on the current puncture range and lesion characteristics, match the first feature vector in the first vector database, and determine the actual exposure range corresponding to the first reference vector with the highest similarity as the current corresponding exposure range.

[0158] The first vector database includes a first reference vector and its corresponding actual exposure range. The first reference vector is composed of the actual historical puncture range and historical lesion characteristics in the historical records. The corresponding actual exposure range can be determined based on the actual exposure range with the highest lesion coverage during the historical surgery. Similarity can be calculated based on cosine distance, Euclidean distance, etc.

[0159] In some embodiments, the processor may determine the exposure range based on the puncture range and lesion characteristics through an exposure range prediction model.

[0160] In some embodiments, the exposure range prediction model may be a machine learning model, for example, a neural network model, a deep neural network model, or any combination thereof.

[0161] In some embodiments, the exposure range prediction model can be obtained through training using a first training sample and a first label. The first training sample can be a sample puncture range and sample lesion characteristics in the historical records, and the first label can be constructed based on the first training sample. For example, the processor can construct a first label based on the distance between the center point of the puncture range and the center point of the exposure range in the historical surgery corresponding to the first training sample, such as taking the exposure range with the shortest distance as the first label. For another example, the processor can construct a first label based on the coverage rate of the lesion in the historical surgery corresponding to the first training sample, such as taking the exposure range corresponding to the historical surgery with the highest coverage rate as the first label.

[0162] In some embodiments, the processor may input a large number of first training samples into the initial exposure range prediction model, construct a loss function based on the output of the initial exposure range prediction model and the first label, and iteratively update the initial exposure range prediction model based on the loss function. When the value of the loss function meets an iteration completion condition, training is completed, resulting in a trained exposure range prediction model. The iteration completion condition may include convergence of the loss function, a number of iterations reaching a threshold, etc.

[0163] In some embodiments of the present specification, by determining the exposure range based on the puncture range and lesion characteristics, a more reasonable exposure range can be determined to obtain a more accurate scan image, thereby providing the doctor with a better real-time image reference during the puncture process.

[0164] In some embodiments, the processor may further determine the exposure range based on the adjustment range of the posture of the puncture needle.

[0165] The adjustment range of the puncture needle's posture refers to an indicator reflecting the adjustment of the puncture needle's posture. For example, it may include the adjustment angle and adjustment distance of the puncture needle's posture. The adjustment angle of the puncture needle's posture may include angles related to intra-layer adjustment and angles related to cross-layer adjustment. For more information on the adjustment of the puncture needle's posture, please refer to the relevant descriptions of Figure 4 or Figures 9-11.

[0166] In some embodiments, the processor may determine the exposure range based on a second preset relationship between the adjustment amplitude of the puncture needle posture and the exposure range. The second preset relationship may be preset based on actual conditions and historical experience. For example, the second preset relationship may include a correlation between the adjustment angle of the puncture needle posture and the exposure range. For example, when the adjustment angle of the puncture needle posture is a cross-layer posture adjustment angle, the center of the scanned image is adjusted so that the scanned image can be marked with the predicted target point after the puncture needle posture adjustment.

[0167] In some embodiments, the processor may determine the exposure range based on the adjustment amplitude of the puncture needle posture using a second vector database. For example, the processor may construct a second vector database based on the historical records of the adjustment amplitude of the puncture needle posture and the actual exposure range, construct a second feature vector based on the current adjustment amplitude of the puncture needle posture, match it with the second reference vector in the second vector database, and determine the actual exposure range corresponding to the second reference vector with the highest similarity as the current corresponding exposure range.

[0168] Among them, the second vector database includes a second reference vector and its corresponding actual exposure range. The second reference vector is composed of the adjustment amplitude of the actual historical puncture needle posture in the historical records. The corresponding actual exposure range can be determined in a similar way to determining the first label, and the similarity can be calculated based on cosine distance, Euclidean distance, etc.

[0169] In some embodiments, the input to the exposure range prediction model may further include an adjustment range of the puncture needle's posture. In some embodiments, when the input to the exposure range prediction model includes the adjustment range of the puncture needle's posture, the first training sample may further include historical puncture needle posture adjustment ranges in the historical data.

[0170] In some embodiments of this specification, when determining the exposure range, by considering the adjustment range of the puncture needle posture, the determined exposure range can be made more consistent with the actual situation, and a more reference-worthy scan image can be obtained, which helps doctors perform more accurate puncture operations.

[0171] In some embodiments, the processor may further determine the interval between two adjacent exposures based on the riskiness of the current puncture needle area, lesion characteristics, and the adjustment range of the puncture needle posture.

[0172] The hazard of the puncture needle area is an indicator that reflects the possibility and degree of danger of the puncture needle causing harm to the target object.

[0173] In some embodiments, the processor may construct a first preset table based on the distance between the puncture needle tip and the blood vessel or bone, and the riskiness of the puncture needle area, and determine the riskiness of the current puncture needle area by querying the first preset table. The first preset table contains a correspondence between the distance between the puncture needle tip and the blood vessel or bone, and the riskiness of the puncture needle area, and may be constructed based on historical data or manually preset based on experience. For example, the correspondence in the first preset table may be that the smaller the distance between the puncture needle tip and the blood vessel or bone, the higher the riskiness of the corresponding puncture needle area.

[0174] In some embodiments, the processor may determine the exposure interval by matching a third vector database based on the current riskiness of the puncture needle area, the characteristics of the lesion, and the adjustment range of the puncture needle posture. For example, the processor may construct a third vector database based on the historical riskiness of the puncture needle area, the characteristics of the lesion, the adjustment range of the puncture needle posture, and the actual exposure interval. The processor may construct a third feature vector based on the current riskiness of the puncture needle area, the characteristics of the lesion, and the adjustment range of the puncture needle posture, match the third feature vector in the third vector database, and determine the actual exposure interval corresponding to the third reference vector with the highest similarity as the current corresponding exposure interval.

[0175] Among them, the third vector database includes a third reference vector and its corresponding actual exposure interval time. The third reference vector is composed of the actual dangerousness of the historical puncture needle area, the historical lesion characteristics and the adjustment range of the historical puncture needle posture in the historical records. The corresponding actual exposure interval time can be determined in a similar way to determining the second label later, and the similarity can be calculated based on cosine distance, Euclidean distance, etc.

[0176] In some embodiments, the processor may determine the exposure interval based on the riskiness of the current puncture needle area, lesion characteristics, and the adjustment range of the puncture needle posture using an exposure interval prediction model. In some embodiments, the exposure interval prediction model may be a machine learning model. For example, the exposure interval prediction model may be a neural network model, a deep neural network model, or any combination thereof.

[0177] In some embodiments, the exposure interval prediction model can be obtained through training using a second training sample and a second label. The second training sample can be the dangerousness of the sample puncture needle area in the historical record, the sample lesion characteristics, and the adjustment range of the sample puncture needle posture. The second label can be constructed based on the second training sample. For example, the processor can construct a second label based on the overlap rate between the puncture needle path captured by the medical imaging device after the puncture needle posture adjustment in the historical operation corresponding to the second training sample and the planned puncture needle path, such as using the exposure interval time corresponding to the historical operation with the highest overlap rate as the second label.

[0178] The specific training process of the exposure interval prediction model is similar to that of the exposure range prediction model. For more information, please refer to the corresponding instructions above.

[0179] In some embodiments of this specification, by determining the exposure interval based on the dangerousness of the current puncture needle area, the characteristics of the lesion, and the adjustment range of the puncture needle posture, the actual situation can be better combined to determine an exposure interval that better meets the current needs to display a more accurately labeled image.

[0180] By flexibly setting and displaying the exposure range and exposure interval of the annotated image, the display of the annotated image can be made more in line with the user's actual needs, further improving the user experience.

[0181] FIG4 is an exemplary schematic diagram of a puncture procedure according to some embodiments of the present specification.

[0182] As shown in FIG4 , the puncture process mainly includes: S401 to S406 , and each step is described below.

[0183] S401, preoperative imaging scan.

[0184] In some embodiments, before the puncture procedure begins, a medical imaging device is used to scan the affected area of ​​the target subject. For example, when the medical imaging device is a CT scanner, the CT scanner can be used to perform multi-phase contrast-enhanced scanning and plain scanning. The images obtained through the scan can clearly identify the characteristics of the lesion (such as the nature, contour, and extent of the lesion) and enable analysis of important organs.

[0185] For example, during the preoperative scanning phase, the liver in the abdomen and the lung lobes in the chest can be reconstructed in three dimensions in the transverse, coronal and sagittal directions, and displayed using VR (Virtual Reality).

[0186] In the embodiment shown in FIG4 , the tumor site is used as the target location for puncture, and operations such as target organ segmentation, tumor site extraction, and dangerous tissue identification can be achieved during the preoperative image scanning stage.

[0187] For example, during the preoperative imaging scanning stage, the contours of important tissues and organs such as lesions, soft tissues, and target organs can be extracted based on the Dicom images obtained by the CT scanning equipment, and the processor displays the extracted images through the display module.

[0188] Figure 5 is an exemplary schematic diagram of tissue segmentation obtained during a preoperative imaging scan according to some embodiments of this specification. As shown in Figure 5 , in this embodiment, a CT scanner is used for scanning, wherein a lesion is located in the liver, and the outline of the lesion 501 is marked in Figure 5 . In this embodiment, marking the lesion in the image during the preoperative imaging scan can assist the physician in locating the patient's lesion and understanding the surrounding conditions.

[0189] S402, initial puncture path planning.

[0190] Based on the tissue model obtained during the preoperative imaging scan, i.e., the tissue segmentation diagram shown in Figure 5 , the initial puncture path is planned using a variety of planning methods (such as automatic planning, semi-automatic planning, and manual planning) while avoiding important tissue organs such as blood vessels, nerves, and bones.

[0191] The specific contents of the initial puncture path may include: the target object's surface puncture point, the puncture target point, the needle insertion angle, and the needle insertion depth. For example, Figure 6 is an exemplary schematic diagram of an initial puncture path displayed on an image obtained by a scanning device according to some embodiments of this specification. As shown in Figure 6, in this embodiment, the initial puncture path includes a surface puncture point 601, a puncture target point 602, and a needle track 603 connected to the surface puncture point 601 and the puncture target point 602, wherein the puncture target point 602 is located within the lesion outline 604.

[0192] In some embodiments, the initial puncture path may also be displayed on a 3D image, wherein the conversion of a two-dimensional image to a 3D image may be achieved by three-dimensional reconstruction, which will not be elaborated in this specification.

[0193] S403, navigation and positioning.

[0194] The terminal puncture execution device is connected to the slave hand of the surgical robot, and the terminal puncture execution device is fixed next to the medical imaging device. Then, the registration relationship between the three spaces of the target object space, the robotic arm space of the slave hand of the surgical robot, and the medical imaging space of the medical imaging device is determined by the navigation device of the surgical robot to obtain a registration matrix. The coordinate conversion between the three spaces can be achieved using the registration matrix, wherein the target object space can also be referred to as the patient space. In some embodiments, the navigation device refers to an NDI stereotactic navigation device, etc. In some embodiments, no navigation device is required. A medical imaging device can be used to scan the puncture execution device on the end of the surgical robot to achieve registration and registration of the surgical robot and the medical imaging device. Then, the coordinate conversion relationship between the medical imaging device and the target object is obtained through the preoperative image, and then the coordinate conversion relationship of the three spatial coordinate systems is obtained.

[0195] S404, puncture preparation.

[0196] The doctor selects a needle of appropriate length based on the insertion depth in the initial puncture path plan and installs the needle on the puncture tip of the surgical robot's slave hand. Once the needle is installed, the robot's slave hand automatically controls the robotic arm to move the needle to the puncture starting point, which is the surface puncture point in the initial path plan. The doctor also adjusts the needle's insertion angle to match the initial puncture path plan.

[0197] S405, intraoperative puncture.

[0198] As shown in Figure 4, the intraoperative puncture phase primarily includes three steps: needle track posture adjustment, needle track insertion, and real-time exposure. Needle track posture adjustment primarily involves adjusting the puncture needle's posture, while needle track insertion primarily involves adjusting the puncture depth. Real-time exposure primarily involves scanning the target object during posture adjustment and needle insertion.

[0199] In the embodiments of this specification, the physician can select automatic or manual posture adjustment based on the clinical situation. Automatic posture adjustment refers to the puncture assist device automatically controlling the surgical robot to adjust the posture of the puncture needle. Manual posture adjustment refers to the physician manipulating the surgical robot to adjust the posture of the puncture needle, such as by translating or rotating it.

[0200] In the embodiments of the present specification, during the process of needle track posture adjustment and needle insertion, real-time exposure is performed by controlling the medical imaging device, and the exposure range is set within the puncture range, that is, the exposure range is set within the fault range where the needle track is located, so as to observe the movement direction of the puncture needle in the patient's body and the actual position reached by the needle tip of the puncture needle in real time, and then judge the current puncture situation, thereby avoiding the impact of the puncture on the patient's important tissues and organs.

[0201] S406, completing the puncture.

[0202] After the needle is inserted into the needle tract, the needle tip of the puncture needle reaches the lesion area, completing the puncture operation.

[0203] It should be understood that the lesion area in the embodiments of this specification is a broad concept, and the definition of the lesion area may vary depending on the type of surgery. For example, for puncture procedures such as biopsy and injection, the lesion area refers to the area enclosed by the lesion outline; for puncture procedures such as ablation and seed implantation, the lesion area includes the area enclosed by the lesion outline, as well as the area within a certain range of the lesion edge or periphery.

[0204] It is understood that in the above-mentioned puncture process, the intraoperative puncture stage is the most critical and important stage. The puncture method in the embodiment of this specification is mainly improved for the intraoperative puncture stage. The puncture method in the embodiment of this specification is described below with reference to the accompanying drawings.

[0205] For example, the processor can display the tomographic image where the center of the needle tip point is located among all tomographic images marked with the needle tip point through the display module; and the processor can display the tomographic image where the center of the predicted target point is located among all tomographic images marked with the predicted target point through the display module.

[0206] It should be noted that the above description of the relevant processes is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the process under the guidance of this specification. For example, the order of step S401, step S402 and step S403 is only for example and explanation, and step S403 is not necessarily after step S402. Exemplarily, the registration and alignment of the surgical robot and the medical imaging device can be performed before step S401. Step S401 obtains the preoperative image of the target object, that is, obtains the coordinate relationship between the target object and the medical imaging device, and then obtains the coordinate transformation relationship of the three spatial coordinate systems. However, these modifications and changes are still within the scope of this specification.

[0207] In some embodiments, the position information of the first needle tip point can be calculated based on the relative position relationship between the puncture needle and the surgical robot. For more details, please refer to the relevant description of Figure 2 above.

[0208] In some embodiments, the processor may further determine the needle tract profile of the puncture needle based on the target image. The needle tract profile may represent the three-dimensional structure of the puncture needle. In some embodiments, the needle tract profile of the puncture needle may be obtained by three-dimensionally reconstructing the puncture needle based on the target image. In some embodiments, the processor may employ an image recognition algorithm to obtain the needle tract profile of the puncture needle, wherein the image recognition algorithm may be a machine learning algorithm. Both the three-dimensional reconstruction and image recognition algorithms may employ conventional algorithms, and this specification does not limit or elaborate on these algorithms.

[0209] For example, FIG7 is an exemplary schematic diagram of an extracted needle tract contour according to some embodiments of this specification. As shown in FIG7 , the medical imaging device in this embodiment is a CT scanner. FIG7 a shows the target image before needle tract contour extraction, and FIG7 b shows the target image after needle tract contour extraction. FIG7 b also shows the target lesion, i.e., FIG7 b shows a needle tract contour 701 and a target lesion 702.

[0210] In some embodiments, the processor may determine the position information of the second needle tip point based on the needle track profile.

[0211] The second needle tip location information represents the location of the needle tip of the puncture needle within the needle tract contour, for example, the actual location of the needle tip. The second needle tip location information may include actual coordinates of the needle tip. The actual coordinates of the needle tip may be obtained based on the extracted needle tract contour.

[0212] In some embodiments, the processor may determine whether there is an abnormality in the needle track profile based on the position information of the first needle tip point and the position information of the second needle tip point.

[0213] For ease of understanding, the following is an exemplary description of the process of determining whether there is an abnormality in the needle track profile based on the position information of the first needle tip point and the position information of the second needle tip point.

[0214] Exemplarily, the position information of the first needle tip point includes the theoretical coordinates of the needle tip point, which are the coordinates of the needle tip point reported by the surgical robot; the position information of the second needle tip point includes the actual coordinates of the needle tip point, which are obtained based on the extracted needle track contour; since the actual coordinates of the needle tip point are located in the image coordinate system, when determining whether there is an abnormality in the needle track contour, the actual coordinates of the needle tip point are first converted to the robot coordinate system to obtain the converted actual coordinates of the needle tip point; then, in the robot coordinate system, the converted actual coordinates of the needle tip point are compared with the theoretical coordinates of the needle tip point; if the distance between the converted actual coordinates of the needle tip point and the theoretical coordinates of the needle tip point is greater than a preset error threshold, it is determined that there is an abnormality in the needle track contour; if the distance between the converted actual coordinates of the needle tip point and the theoretical coordinates of the needle tip point is less than or equal to the preset error threshold, it is determined that there is no abnormality in the needle track contour.

[0215] In some embodiments, when determining whether there is an abnormality in the needle path contour, the theoretical coordinates of the needle tip point in the robot coordinate system can be first converted to the image coordinate system to obtain the converted theoretical coordinates of the needle tip point; then, in the image coordinate system, the converted theoretical coordinates of the needle tip point are compared with the actual coordinates of the needle tip point. If the distance between the converted theoretical coordinates of the needle tip point and the actual coordinates of the needle tip point is greater than a preset error threshold, it is determined that there is an abnormality in the needle path contour; if the distance between the converted theoretical coordinates of the needle tip point and the actual coordinates of the needle tip point is less than or equal to the preset error threshold, it is determined that there is no abnormality in the needle path contour.

[0216] In some embodiments, in response to the presence of an abnormality in the needle track profile, the processor may issue a needle track abnormality warning.

[0217] For example, the needle tract abnormality warning generated by the processor can be displayed on a display module. For example, the processor can display a needle tract abnormality prompt on the display module. The needle tract abnormality prompt can be a prompt window; alternatively, the abnormality warning can be issued in other forms, such as an alarm module installed in the processor that issues an abnormality alarm. This specification does not limit the specific form of the needle tract abnormality warning.

[0218] In this embodiment, the needle tract abnormality warning can promptly prompt the user with needle tract abnormality information, thereby preventing the influence of the needle tract abnormality on the puncture accuracy.

[0219] In some embodiments, the processor can provide an early warning of needle path deviation of the puncture needle during the puncture process.

[0220] Needle track deviation is the deviation between the actual needle puncture path and the target puncture path. For more information about the actual puncture path and the target puncture path, please refer to the following relevant descriptions.

[0221] The early warning is used to remind the user in advance of the possible risks of the next puncture. The early warning corresponds to the early warning parameter. In some embodiments, the early warning parameters may include the distance for the early warning, the content displayed in the early warning, etc. The distance for the early warning refers to the distance between the actual needle tip position of the next puncture and the target needle tip position of the next puncture. The content displayed in the early warning refers to the content related to the next puncture displayed to the user. For example, the content displayed in the early warning may include the needle track is about to deviate, the needle track deviation is too large, please adjust it in time, etc. In some embodiments, the early warning parameters may also include preset thresholds. For more details, please refer to the relevant description below.

[0222] In some embodiments, the processor may determine early warning parameters for early warning based on information about the target of the next puncture and the importance of tissues surrounding the puncture location.

[0223] The object information for the next puncture is information reflecting relevant features corresponding to the part to be punctured next. For example, it may include the outline of the organ, blood vessel, and bone on the puncture needle trajectory. In some embodiments, the processor may determine the object information for the next puncture based on the target image or the target puncture path.

[0224] The importance of the tissue surrounding the puncture site refers to the importance of the tissue surrounding the puncture site to the human body. In some embodiments, a technician or user can preset a second preset table containing the importance of various tissues of the human body, and the processor can determine the importance of the tissue surrounding the puncture site by querying the second preset table.

[0225] In some embodiments, the processor may determine the early warning parameter for the early warning based on a third preset relationship between the object information for the next puncture, the importance of the tissue surrounding the puncture location, and the early warning parameter. The third preset relationship may be preset based on actual conditions and historical experience. For example, the third preset relationship may include a correlation between the object information for the next puncture and the distance for early warning, which may be preset manually based on experience. For another example, the third preset relationship may also include a correspondence between the distance for early warning, the importance of the tissue surrounding the puncture location, and the content displayed in the early warning, which may also be preset manually.

[0226] In some embodiments of this specification, by determining early warning parameters based on the object information of the next puncture and the importance of the tissue around the puncture location, early warning parameters that are more in line with the actual situation can be determined to provide more effective early warnings and make reasonable predictions on the deviation of the needle track.

[0227] In some embodiments, the processor can predict the estimated deviation of the next puncture based on the deviation information of the actual puncture path, the target puncture path and the needle track deviation; and determine the early warning parameter based on the estimated deviation.

[0228] The actual puncture path refers to the path that the puncture needle track currently actually travels.

[0229] The target puncture path refers to the initially planned puncture path or the ideal path of the puncture needle after posture adjustment.

[0230] The needle track deviation information is information reflecting the degree of deviation between the actual puncture path and the target puncture path, and may include, for example, a deviation angle, a deviation distance of the puncture needle tip position, and the like.

[0231] In some embodiments, the processor may determine deviation information based on the actual puncture path and the target puncture path. For example, the processor may determine the angle between the tangent of the actual puncture path and the tangent of the target puncture path as the deviation angle, and determine the straight-line distance between the current endpoints of the actual puncture path and the target puncture path as the deviation distance of the puncture needle tip position.

[0232] The estimated deviation refers to the predicted value of the deviation information of the next puncture, for example, it may include the estimated deviation angle and the estimated deviation distance of the puncture needle tip.

[0233] In some embodiments, the processor can determine the estimated deviation of the next puncture in a variety of ways. For example, the processor can determine the deviation information based on the actual puncture path and the target puncture path. The processor can determine the estimated deviation of the next puncture based on the deviation information of the actual puncture path and the needle track deviation. Exemplarily, the processor can determine the actual needle tip position for the next puncture based on the deviation angle in the deviation direction of the needle track deviation plus a preset length at the end point of the actual puncture path. The processor can determine the estimated deviation angle and the estimated deviation distance of the puncture needle tip based on the actual needle tip position for the next puncture and the target needle tip position for the next puncture. The preset length can be preset manually. The target needle tip position for the next puncture can be determined based on the target puncture path.

[0234] In some embodiments, the processor may determine the early warning parameter based on a fourth preset relationship between the estimated deviation and the early warning parameter. For example, the fourth preset relationship may be that when the estimated deviation exceeds a deviation threshold, the estimated deviation is used as the early warning parameter.

[0235] In some embodiments, the processor can determine early warning parameters using a deviation prediction model based on deviation information of the actual puncture path, the target puncture path, and needle track deviation. In some embodiments, the deviation prediction model can be a machine learning model. For example, the deviation prediction model can be a neural network model, a deep neural network model, or any combination thereof.

[0236] In some embodiments, the deviation prediction model can be obtained through training using a third training sample and a third label. The third training sample can be the actual puncture path of the sample, the target puncture path of the sample, and the sample deviation information in the historical records. The third label is the actual early warning parameter of the historical operation corresponding to the third training sample. The third training sample may include a variety of data. For example, during a historical operation, after the early warning, if the corresponding processing is performed according to the early warning, and no abnormality occurs subsequently, then this type of data can be constructed into a training set of samples and corresponding labels, and the corresponding labels are the actual early warning parameters corresponding to the early warning. For another example, during a historical operation, if after the early warning, no corresponding processing is performed for the early warning, and an abnormality occurs subsequently, then this type of data can be constructed into a set of training samples and corresponding labels, and the corresponding labels are the actual early warning parameters corresponding to the early warning.

[0237] In some embodiments, the third training sample and third label can also be constructed based on whether the deviation of subsequent punctures in historical surgeries exceeds a preset threshold. For example, if the deviation of a subsequent puncture exceeds the preset threshold, the actual needle tip position of the puncture needle at the time the deviation exceeded the preset threshold is recorded, and the distance between the actual needle tip position and the target needle tip position at the time corresponding to the sample is used as the sample distance for early warning in the third label. The preset threshold is used to limit the deviation of the needle track. The preset threshold can be determined based on experience.

[0238] The specific training process of the deviation prediction model is similar to that of the exposure range prediction model. For more information, please refer to the corresponding description of Figure 3.

[0239] In some embodiments of this specification, by determining early warning parameters based on deviation information of the actual puncture path, the target puncture path, and the needle track deviation, it is possible to reasonably estimate the deviation that may occur in the next step based on actual conditions, and promptly remind the user to avoid unnecessary harm during the operation.

[0240] In some embodiments of this specification, by providing early warning of needle path deviation during the puncture process, excessive needle path deviation can be effectively avoided, the accuracy of the puncture needle's path during the operation can be improved, and the operation efficiency can be improved.

[0241] In some embodiments, the needle tip point and the predicted target point can be located on the same layer of the target image or on different layers of the target image. When the needle tip point and the predicted target point are marked on the same layer of the target image, the line connecting the needle tip point and the predicted target point is also displayed on the marked image.

[0242] In some embodiments, when the needle tip point and the predicted target point are located in the same layer of the target object, the needle tip point and the predicted target point will be marked on the same target image. At this time, the marked image displayed by the display module can be the target image marked with the needle tip point and the predicted target point.

[0243] Figure 8 is a schematic diagram illustrating an interface in which the needle tip and predicted target points are located on the same target image, according to some embodiments of this specification. As shown in Figure 8 , in this embodiment, the medical imaging device is a CT scanner, and in this embodiment, a total of four different scan images are displayed, with the needle tip 801 and predicted target point 802 located on the same scan image (i.e., the target image).

[0244] For example, when the needle tip and predicted target are annotated on the same target image, a line connecting the needle tip and predicted target is also displayed on the annotated image (the annotated target image). For example, in Figure 8, line 803 is also displayed between needle tip 801 and predicted target 802. Line 803 indicates the direction of the needle tract when the puncture needle is performing a puncture in the current puncture state, allowing the user to more clearly determine the location of the needle tract and, therefore, more accurately determine whether to perform a puncture based on the current puncture state.

[0245] It is understood that if the image shown in Figure 8 is the second image of the needle insertion state, then the needle tip point 801 and the predicted target point 802 are located on the same second image, and therefore the needle insertion process is a same-layer needle insertion. In this case, the needle tip point and the predicted target point are both on the -729.125mm layer. The two marked points are connected by a connecting line 803, where connecting line 803 can be a dotted line to indicate the subsequent needle insertion path of the needle tip, facilitating the physician's assessment of the tissue information that will be passed through during the subsequent puncture process. As the needle insertion depth increases, the needle tip point 801 will continue to approach the predicted target point 802 along the dotted line until the puncture is complete.

[0246] In some embodiments, when the needle tip point and the predicted target point are located in two slices of different layers of the target object, the needle tip point and the predicted target point will be marked on two different target images.

[0247] Figure 9 is a schematic diagram illustrating an interface in which the needle tip and predicted target are located in different target images, according to some embodiments of this specification. As shown in Figure 9 , in this embodiment, the medical imaging device is a CT scanner, and a total of four different scan images are displayed, with the needle tip 901 and predicted target 902 located in two different target images.

[0248] It is understandable that if the image shown in Figure 9 is the second image of the needle insertion state, then the needle tip point 901 and the predicted target point 902 are located on different second images, so the needle insertion process is a cross-layer needle insertion. At this time, the real-time needle tip point 901 is at the -729.125mm layer and the predicted target point 902 is at the -727.875mm layer. As the needle insertion depth increases, the real-time needle tip point 901 will puncture from the initial layer (i.e., the -729.125mm layer) to the target layer (i.e., the -727.875mm layer) until the puncture is completed.

[0249] It is understandable that in FIG8 and FIG9 , in order to facilitate the user to distinguish, marks with different colors and shapes can be used to mark the needle tip point and the predicted target point.

[0250] In some embodiments, a mark of the same shape but different color may be used to mark the needle tip point and the predicted target point, or a mark of the same color but different shape may be used to mark the needle tip point and the predicted target point. This specification does not limit the specific form of the mark.

[0251] In some embodiments, when the needle tip point and the predicted target point are marked on target images of different layers, the layer number of the target point coordinates can also be displayed on the marked image, and the target point is marked on the target image of the layer corresponding to the target point.

[0252] The target point is the spatial point that the needle tip is ultimately intended to reach after the puncture needle is inserted. The target point coordinates are the coordinates of the spatial point that the needle tip is ultimately intended to reach. The layer number of the target point coordinates is the layer number of the target image corresponding to the target point.

[0253] In some embodiments, when the deviation information of the needle track deviation is greater than a preset threshold, the processor can display the predicted needle track trajectory on the annotated image in advance. For more information about the preset threshold, please refer to the above description.

[0254] In some embodiments, the processor may determine a predicted needle tract trajectory using a deviation prediction model. In some embodiments, the processor may include the actual needle tract trajectory in a third label corresponding to a third training sample in the historical record when training the deviation prediction model. The actual needle tract trajectory may be constructed based on the needle tract trajectory of the puncture needle measured during the historical surgery corresponding to the third training sample within a time period after the current historical time point.

[0255] In some embodiments, the preset threshold may include a deviation angle threshold, a deviation distance threshold of the needle tip position of the puncture needle, etc. In some embodiments, the preset threshold may be manually set based on experience.

[0256] In some embodiments of the present specification, by displaying the predicted needle track trajectory in advance when the deviation information of the needle track deviation is greater than a preset threshold, the doctor can be provided with a more intuitive reference by annotating the image, thereby avoiding excessive deviation of the needle track deviation that affects the doctor's judgment.

[0257] In some embodiments, the processor may determine the preset threshold based on the puncture depth, the puncture speed, and the importance of the tissue surrounding the puncture location.

[0258] For details on the puncture depth and puncture speed, please refer to the description of Figure 2. For details on the importance of the tissue surrounding the puncture site, please refer to the description above.

[0259] In some embodiments, the processor may determine the preset threshold based on a fifth preset relationship between the puncture depth, puncture speed, and importance of the tissue surrounding the puncture location, and the preset threshold. The fifth preset relationship may be preset based on actual conditions and historical experience. For example, the fifth preset relationship may be such that the deeper the puncture depth, the faster the puncture speed, and the more important the tissue surrounding the puncture location, the smaller the preset threshold (deviation angle threshold, deviation distance threshold).

[0260] In some embodiments of the present specification, by determining the preset threshold based on the puncture depth, the puncture speed, and the importance of the tissue surrounding the puncture location, the accuracy of the determined preset threshold can be further improved.

[0261] In some embodiments, the early warning parameter may further include a preset threshold.

[0262] In some embodiments, when the early warning parameter includes a preset threshold, the processor can determine the early warning parameter including the preset threshold using a deviation prediction model. In some embodiments, when the early warning parameter includes a preset threshold, the processor can, when training the deviation prediction model, incorporate the sample puncture depth, sample puncture speed, and the importance of tissue surrounding the sample puncture location corresponding to a third training sample in historical records into the third training sample for training. The third label corresponding to the third training sample includes an actual preset threshold. The actual preset threshold can be constructed based on deviation data actually measured during the historical surgery corresponding to the third training sample.

[0263] In some embodiments of this specification, by including a preset threshold in the early warning parameter, the determination of the preset threshold can be made more flexible and reasonable, so that the early warning parameter can more accurately reflect the actual situation and provide more reliable prompt information.

[0264] In some embodiments of the present specification, by displaying the actual needle track trajectory on the annotated image in advance when the deviation information of the needle track deviation is greater than a preset threshold, it helps the user to judge the current situation when the needle track deviates and improves the accuracy of the puncture process.

[0265] In some embodiments, while displaying the annotated image on the display module, the processor also displays adjacent scan images of the annotated image on the display module. By displaying the adjacent scan images, the movement of the needle tip and the predicted target point in different slices can be observed during puncture. Furthermore, when the needle tip and the predicted target point move onto the adjacent scan images, they can be quickly displayed.

[0266] It is understood that the adjacent scanned images of an annotated image can be the scanned image corresponding to the slice preceding the slice corresponding to the annotated image, or the scanned image corresponding to the slice following the slice corresponding to the annotated image. Furthermore, the scanned images corresponding to the slice preceding the slice can include one or more, and the scanned images corresponding to the slice following the slice can also include one or more, and this specification does not impose any limitation on this.

[0267] For example, in the embodiment shown in FIG8 , in addition to the annotated image marked with the needle tip point 801 and the predicted target point 802 , FIG8 also shows three scanned images adjacent to the annotated image.

[0268] For another example, in the embodiment shown in FIG9 , in addition to two different annotated images marked with the needle tip point 901 and the predicted target point 902 , FIG9 also shows two scanned images adjacent to the annotated images, and each of the two scanned images is adjacent to one annotated image.

[0269] It should be understood that when the user determines, based on the image displayed by the display module, that the predicted target point and the target lesion do not overlap, or that the predicted target point is outside the outline of the target lesion, it is necessary to adjust the posture of the puncture needle. For ease of understanding, the posture adjustment process in the puncture method of the embodiment of this specification is exemplarily described below with reference to the accompanying drawings.

[0270] In some embodiments, the puncture assistance system provides two posture adjustment methods: automatic posture adjustment and manual posture adjustment. The user can choose according to the clinical situation. The following is an exemplary description of automatic posture adjustment and manual posture adjustment.

[0271] When the user selects automatic posture adjustment, the puncture assisting device provides adjustment data for posture adjustment.

[0272] In some embodiments, in response to receiving a first posture adjustment instruction, the processor may determine lesion location information of a target lesion in the target object based on the multiple scanned images. First posture adjustment data is obtained based on the lesion location information and the puncture status information. The first posture adjustment data represents the posture change data of the puncture needle when the predicted target point is adjusted to the lesion location corresponding to the lesion location information. A first movement instruction corresponding to the first posture adjustment data is sent to the surgical robot. The first movement instruction is used to control the surgical robot to move the puncture needle to adjust the posture.

[0273] The first posture adjustment instruction indicates automatic posture adjustment, i.e., the user selects the automatic posture adjustment function. In this embodiment, the processor obtains the lesion location of the target lesion by performing image recognition on multiple scanned images. The lesion location of the target lesion is the lesion location information, and the first posture adjustment data is obtained based on the lesion location information and the puncture status information. This method can achieve automatic posture adjustment, saving the user's posture adjustment time.

[0274] In some embodiments, before sending the first movement instruction corresponding to the first posture adjustment data to the surgical robot, the processor may be further configured to display the first posture adjustment data. In response to receiving a user confirmation to adopt the first posture adjustment data, the processor sends the first movement instruction corresponding to the first posture adjustment data to the surgical robot. This means that in this embodiment, the automatic posture adjustment process requires user confirmation before specific posture adjustment can be performed, adding a step of doctor review and confirmation, thereby improving the reliability of the puncture method and the accuracy of the puncture results.

[0275] In some embodiments, when performing automatic posture adjustment, the processor may also determine whether it is necessary to withdraw the needle before adjusting the posture. In some embodiments, before sending the first movement instruction corresponding to the first posture adjustment data to the surgical robot, the processor is further configured to determine whether at least one of the following needle withdrawal conditions exists. Needle withdrawal conditions include: the puncture depth of the puncture needle is greater than a first threshold, or the distance between the predicted target point and the lesion location is greater than a second threshold, or the posture adjustment based on the first posture adjustment data will cause the puncture needle to interfere with the target tissue or organ of the target object, etc.

[0276] In some embodiments, in response to at least one of the needle withdrawal conditions, the processor sends a second movement instruction to the surgical robot, controlling the surgical robot to move the puncture needle for needle withdrawal. The second movement instruction is used to control the surgical robot to move the puncture needle for needle withdrawal. After needle withdrawal is completed, the processor sends a first movement instruction corresponding to the first posture adjustment data to the surgical robot, controlling the surgical robot to move the puncture needle for posture adjustment.

[0277] It is understandable that in the case of automatic posture adjustment, if the puncture needle's puncture depth is greater than the first threshold, it indicates that the puncture needle has punctured to a deeper depth, and direct posture adjustment may cause greater damage to the target object; if the distance between the predicted target point and the first position is greater than the second threshold, it indicates that the predicted target point deviates at a large angle. In this case, if the puncture needle needs to be adjusted more significantly when the posture is adjusted directly, it will cause greater damage to the target object; interference between the puncture needle and the target tissue and organ of the target object means that the puncture needle may touch the target tissue and organ of the target object, and may then damage the touched target tissue and organ. Therefore, under these three conditions, controlling the surgical robot to retract the needle first and then adjust the posture can substantially reduce damage to the target object and improve the safety of automatic posture adjustment. Among them, the target tissue and organ refers to the important tissue and organ of the target object, for example, the target tissue and organ can be an important tissue and organ such as blood vessels, bones, nerves, etc.

[0278] In some embodiments, when performing automatic posture adjustment, before sending the first movement instruction corresponding to the first posture adjustment data to the surgical robot, the processor is used to perform the following steps: if the puncture assist device determines that the puncture depth of the puncture needle is less than or equal to the first threshold, and predicts that the distance between the target point and the first position is less than or equal to the second threshold, and direct posture adjustment based on the first posture adjustment data will not cause interference between the puncture needle and the target tissue organ of the target object, the processor sends the first movement instruction to the surgical robot, and the first movement instruction is used to control the surgical robot to move the puncture needle to adjust the posture.

[0279] It is understandable that in the case of automatic posture adjustment, if the puncture needle's puncture depth is less than or equal to the first threshold, it indicates that the puncture needle's puncture depth is shallow, and direct posture adjustment will cause less damage to the target object; if the distance between the predicted target point and the first position is less than or equal to the second threshold, it indicates that the predicted target point's deviation angle is small. In this case, direct posture adjustment of the puncture needle requires a smaller adjustment range, and the damage to the target object is also less; the puncture needle does not interfere with the target tissue or organ of the target object, which means that the puncture needle will not touch the target tissue or organ of the target object. Therefore, when all three conditions are met, the surgical robot can be controlled to directly adjust the posture without retracting the needle.

[0280] For example, after the puncture assist device determines that direct posture adjustment is possible without needle withdrawal, it displays the first posture adjustment data on the display module before sending the first movement instruction corresponding to the first posture adjustment data to the surgical robot. If an instruction to use the first posture adjustment data is received, the first movement instruction corresponding to the first posture adjustment data is sent to the surgical robot. In other words, in this embodiment, the automatic posture adjustment process requires user confirmation before specific posture adjustment can be performed, which adds a step of doctor review and confirmation, thereby improving the reliability of the puncture method and the accuracy of the puncture results.

[0281] In some embodiments, the processor may determine the first threshold based on the puncture range, lesion characteristics, and target coordinates.

[0282] For details about the puncture range and lesion characteristics, please refer to the relevant description in Figure 2.

[0283] In some embodiments, the processor may determine the first threshold based on a sixth preset relationship among the puncture range, lesion characteristics, target coordinates, and the first threshold. For example, the sixth preset relationship may be that the larger the puncture range, the farther the lesion is from the body surface, and the deeper the predicted target coordinates are, the larger the first threshold.

[0284] In some embodiments, the processor may perform matching in a fourth vector database based on the puncture range, lesion characteristics, and predicted target coordinates to determine the first threshold. For example, the processor may construct a fourth feature vector based on the puncture range, lesion characteristics, and predicted target coordinates, match it in the fourth vector database, and determine the first threshold corresponding to the fourth reference vector with the highest similarity as the corresponding first threshold.

[0285] Among them, the fourth vector database includes a fourth reference vector and its corresponding first threshold. The fourth reference vector is composed of the actual historical puncture range, historical lesion characteristics, and historical predicted target coordinates in the historical records. The corresponding first threshold can be determined in a similar way to determining the fourth label later, and the similarity can be calculated based on cosine distance, Euclidean distance, etc.

[0286] In some embodiments, the processor may determine the first threshold value based on the puncture range, lesion characteristics, and predicted target coordinates using a first threshold prediction model. In some embodiments, the first threshold prediction model may be a machine learning model. For example, the first threshold prediction model may be a neural network model, a deep neural network model, or any combination thereof.

[0287] In some embodiments, the first threshold prediction model can be obtained through training using a fourth training sample and a fourth label. The fourth training sample can be the sample puncture range, sample lesion characteristics, and sample predicted target coordinates in the historical records, and the fourth label can be the actual first threshold corresponding to the historical surgery with the best needle withdrawal compensation effect (for example, timely needle withdrawal and high surgical completion rate), which can be constructed based on the fourth training sample. In some embodiments, the processor can construct a fourth label based on the puncture needle puncture depth and the corresponding puncture abnormality frequency in the historical surgery corresponding to the fourth training sample.

[0288] The training method of the first threshold prediction model is similar to that of the exposure range prediction model, and the relevant description can be found in FIG3 .

[0289] In some embodiments of the present specification, by determining the first threshold based on the puncture range, lesion characteristics, and predicted target coordinates, the first threshold can be set more specifically and more in line with actual conditions, allowing a more reasonable judgment on whether to perform a needle withdrawal operation.

[0290] In some embodiments, the processor may determine the second threshold based on a seventh preset relationship among the puncture speed, the position information of the first needle tip, the lesion characteristics, and the second threshold. For example, the seventh preset relationship may be such that the faster the puncture speed, the closer the position of the first needle tip is to the lesion, and the smaller the size of the lesion in the lesion characteristics, the smaller the second threshold may be set to avoid the puncture needle reaching an incorrect target point (e.g., a non-lesion site).

[0291] In some embodiments, the processor may determine the second threshold by matching the fifth vector database based on the puncture speed, the position information of the first needle tip point, and the lesion characteristics. For example, the processor may construct a sixth feature vector based on the puncture speed, the position information of the first needle tip point, and the lesion characteristics, match it in the fifth vector database, and determine the second threshold corresponding to the fifth reference vector with the highest similarity as the corresponding second threshold.

[0292] Among them, the fifth vector database includes the fifth reference vector and its corresponding second threshold. The fifth reference vector is composed of the actual historical puncture speed in the historical records, the position information of the historical first needle tip point, and the historical lesion characteristics. The corresponding second threshold can be determined in a similar way to determining the fifth label later, and the similarity can be calculated based on cosine distance, Euclidean distance, etc.

[0293] In some embodiments, the processor may determine the second threshold value using a second threshold prediction model based on the puncture speed, the position information of the first needle tip, and the lesion characteristics. In some embodiments, the second threshold prediction model may be a machine learning model. For example, the second threshold prediction model may be a neural network model, a deep neural network model, or any combination thereof.

[0294] In some embodiments, the second threshold prediction model can be obtained through training using a fifth training sample and a fifth label. The fifth training sample can be a sample puncture speed, the position information of the sample first needle tip point, and the sample lesion characteristics in the historical records. The fifth label is the actual second threshold corresponding to the historical surgery with the best needle withdrawal compensation effect (for example, the best subsequent posture adjustment effect), which can be constructed based on the fifth training sample. In some embodiments, the processor can construct the fifth label based on the predicted distance between the target point and the lesion location and the corresponding puncture abnormality frequency in the historical surgery corresponding to the fifth training sample.

[0295] The training method of the second threshold prediction model is similar to that of the exposure range prediction model, and the relevant description can be found in Figure 3.

[0296] In some embodiments of this specification, the second threshold is determined by the puncture speed, the position information of the first needle tip point and the lesion characteristics, so that the second threshold can be set more targeted and more in line with the actual situation, so that it can be more reasonable to judge whether to withdraw the needle.

[0297] In some embodiments, during manual posture adjustment, the user controls the surgical robot to adjust the posture of the puncture needle. Exemplarily, the processor may also: in response to receiving the second posture adjustment instruction, the processor may send a third movement instruction to the surgical robot to control the surgical robot to adjust the posture of the puncture needle. The second posture adjustment instruction may be generated based on the user's posture adjustment movement on the main hand end of the surgical robot. In this embodiment, the provided posture adjustment process can meet the user's demand for high user autonomy and facilitate the user's manual posture adjustment.

[0298] It is understood that in the specific posture adjustment process, posture adjustment can include: same-layer posture adjustment, cross-layer posture adjustment, and free mode posture adjustment. The following is an exemplary description of same-layer posture adjustment, cross-layer posture adjustment, and free mode posture adjustment with reference to the accompanying figures. It is understood that the medical imaging device in the following examples is a CT scanner.

[0299] FIG10 is a schematic diagram of an interface for adjusting posture on the same layer according to some embodiments of this specification.

[0300] (1) Adjust posture on the same level.

[0301] Same-layer posture adjustment means that the predicted target before and after posture adjustment are located in the same transverse plane (i.e., on the same tomographic image). In this case, only the same-layer angle (i.e., left-right deflection) needs to be changed, and the cross-layer angle (i.e., head-foot deflection) does not need to be changed. As shown in Figure 10, Figure 10a is the tomographic image before posture adjustment, and Figure 10b is the tomographic image after posture adjustment. As shown in Figure 10a, the predicted target 1001 and the target lesion 1002 before posture adjustment are located on the same tomographic image, but do not overlap. Therefore, same-layer posture adjustment is required. After the same-layer angle is changed, the position of the predicted target on the Dicom image in the same transverse plane (Z-axis coordinate -729.125mm) changes accordingly. Figure 10b is the tomographic image after posture adjustment. As shown in Figure 10b, the predicted target 1003 and the target lesion 1002 overlap after posture adjustment, completing the posture adjustment.

[0302] In some embodiments, a same-layer gait adjustment mode can be configured. If the predicted target and target lesion are located on the same slice, the user can select this mode during automatic or manual gait adjustment. In this mode, only the same-layer angle can be changed during gait adjustment, not cross-slice angles. By limiting gait adjustment to only the same-layer angle, this mode can prevent the possibility of accidentally crossing slices during gait adjustment.

[0303] FIG11 is a schematic diagram of an interface for cross-layer posture adjustment according to some embodiments of this specification.

[0304] (2) Cross-layer posture adjustment.

[0305] Cross-layer posture adjustment refers to the phenomenon that the predicted target before posture adjustment and the predicted target after posture adjustment have a cross-layer phenomenon in the transverse position (i.e., they are located on different tomographic images). The angle on the same layer (left-right deflection angle) may remain unchanged or change, and the cross-layer angle (head-foot deflection angle) changes. As shown in Figure 11, Figure 11a is the tomographic image before posture adjustment, and Figure 11b is the tomographic image after posture adjustment. As shown in Figure 11a, the tomographic image where the predicted target 1101 is located before posture adjustment does not have the target lesion, so the predicted target needs to be adjusted to a tomographic image that shows the target lesion. In this case, cross-layer posture adjustment is required. After the cross-layer angle is changed, the Z-axis coordinate of the predicted target 1102 is updated from -729.125mm to -727.875mm, and the predicted target 1102 after posture adjustment coincides with the target lesion 1103, completing the posture adjustment.

[0306] In some embodiments, a cross-layer positioning mode can be configured. If the predicted target and the target lesion are located in different slices, the user can select cross-layer positioning mode during automatic or manual positioning. In this mode, only cross-layer angles can be changed during positioning, not within the same slice. This allows the predicted target to be quickly aligned with the slice where the target lesion is located, improving positioning efficiency.

[0307] It is understandable that after adjusting the predicted target to the fault where the target lesion is located, if the predicted target and the target lesion still cannot coincide, the same-layer posture adjustment mode can be selected at this time, and the final posture adjustment can be completed in the same-layer posture adjustment mode.

[0308] (3) Free posture adjustment.

[0309] Free posture adjustment means that the surgeon can control the surgical robot's main hand to adjust the posture both within and across layers. After the needle track posture changes, the predicted target position can change both within and across layers. This will not be elaborated here.

[0310] In some embodiments, before sending the third movement instruction to the surgical robot, the processor can be further configured to: in response to receiving the needle withdrawal action of the user at the main hand end of the surgical robot, send a fourth movement instruction to the surgical robot to control the surgical robot to move the puncture needle for needle withdrawal.

[0311] The fourth movement instruction refers to an instruction for controlling the surgical robot to withdraw the needle. In some embodiments, the processor can generate the fourth movement instruction based on the user's needle withdrawal action.

[0312] After introducing the posture adjustment process, the triggering process of the medical imaging device in the embodiment of this specification is described below.

[0313] In some embodiments, the processor is further configured to send a scanning instruction to the processor in response to receiving a signal indicating that the surgical robot is performing a puncture operation, so that the processor controls the medical imaging device to perform a scan.

[0314] In some embodiments, the puncture assist device automatically controls the medical imaging device to perform a scanning operation.

[0315] For example, when the puncture assist device automatically controls the medical imaging device to perform a scanning operation, upon receiving a signal from the surgical robot to perform a puncture operation, the processor sends a scanning instruction to the processor to control the medical imaging device to perform the scanning operation. The puncture operation includes a posture adjustment operation and a needle insertion operation. The posture adjustment operation is used to adjust the posture of the puncture needle. The needle insertion operation is used to adjust the puncture depth of the puncture needle. In this embodiment, the puncture assist device can automatically control the medical imaging device to perform a scanning operation, which can improve the synchronization between the medical imaging device and the puncture operation and enhance the real-time performance of the scan.

[0316] In some embodiments, the puncture operation may also include a needle withdrawal operation. For more details, please refer to the relevant description of Figure 26.

[0317] In some embodiments, after receiving the first puncture operation instruction, the main hand of the surgical robot prompts the puncture auxiliary device to control the medical imaging device to perform a scanning operation, so that the medical imaging device can be automatically controlled to perform real-time exposure scanning during the puncture operation.

[0318] In some embodiments, medical imaging scanning is performed using the time interval between when the surgical robot's main hand receives the puncture operation instruction and when the surgical robot's slave hand executes the puncture operation instruction. Medical imaging scanning can be performed before the puncture operation is performed, thereby avoiding blind puncture caused by the puncture operation having been performed but the medical imaging scan having not been performed.

[0319] In some embodiments, a processor is used to send a sixth movement instruction corresponding to the second puncture operation instruction to the surgical robot's slave hand, and a medical image scan is performed during the time interval between the surgical robot's slave hand executing the sixth movement instruction. The medical image scan can be performed before the puncture operation is performed, thereby avoiding blind puncture caused by the puncture operation having been performed but the medical image scan having not been performed.

[0320] In some embodiments, the processor can determine whether the puncture action is completed by detecting the movement state of the surgical robot's hands. For example, it can be determined based on the information reported by the robot components, and when the completion of the puncture action is detected, the medical imaging device is controlled to stop exposure.

[0321] In some embodiments, the processor can determine whether the puncture action is completed based on whether the real-time needle tip position reported by the surgical robot has changed.

[0322] For more descriptions about the first puncture operation instruction, the second puncture operation instruction, the sixth movement instruction and the like related to automatic scanning, please refer to the description in the embodiment of the puncture assisting system in FIG1 , which will not be repeated here.

[0323] In some embodiments, the processor may determine the frequency of acquiring the first image based on the lesion characteristics and the patient characteristics. For relevant descriptions of the lesion characteristics, please refer to the corresponding description of FIG2 .

[0324] Patient characteristics refer to relevant characteristics reflecting the patient's vital signs. For example, patient characteristics may include the patient's heart rate, the patient's normal respiratory rate, etc.

[0325] In some embodiments, the processor may determine the frequency of acquiring the first image based on an eighth preset relationship among lesion characteristics, patient characteristics, and the frequency of acquiring the first scan image. The eighth preset relationship may be preset based on actual conditions and historical experience. For example, the eighth preset relationship may include a correlation between lesion characteristics and the frequency of acquiring the first scan image. For example, based on the lesion characteristics, when the organ where the lesion is located is greatly affected by respiration, it indicates that the degree of deformation of the lesion is greater. In this case, the processor may increase the frequency of acquiring the first image to more accurately reflect the impact of respiration on organ deformation; otherwise, the processor may reduce the frequency of acquiring the first image.

[0326] For another example, the eighth preset relationship may include a correlation between patient characteristics and the frequency of acquiring the first scan image. For example, based on the patient characteristics, when the patient's heart rate and respiratory rate are high, it indicates that the frequency of organ deformation is fast. In this case, the processor may increase the frequency of acquiring the first image to more accurately reflect the impact of respiration on organ deformation. Conversely, the processor may decrease the frequency of acquiring the first image.

[0327] In some embodiments, the processor may further perform matching in a sixth vector database based on lesion characteristics and patient characteristics to determine the frequency of acquiring the first image. For example, the processor may construct a sixth vector database based on actual historical lesion characteristics, historical patient characteristics, and the corresponding frequency of acquiring the first image in historical records, construct a sixth feature vector based on current lesion characteristics and patient characteristics, match it in the sixth vector database, and determine the frequency of acquiring the first image corresponding to the sixth reference vector with the highest similarity as the current frequency of acquiring the first image.

[0328] Among them, the sixth vector database includes the sixth reference vector and its corresponding frequency of obtaining the first image. The sixth reference vector is composed of the actual historical lesion characteristics and historical patient characteristics in the historical records. The corresponding frequency of obtaining the first image can be determined based on the actual frequency of obtaining the first image in the historical operation, and the similarity can be calculated based on cosine distance, Euclidean distance, etc.

[0329] In some embodiments, the processor may further determine the frequency of acquiring the first image based on the lesion characteristics and the patient characteristics using a frequency prediction model. In some embodiments, the frequency prediction model may be a machine learning model. For example, the frequency prediction model may be a neural network model (NN), a deep neural network model (DNN), or any combination thereof.

[0330] In some embodiments, the frequency prediction model can be obtained through training using a sixth training sample and a sixth label. The sixth training sample can be a sample patient feature and a sample lesion feature in a historical record, and the sixth label can be constructed based on the sixth training sample. For example, the processor can construct the sixth label based on the accuracy of the target-guided image in the historical surgery corresponding to the sixth training sample, such as using the frequency of obtaining the first image corresponding to the historical surgery with the highest accuracy of the target-guided image as the sixth label. For another example, the processor can construct the sixth label based on the degree of overlap between the second image and the first image in the historical surgery corresponding to the sixth training sample, such as using the frequency of obtaining the first image in the historical surgery where the second image is completely covered in the first image as the sixth label.

[0331] The specific training process of the frequency prediction model is similar to that of the exposure range prediction model. For more information, please refer to the corresponding description of Figure 3.

[0332] In some embodiments of the present specification, by determining the frequency of acquiring the first image based on lesion characteristics and patient characteristics, a more reasonable frequency of acquiring the first image can be determined, so that the acquired first image better reflects the actual situation of the target part in the first time period, which helps to subsequently determine a more accurate target image.

[0333] FIG12 is an exemplary flow chart of determining a target image according to some embodiments of this specification. In some embodiments, process 1200 is executed by a processor. As shown in FIG12 , process 1200 may include the following steps.

[0334] Step 1210 : Determine deformation parameters based on the pre-operative image and the first image.

[0335] For the description of the preoperative image and the first image, please refer to the corresponding description of FIG1 .

[0336] Deformation parameters refer to parameters related to the registration of the target part in the preoperative image and the target part in the first image. For example, deformation parameters may include elastic registration parameters and rigid registration parameters, among others. The registration of the target part in the preoperative image and the target part in the first image may include elastic registration, rigid registration, and the like. Elastic registration refers to registration involving image deformation. Rigid registration refers to registration involving translation and rotation of coordinate axes. By way of example only, rigid registration may be a process in which the target part in the first image is set as a reference image, and the target part in the preoperative image is spatially transformed (e.g., through a series of rotations, translations, and other transformations) based on the reference image, so that the target part in the preoperative image is spatially aligned with the corresponding pixel points in the reference image. Accordingly, the rigid registration parameters may reflect the spatial transformation relationship between the two images (e.g., rotation angle, translation distance, etc.). In some embodiments, the rigid registration parameters may include parameters such as horizontal displacement, vertical translation, and rotation angle during the rigid registration process. In some embodiments, the rigid registration parameters may be represented as numerical values, vectors, matrices, and the like. Elastic registration can be the process of spatially transforming the target part with irregular internal deformation in the preoperative image according to the target part in the first image. Accordingly, the elastic registration parameters can refer to the transformation parameters corresponding to the part of the image that has deformed when the medical image is registered (for example, the geometric shape change of the organ tissue or target area for radiotherapy). The autonomous movement of radiotherapy or the spontaneous existence of organs can cause changes in the size, shape, volume, etc. of internal tissues and organs, resulting in some irregular internal deformations in the collected medical images. For example, with changes in breathing, the shape and size of the target part of the target object will gradually change.

[0337] In some embodiments, when the medical imaging device acquires the preoperative image, the processor can also collect the respiratory amplitude point corresponding to the target object at that moment through the respiratory gating device and determine it as the target respiratory amplitude point.

[0338] The respiratory amplitude point is a parameter used to reflect the degree of chest fluctuation when the target subject breathes. The target respiratory amplitude point is the respiratory amplitude point of the target subject when the preoperative image is collected.

[0339] FIG. 13 is an exemplary schematic diagram of target respiratory amplitude points according to some embodiments of the present specification.

[0340] As shown in Figure 13, the horizontal axis is time, and the vertical axis is the respiratory amplitude point of the target object collected by the processor. When the preoperative image of the target object is collected by the medical imaging device at time t0, the respiratory amplitude point A0 corresponding to the target object is the target respiratory amplitude point.

[0341] In order to enable real-time intervention guidance during subsequent surgical execution, within the first time period, when the medical imaging device acquires the first image, the processor can also collect the respiratory amplitude points corresponding to the target object within the first time period through the respiratory gating device, and each first image corresponds to a respiratory amplitude point.

[0342] In some embodiments, the processor captures a first image using a medical imaging device at a predetermined time in the first time period. For example, the processor captures a first image at a user-specified time within the first time period. Capturing the first image at the predetermined time can reduce radiation dose to the target subject due to scanning.

[0343] In some embodiments, the processor may capture multiple first images at predetermined respiratory amplitude points using a medical imaging device based on the target subject's respiratory amplitude points. For example, the processor may capture a first image at intervals corresponding to a predetermined respiratory amplitude point change until a complete respiratory interval of the target subject is captured.

[0344] Some embodiments of the present specification can display the breathing changes of the target object at different breathing amplitude points within the complete breathing interval through multiple first images collected in the aforementioned manner, so that the deformation parameters determined later can more completely reflect the deformation of the target part of the target object within the breathing interval.

[0345] In some embodiments, when the processor acquires at least one first image, the respiratory pressure value and / or respiratory amplitude of the target object needs to meet a preset condition. The preset condition refers to a pre-set condition that triggers the processor to acquire the first image. For example, the preset condition may include that the respiratory pressure value is greater than the first pressure value and less than the second pressure value. The preset condition may also include other conditions. For example, the preset condition may also include that the respiratory amplitude is greater than the first amplitude and less than the second amplitude. The first pressure value, the second pressure value, the first amplitude and the second amplitude may be default values, preset values, etc. As an example only, the preset condition may include that the respiratory pressure value is greater than 1.375kPa and less than 1.782kPa.

[0346] In some embodiments, when the respiratory pressure value and / or respiratory amplitude of the target object meets a preset condition and the processor captures at least one first image, the respiratory amplitude point of the target object should be within a target respiratory amplitude interval. The target respiratory amplitude interval refers to an interval containing the target respiratory amplitude point. The upper and lower bounds of the interval can be determined based on the maximum and minimum values ​​of the respiratory amplitude points of the target object during the first time period when the processor captures the first image. For example, the processor can determine the maximum value of the respiratory amplitude point of the target object during the first time period when the processor captures the first image as the upper bound of the target respiratory amplitude interval.

[0347] FIG. 14 is an exemplary diagram of target respiratory amplitude intervals according to some embodiments of the present specification.

[0348] By way of example only, when the target subject's respiratory pressure and / or respiratory amplitude meet preset conditions and at least one first image is captured within a first time period t1 to t2, the target subject's respiratory amplitude points captured by the processor may be as shown in FIG14 . When capturing preoperative images, the target subject's target respiratory amplitude point is A0. Within the first time period t1 to t2, the target subject's respiratory amplitude points have a maximum value of A1 and a minimum value of A2, resulting in a target respiratory amplitude interval of [A2, A1]. Target respiratory amplitude point A0 is within the target respiratory amplitude interval [A2, A1].

[0349] In some embodiments of the present specification, the target respiratory amplitude point is included in the target respiratory amplitude range, so that the preoperative image and the first image can be aligned. In the subsequent interventional surgery, the intervention guidance can be completed as long as the target object's respiratory amplitude point is within the target respiratory amplitude range, avoiding multiple scans of the target object in order to find a specific respiratory amplitude point during the surgery, which causes it to receive excessive radiation dose.

[0350] In some embodiments, the target respiratory amplitude interval includes at least one respiratory amplitude interval segment. Each respiratory amplitude interval segment includes a corresponding respiratory amplitude point, and within the respiratory amplitude interval segment, the deformation of the target part of the target object is less than a preset deformation threshold. The at least one respiratory amplitude interval segment may fully or partially cover the target respiratory amplitude interval. The processor may determine the respiratory amplitude interval segment in which the respiratory amplitude point is located based on the respiratory amplitude point corresponding to a first image, thereby determining the respiratory amplitude interval segment corresponding to the first image.

[0351] During the second time period, when the processor acquires multiple second images, it can prompt the target object that the respiratory amplitude point should be within the target respiratory amplitude range. The prompting can be in various ways, such as voice prompting.

[0352] Figure 16 is an exemplary diagram illustrating ideal respiratory amplitude points for a target subject during a surgical procedure according to some embodiments of this specification. As an example only, during an interventional procedure performed during the second time period t3-t4, as shown in Figure 16 , the target subject's respiratory amplitude point should be within the target respiratory amplitude range [A2, A1] and as close as possible to the target respiratory amplitude point A0.

[0353] Some embodiments of the present specification can reduce the deformation difference of tissues and organs caused by respiratory movement between the preoperative image and the second image by making the target object's respiratory amplitude point as close as possible to the target respiratory amplitude point within the second time period, which is beneficial to improving the subsequent registration accuracy of the preoperative image and the second image.

[0354] In some embodiments, during the second time period, the processor can use a respiratory gating device to capture respiratory amplitude points corresponding to the target object in real time, and based on the respiratory amplitude parameters corresponding to each first image, capture a second image at the same respiratory amplitude parameters. The respiratory amplitude parameters are parameters related to the target object's respiratory information. In some embodiments, the respiratory amplitude parameters are respiratory amplitude points or respiratory amplitude intervals to which they belong. The deformation of each respiratory amplitude point within the aforementioned respiratory amplitude interval at the target portion of the target object is less than a preset deformation threshold. For example, when there is only one first image, the processor can capture the target object based on the respiratory amplitude parameters corresponding to the first image to obtain at least one second image, ensuring that deformation parameters corresponding to the second image exist. For more information on deformation parameters, please refer to the relevant content below in this specification. It is understood that the number of first images does not need to correspond to the number of second images. For example, there may be only one first image, but the processor can use medical imaging equipment to capture multiple second images within the respiratory amplitude points or respiratory amplitude intervals corresponding to the first image.

[0355] In some embodiments, the processor may acquire a plurality of first images under different respiratory amplitude parameters, and register the plurality of first images with the preoperative image to acquire a plurality of deformation parameters.

[0356] It can be understood that when there are multiple first images and the respiratory amplitude parameters (such as respiratory amplitude interval segments) corresponding to the aforementioned multiple first images are sufficient to cover the complete respiratory interval of the target object, the processor can capture the second image at any time point within the second time period, and the aforementioned second images captured at any time point have corresponding deformation parameters.

[0357] The processor may process the pre-operative image and the at least one first image based on the registration software to determine at least one deformation parameter.

[0358] In some embodiments, the processor can process the preoperative image and the at least one first image based on the registration deep learning model to determine at least one deformation parameter.

[0359] The registration deep learning model may be a machine learning model for determining deformation parameters. For example, the registration deep learning model may be a deep neural network (DNN) model.

[0360] It should be understood that in order to enable real-time scanning, image reconstruction, and image transmission of the target area, the registration deep learning model can be a lightweight model (e.g., a simplified network structure). The processor can also perform quantization, pruning, knowledge distillation, and other processing on the registration deep learning model to ensure that the registration deep learning model can achieve real-time processing of the preoperative image and the at least one first image. To further improve the computing speed and processing performance of the registration deep learning model, high-performance graphics processing unit (GPU) hardware can be used when deploying the registration deep learning model.

[0361] In some embodiments, the registration deep learning model may input a preoperative image and a first image; its output may include deformation parameters corresponding to respiratory amplitude points in the first image. If there are multiple first images, the registration deep learning model may determine the deformation parameters corresponding to respiratory amplitude points in each first image by inputting the preoperative image and different first images multiple times.

[0362] In some embodiments, the processor may determine the respiratory amplitude interval corresponding to each respiratory amplitude point, and determine the deformation parameter corresponding to the respiratory amplitude point of a certain first image as the deformation parameter of the respiratory amplitude interval corresponding to the respiratory amplitude point. Each respiratory amplitude interval includes a corresponding respiratory amplitude point, and within the respiratory amplitude interval, the deformation of the target part of the target object is less than a preset deformation threshold. When a respiratory amplitude interval corresponds to deformation parameters determined based on multiple first images, the average of the aforementioned multiple deformation parameters may be determined as the deformation parameter corresponding to the respiratory amplitude interval.

[0363] In some embodiments, the registration deep learning model can be trained using a plurality of seventh training samples with the seventh label. The specific training method is similar to the training method of the exposure range prediction model, and can be seen in the relevant description of FIG3 .

[0364] In some embodiments, the seventh training sample includes a sample preoperative image and multiple sample first images of different sample target objects; the seventh label includes a sample deformation parameter corresponding to each sample first image. In some embodiments, the seventh training sample can be obtained based on historical data (e.g., a historical preoperative image and multiple historical first images); and the seventh label can be determined by manually registering the sample preoperative image with the multiple sample first images.

[0365] In some embodiments of the present specification, by adopting a lightweight registration deep learning model and performing quantization, pruning, knowledge distillation and other processes on it, not only can real-time processing of preoperative images and multiple first images be achieved, avoiding occupying more memory and improving computing speed and processing performance, but also the influence of multiple factors is considered at the same time, so that the deformation parameters can be determined efficiently and accurately, avoiding errors in manual determination.

[0366] When the processor only captures one first image within the first time period, the processor may determine a deformation parameter based on the first image and the preoperative image, and store the correspondence between the respiratory amplitude point or respiratory amplitude interval of the first image and the deformation parameter in a storage device.

[0367] In some embodiments of this specification, by only acquiring a first image for registration to determine the deformation parameters corresponding to a certain respiratory amplitude point or respiratory amplitude interval, it is beneficial to reduce the amount of calculation during the early registration and the guidance of the later interventional surgery, shorten the early preparation time of the interventional surgery, and quickly enter the interventional surgery.

[0368] When the processor acquires multiple first images at different respiratory amplitude points within a first time period, the processor can determine a deformation parameter based on each first image and the preoperative image, and store the corresponding relationship between the respiratory amplitude point or respiratory amplitude interval of each first image and the deformation parameter to a storage device, so that the corresponding deformation parameter can be determined directly based on the respiratory amplitude point or respiratory amplitude interval corresponding to the second image. For example, the respiratory amplitude points corresponding to the multiple first images are C0, D0, and E0, respectively, and the corresponding deformation parameters are XC, XD, and XE, respectively. When the processor acquires a second image, the respiratory amplitude point of the target object is C0, then the deformation parameter corresponding to the second image is XC. For another example, the respiratory amplitude points corresponding to the multiple first images are C0, D0, and E0, respectively, and the corresponding deformation parameters are XC, XD, and XE, respectively. If the respiratory amplitude intervals corresponding to the multiple first images are [C1, C2], [D1, D2], and [E1, E2], respectively. Among them, if the respiratory amplitude point C0 is within the respiratory amplitude interval [C1, C2], the deformation parameter corresponding to the respiratory amplitude interval [C1, C2] is XC; if the respiratory amplitude point D0 is within the respiratory amplitude interval [D1, D2], the deformation parameter corresponding to the respiratory amplitude interval [D1, D2] is XD; if the respiratory amplitude point E0 is within the respiratory amplitude interval [E1, E2], the deformation parameter corresponding to the respiratory amplitude interval [E1, E2] is XE. If the processor captures a second image, the respiratory amplitude point of the target object is C3, and the respiratory amplitude point C3 is within the respiratory amplitude interval [C1, C2], then the deformation parameter corresponding to the second image is XC.

[0369] In some embodiments of the present specification, collecting multiple first images for registration can determine the deformation parameters corresponding to multiple different respiratory amplitude points or respiratory amplitude intervals, which is beneficial to improving the update frequency of the target guidance image during subsequent surgical execution, ensuring the accuracy of image registration, and making the determined target guidance image more accurate.

[0370] Step 1220 : Determine a target image for the second time period based on the deformation parameter, the preoperative image, and the second image.

[0371] For relevant descriptions about the second image and the target object, please refer to the corresponding description of FIG. 2 .

[0372] In some embodiments, the target image in the second time period may include a target guidance image. A target guidance image refers to a real-time image used to guide an interventional procedure. For example, the target guidance image may include a puncture needle, a lesion, and a target organ. The target guidance image may be a real-time image used to guide the puncture needle during the procedure.

[0373] The processor may determine at least one target guidance image of the second time period based on the at least one deformation parameter, the preoperative image, and the at least one second image through image fusion software or other suitable means.

[0374] In some embodiments, when there is only one deformation parameter X0, the processor can determine the respiratory amplitude point A3 or respiratory amplitude interval [A4, A5] corresponding to the deformation parameter X0. Respiratory amplitude point A3 is within the respiratory amplitude interval [A4, A5]. During the surgical procedure, when the respiratory amplitude point corresponding to the target object is respiratory amplitude point A3 or within the respiratory amplitude interval [A4, A5], the processor captures at least one second image and determines the deformation parameter corresponding to the at least one second image as the deformation parameter X0. After deforming the preoperative image based on the deformation parameter X0, the processor fuses the image with the at least one second image to determine the target guidance image corresponding to the at least one second image. Some embodiments of the present specification can reduce the amount of computation required for guidance during interventional surgery by determining a target guidance image for interventional surgery only within the respiratory amplitude point or respiratory amplitude interval corresponding to one deformation parameter.

[0375] In some embodiments, when there are multiple deformation parameters, the processor obtains the respiratory amplitude parameter corresponding to each deformation parameter. During the surgical procedure, when the target object is at the same respiratory amplitude parameter, the processor captures at least one second image. For each second image, the processor may determine the deformation parameter corresponding to the second image based on the respiratory amplitude parameter corresponding to the second image. After deforming the preoperative image using the corresponding deformation parameter, the preoperative image is fused with the second image to determine the target guidance image corresponding to the second image. As an example only, the deformation parameter corresponding to the respiratory amplitude interval [C1, C2] is XC, the deformation parameter corresponding to the respiratory amplitude interval [D1, D2] is XD, and the deformation parameter corresponding to the respiratory amplitude interval [E1, E2] is XE; if the respiratory amplitude points corresponding to the multiple second images F1, F2, and F3 collected by the processor are C3, D3, and E3 respectively, and the respiratory amplitude point C3 is within the respiratory amplitude interval [C1, C2], the respiratory amplitude point D3 is within the respiratory amplitude interval [D1, D2], and the respiratory amplitude point E3 is within the respiratory amplitude interval [E1, E2], then the deformation parameters corresponding to the second images F1, F2, and F3 are XC, XD, and XE respectively. The processor deforms the preoperative image G0 based on the deformation parameters XC, XD, and XE, respectively, to obtain deformed preoperative images G1, G2, and G3, and fuses the deformed preoperative images G1, G2, and G3 with the corresponding second images F1, F2, and F3, respectively, to obtain target guidance images H1, H2, and H3. It is understandable that when multiple deformation parameters cover the complete respiratory interval of the target object, the user can adjust the acquisition frequency of the second image (for example, 30 Hz) as needed, so that the target guidance image can be continuously output to guide the interventional surgery in real time. Some embodiments of the present specification can ensure the output frequency of the target guidance image through multiple deformation parameters, thereby ensuring the real-time and accuracy of the interventional guidance.

[0376] In some embodiments, the target guide image includes a first guide image and / or a second guide image.

[0377] The first guide image is represented by a scanned image of at least one slice associated with at least one second image and a preoperative image. For example, the first guide image is represented by a two-dimensional image of at least one slice of the target area of ​​the target object. The first guide image is represented by two-dimensional images of multiple slices of the target area of ​​the target object. The user can adjust the display of the two-dimensional images of different slices of the target area as needed.

[0378] Figure 20 is an exemplary schematic diagram of a first guide image according to some embodiments of this specification. As an example only, as shown in Figure 20 , first guide image 2000 is a two-dimensional image of a slice of the target area. First guide image 2000 is a two-dimensional image obtained by fusing a preoperative image 2010 subjected to deformation processing of the slice with a second image 2020 of the slice.

[0379] The second guidance image is represented by a three-dimensional model associated with at least one second image and the preoperative image. For example, the second guidance image is represented by a three-dimensional model including the puncture needle and related parts. The related parts may include the target part of the target patient and surrounding tissues and organs.

[0380] FIG21 is an exemplary schematic diagram of a second guide image according to some embodiments of the present specification. As an example only, as shown in FIG21 , the second guide image 2100 is represented by a 3D model of a puncture needle 2120 and a related part 2110 .

[0381] In some embodiments, the display module is further configured to display the first guidance image and / or the second guidance image.

[0382] In some embodiments of this specification, the target guidance image is divided into a first guidance image and a second guidance image. One or more of these images can be displayed based on the user's needs to meet the operating habits of different users. It should be understood that the target guidance image determined in some embodiments of this specification is only used to provide guidance to the user or doctor during interventional surgery.

[0383] In some embodiments, for each second image, the processor can obtain a respiratory amplitude parameter corresponding to the target subject at the time the second image was acquired; perform deformation processing on the preoperative image based on the deformation parameter corresponding to the respiratory amplitude parameter, and fuse the image with the second image to determine the first guidance image at the time corresponding to the second image. For more information on the above embodiment, please refer to Figure 17 and its related description.

[0384] In some embodiments, for each second image, the processor can segment the second image to determine a puncture needle image in the second image; obtain respiratory amplitude parameters corresponding to the target object when the second image was acquired; segment and 3D reconstruct the preoperative image, and deform the 3D reconstruction result based on deformation parameters corresponding to the respiratory amplitude parameters to obtain an initial model corresponding to the deformed preoperative image; and determine a second guide image at the corresponding moment of the second image based on the puncture needle image and the initial model. For more information on the above embodiments, please refer to Figure 17 and its related description.

[0385] In some embodiments of the present specification, image registration is completed during the preparation time before the interventional surgery, and real-time mapping of the preoperative image to the second image is performed during the surgery. The preoperative image obtained by the large-scale scan before the surgery is fused with the second image obtained by the narrow-scale scan during the surgery, and dynamic updates are performed at the same time. This achieves real-time tracking of organ deformation and a large field of view for surgical guidance, thereby improving the sense of space during the interventional process, controlling the radiation dose, and improving the safety of the surgery.

[0386] FIG17 is an exemplary flow chart of determining a first guide image according to some embodiments of this specification. In some embodiments, process 1700 may be executed by a processor. As shown in FIG17 , process 1700 may include the following steps.

[0387] In some embodiments, the deformation parameters include multiple ones, and each deformation parameter corresponds to a breathing amplitude parameter.

[0388] Step 1710: Acquire the respiratory amplitude parameter corresponding to the target object when the second image is acquired.

[0389] For more information about the respiratory amplitude parameter, please refer to Figure 12 and its related description.

[0390] In some embodiments, when the medical imaging device captures multiple second images, the processor may further use a respiratory gating device to capture respiratory amplitude points corresponding to the target subject within a second time period, with each second image corresponding to a respiratory amplitude point. Based on the respiratory amplitude point corresponding to a particular second image, the processor may determine the respiratory amplitude interval within which the respiratory amplitude point falls, thereby determining the respiratory amplitude interval corresponding to the second image.

[0391] Step 1720 : Perform deformation processing on the preoperative image based on the deformation parameter corresponding to the respiratory amplitude parameter to obtain the preoperative image after deformation processing.

[0392] In some embodiments, for each second image, the processor can access the storage device based on the respiratory amplitude parameters corresponding to the second image, determine the deformation parameters corresponding to the second image through the correspondence between the respiratory amplitude parameters and the deformation parameters, perform elastic deformation processing and / or rigid deformation processing on the target part in the preoperative image, and obtain the preoperative image after deformation processing.

[0393] In some embodiments, the processor may also consider other factors that affect organ deformation when determining the deformation parameters.

[0394] In some embodiments, other factors that affect organ deformation may include: the influence of other organ work on organ deformation (eg, the influence of heartbeat on organ deformation), and the influence of the patient's fat and muscle content on organ deformation.

[0395] In some embodiments, the processor may construct a third preset table that correlates other factors influencing organ deformation with changes in deformation parameters, and determine the deformation parameters by querying the third preset table. The third preset table, which includes correspondences between other factors influencing organ deformation and changes in deformation parameters, may be manually preset based on experience.

[0396] In some embodiments of this specification, by considering the influence of other factors affecting organ deformation on the organ shape, the deformation parameters finally determined can be made more consistent with the actual situation, so that the deformed image is more accurate and more referenceable.

[0397] Step 1730 : Fusing the deformed preoperative image with the second image to determine the first guiding image at the time corresponding to the second image.

[0398] For more information about the first guide image, please refer to FIG. 12 and its related description.

[0399] In some embodiments, the processor can perform image fusion on the target part in the deformed preoperative image and the target part in the second image to determine the first guide image when the second image is acquired within the second time period.

[0400] During the interventional surgery, the preparation time before the interventional surgery is used to complete image registration, and the deformation parameters corresponding to each respiratory amplitude parameter are saved according to the respiratory amplitude point of the target object. During the surgery, the deformation parameters of the respiratory amplitude parameters corresponding to the respiratory amplitude point of the target object at the moment of collecting the second image are directly used to deform the preoperative image and fuse it with the second image to obtain the guidance image when the second image is collected, which reduces the amount of calculation during image registration and image fusion and ensures the real-time nature of the guidance image.

[0401] In some embodiments, for each second image, the processor can segment the second image to determine the puncture needle image in the second image; obtain the respiratory amplitude parameters corresponding to the target object when the second image was acquired; segment and three-dimensionally reconstruct the preoperative image, deform the three-dimensionally reconstructed result based on the deformation parameters corresponding to the respiratory amplitude parameters, and obtain the initial model corresponding to the deformed preoperative image; and determine the second guide image at the corresponding moment of the second image based on the puncture needle image and the initial model. For more information about the second guide image, please refer to Figure 12 and its related description. For more information about obtaining the respiratory amplitude parameters corresponding to the target object when the second image was acquired, please refer to the related description of Figure 17.

[0402] In some embodiments, for each second image, the processor can process the second image using image segmentation software to obtain a puncture needle image.

[0403] In some embodiments, for each second image, the processor can process the second image based on the segmentation deep learning model to determine the puncture needle image in the second image.

[0404] FIG. 19 is an exemplary schematic diagram of a puncture needle in a second image according to some embodiments of the present specification.

[0405] As shown in FIG19 , the processor may process the second image shown in FIG19 based on the segmentation deep learning model to segment and obtain an image of the puncture needle B.

[0406] The segmentation deep learning model can be a machine learning model for determining the puncture needle. For example, the segmentation deep learning model can be a deep neural network (DNN) model.

[0407] It should be understood that in order to achieve real-time segmentation of the second image to obtain the puncture needle image, the segmentation deep learning model can be a lightweight model (for example, a simplified network structure). The processor can also quantize, prune, and perform knowledge distillation on the segmentation deep learning model to ensure that the segmentation deep learning model can achieve real-time processing of multiple second images. In order to further improve the computing speed and processing performance of the segmentation deep learning model, high-performance image processing hardware can be used when deploying the segmentation deep learning model.

[0408] In some embodiments, the input to the segmentation deep learning model can include a second image; the output can include an image of the puncture needle in the second image.

[0409] It should be understood that the processor can process the second images of different slices separately through the segmentation deep learning model to obtain the puncture needle images in the second images of different slices, and perform three-dimensional reconstruction of the puncture needle images to obtain a three-dimensional puncture needle image.

[0410] In some embodiments, the segmentation deep learning model can be trained using a plurality of eighth training samples with the eighth label. The training method of the segmentation deep learning model is similar to the training method of the exposure range prediction model. For details on the training of the segmentation deep learning model, please refer to the training of the exposure range prediction model in Figure 3.

[0411] In some embodiments, the eighth training sample can include a sample second image of a sample target object; the eighth label can include a sample puncture needle image. In some embodiments, the eighth training sample can be obtained based on historical data (e.g., a historical second image); and the eighth label can be determined based on manually segmenting the sample second image.

[0412] In some embodiments of the present specification, by adopting a lightweight segmentation deep learning model and performing quantization, pruning, knowledge distillation and other processing on it, not only can real-time processing of multiple second images be achieved, avoiding occupying more memory, improving computing speed and processing performance, but also considering the influence of multiple factors at the same time, making the segmented puncture needle image more efficient and accurate, and avoiding errors determined manually.

[0413] In some embodiments, the processor can segment the organ tissues of preoperative images of different slices based on the segmentation deep learning model, and perform three-dimensional reconstruction based on the organ tissues segmented from all slices to obtain an initial model corresponding to the preoperative image.

[0414] In some embodiments, the processor can perform deformation processing on the initial model corresponding to the preoperative image based on the deformation parameters corresponding to the second image to obtain the initial model corresponding to the preoperative image after deformation processing.

[0415] It should be understood that the processor can also perform segmentation, 3D reconstruction, and deformation processing on the preoperative image in a different order. For example, the processor can perform deformation processing on the preoperative image of different slices based on the deformation parameters corresponding to the second image to obtain preoperative images of different slices after deformation processing. Furthermore, the processor can segment the organs and tissues in the preoperative images of different slices after deformation processing based on the segmentation deep learning model, and perform 3D reconstruction based on the organs and tissues obtained from the segmentation of all slices to obtain the initial model corresponding to the preoperative image after deformation processing.

[0416] In some embodiments, the processor can determine the second guiding image at the time corresponding to the second image by a formula based on the initial model corresponding to the puncture needle image and the deformed preoperative image. The second guiding image can be determined by the following formula (4): Curent =F Curent +DVF A′ (F Before ) (4)

[0417] Among them, V Curent is the second guiding image at the corresponding moment of the second image, F Curent is the puncture needle image in the second image, F Before is the initial model corresponding to the preoperative image, DVF A′ (F Before) is the initial model corresponding to the preoperative image after deformation processing. That is, for each second image, the corresponding deformation parameter is determined by its corresponding respiratory amplitude parameter, and the deformation parameter is used to transform the initial model F corresponding to the preoperative image. Before Perform deformation processing to obtain the initial model DVF corresponding to the preoperative image after deformation processing A′ (F Before ), and compare it with the puncture needle image F in the second image Curent The second guide image V corresponding to the moment of collecting the second image is obtained by fusion. Curent .

[0418] In some embodiments, by segmenting the second image, a real-time image of the puncture needle during the interventional procedure is obtained, effectively reflecting the deformation of the puncture needle. By segmenting and reconstructing tissues and organs from different slices of preoperative images, an initial model corresponding to the preoperative image is obtained. This deformed model is then fused with the segmented puncture needle image. This not only makes the determined guidance image more accurate and better reflects the actual condition of the target area during the procedure, but also allows the second guidance image to be presented in three dimensions, giving it a greater sense of depth and spatiality.

[0419] FIG18 is an exemplary flow chart of determining an updated target guidance image according to some embodiments of this specification. In some embodiments, process 1800 may be executed by a processor. As shown in FIG18 , process 1800 may include the following steps.

[0420] Step 1810 : For each target guidance image, determine whether the puncture needle is completely displayed in the target guidance image.

[0421] It should be understood that, since the second image is acquired based on a smaller scanning range, the entire puncture needle may not be displayed in the second image, or there may even be no puncture needle in the second image.

[0422] In some embodiments, the processor can process the second image corresponding to the target guidance image through image recognition software to determine whether the puncture needle is completely displayed in the target guidance image.

[0423] In some embodiments, for each target guide image, the processor can determine the puncture needle image in the second image based on the second image corresponding to the target guide image; and determine whether the puncture needle is completely displayed in the target guide image based on the puncture needle image and the standard puncture needle image.

[0424] In some embodiments, the processor can process the second image corresponding to the target guidance image based on the segmentation deep learning model to determine the puncture needle image in the second image. For more information about the segmentation deep learning model, please refer to Figure 17 and its related description.

[0425] The standard puncture needle image refers to a complete image of the puncture needle used as a reference. The standard puncture needle image can be pre-stored in a storage device.

[0426] In some embodiments, the processor can obtain a standard puncture needle image based on the puncture needle image segmented from the second image corresponding to the guide image by accessing a storage device, and compare the segmented puncture needle image with the standard puncture needle image to determine whether the segmented puncture needle image is complete, thereby determining whether the puncture needle is completely displayed in the target guide image.

[0427] In some embodiments of the present specification, the puncture needle image obtained by segmenting the second image during the aforementioned image fusion is compared with a pre-stored standard puncture needle image. This can not only determine whether the puncture needle is completely displayed in the target guidance image, but also reduce the computational complexity of the processor and shorten the update time of the target guidance image.

[0428] In some embodiments, for each target guidance image, the processor can determine the first position information of the puncture needle in the image coordinate system based on the positioning position information of the puncture needle in the robot coordinate system at the moment corresponding to the target guidance image; obtain the third position information of the second image corresponding to the target guidance image in the image coordinate system; and based on the first position information and the third position information, determine whether the puncture needle is completely displayed in the target guidance image.

[0429] Position information refers to parameters related to the position of the puncture needle in the robot coordinate system. For example, the position information may include the three-dimensional coordinates of the starting point and the end point of the puncture needle in the robot coordinate system.

[0430] In some embodiments, the processor can obtain the positioning position information of the puncture needle in the robot coordinate system at the moment corresponding to the target guidance image by accessing the storage device of the surgical robot.

[0431] The first position information refers to parameters related to the position of the puncture needle in the image coordinate system.

[0432] In some embodiments, the processor can convert the positioning position information of the puncture needle in the robot coordinate system into first position information in the image coordinate system based on the mapping relationship between the robot coordinate system and the image coordinate system.

[0433] The third position information refers to a parameter related to the position of the second image in the image coordinate system. For example, the third position information may include the three-dimensional coordinates of the boundary contour of the second image in the image coordinate system.

[0434] In some embodiments, the processor can determine whether the puncture needle is completely displayed in the target guidance image by comparing whether the first position information of the puncture needle is entirely within the coverage range of the third position information of the second image. When the first position information of the puncture needle is entirely within the coverage range of the third position information of the second image, the processor determines that the puncture needle is completely displayed in the target guidance image; when the first position information of the puncture needle is partially within the coverage range of the third position information of the second image, the processor determines that the puncture needle is partially displayed in the target guidance image; when the first position information of the puncture needle is not within the coverage range of the third position information of the second image, the processor determines that the puncture needle is not displayed in the target guidance image.

[0435] In some embodiments of this specification, the second position information of the puncture needle in the image coordinate system is mapped through the positioning position information of the puncture needle in the robot coordinate system, and the display of the puncture needle in the target guidance image is judged based on the third position information of the second image in the image coordinate system. This is not only applicable to the case where the puncture needle is not displayed completely, but also to the case where the puncture needle is not displayed at all.

[0436] In some embodiments, if the segmented puncture needle image in the second image completely displays the entire puncture needle, the processor can directly fuse the segmented puncture needle image into the deformed preoperative image to determine the target guidance image at the time corresponding to the acquisition of the second image.

[0437] Step 1820 , in response to no, determine the time corresponding to the target guidance image, and obtain the positioning position information of the puncture needle in the robot coordinate system at that time through the surgical robot.

[0438] In some embodiments, the processor can obtain the positioning position information of the puncture needle in the robot coordinate system at the moment corresponding to the target guidance image by accessing the storage device of the surgical robot.

[0439] Step 1830 : Map the puncture needle to the target guidance image based on the positioning position information, and determine an updated target guidance image.

[0440] In some embodiments, the processor can convert the positioning position information of the puncture needle in the robot coordinate system into first position information in the image coordinate system based on the mapping relationship between the robot coordinate system and the image coordinate system, correct the first position information based on the second position information of the puncture needle, and fuse the pre-stored standard puncture needle image into the second image. It should be understood that due to deformation of the puncture needle during the surgical procedure, the real-time image of the puncture needle may differ from the pre-stored standard puncture needle image in the storage device. Therefore, after determining the first position information of the puncture needle in the image coordinate system, if the puncture needle is completely absent from the target guided image, the processor can directly correct the first position information or the positioning position information based on the second position information, and then map the pre-stored standard puncture needle image into the target guided image to determine an updated target guided image. If the puncture needle is partially but not completely visible in the target guided image, the pre-stored standard puncture needle image cannot be directly replaced with the segmented puncture needle image in the second image. The processor must supplement the missing portion of the segmented puncture needle image from the second image based on the standard puncture needle image to determine an updated target guided image. For more information about image fusion, please refer to Figure 17 and its related description.

[0441] It is worth noting that, since the puncture needle image may be deformed during the interventional procedure, if the pre-stored complete puncture needle image is directly used, the deformation of the puncture needle will not be reflected in the target guidance image. However, some embodiments of this specification perform real-time segmentation on the second image to obtain the puncture needle image corresponding to each second image and fuse it with the deformed preoperative image, taking into account the deformation of the puncture needle, and the resulting target guidance image is more accurate. When the segmented puncture needle image is incomplete or not displayed in the target guidance image, some embodiments of this specification use the positioning position information of the puncture needle in the robot coordinate system to fuse the pre-stored standard puncture needle image into the second image to obtain the target guidance image, thereby ensuring the integrity of the target guidance image during the interventional procedure.

[0442] FIG22 is an exemplary flow chart of a method for correcting the position of a puncture needle according to some embodiments of the present disclosure. In some embodiments, process 2200 is performed by a system for correcting the position of a puncture needle. As shown in FIG22 , process 2200 includes the following steps:

[0443] In some embodiments, the puncture needle position correction system further includes a positioning correction module, which can be configured to perform the following steps:

[0444] S2201: Determine first position information of the puncture needle in the image coordinate system based on the positioning position information of the puncture needle in the robot coordinate system.

[0445] For relevant descriptions of the positioning position information, the first position information, and the second position information, please refer to the corresponding description of Figure 18. In some embodiments, the positioning position information, the first position information, and the second position information may include information on the needle tip position and the needle tail position of the puncture needle.

[0446] The puncture image is an image obtained by scanning the puncture needle during the puncture process. In some embodiments, the puncture image includes a second image and / or a target guidance image. For more information about the second image, see the description related to FIG1 . For more information about the target guidance image, see the description related to FIG12 .

[0447] In the embodiment of this specification, during the puncture process using the puncture needle by the surgical robot, the positioning correction module can calculate the needle tip position and / or needle tail position of the puncture needle in real time, that is, calculate and obtain the first position information. The surgical robot can also be connected to a medical imaging device, and the medical imaging device performs an image scan on the puncture needle during the puncture process of the surgical robot to obtain a puncture image including the puncture needle. After the medical imaging device scans and obtains each frame of the puncture image, it can transmit each frame of the puncture image to the positioning correction module in real time, so that the positioning correction module can then verify the position of the puncture needle in the puncture image. In some embodiments, when the above-mentioned first position information is the position information in the robot coordinate system, after the surgical robot calculates the first position information, it can also convert the first position information in the robot coordinate system into the first position information in the image coordinate system according to the coordinate system registration information registered by itself, so as to feed the first position information back to the puncture image for post-processing and other operations.

[0448] S2202: Identify the needle tip position in the puncture image based on the first position information to obtain second position information of the puncture needle in the puncture image.

[0449] In some embodiments, the second position information includes needle tip information and needle tail information of the puncture needle in the puncture image.

[0450] In an embodiment of the present specification, when the positioning correction module obtains the first position information and the puncture image based on the aforementioned steps, it can identify the needle tip position of the puncture needle in the puncture image with reference to the needle tip information and needle tail information in the first position information, thereby obtaining second position information including the needle tip position information. This identification method may include: first, inputting the first position information and the puncture image together into a needle tip recognition network for needle tip recognition, thereby directly obtaining the second position information. The needle tip recognition network can be pre-trained for the positioning correction module based on a sample set of puncture images and a corresponding annotated image set. The annotated image set can be obtained by annotating the puncture image corresponding to the needle tip position of the puncture needle provided by the surgical robot. Second, processing the puncture image based on the first position information to obtain a puncture image that is easy to identify, and then inputting the processed puncture image into the needle tip recognition network for needle tip recognition to obtain the second position information.

[0451] S2203: Verify the first location information based on the second location information to obtain a verification result, and determine the target location information according to the verification result.

[0452] In some embodiments, the verification result includes whether the first location information is accurate or inaccurate.

[0453] In the embodiment of this specification, when the surgical robot obtains the first position information of the puncture needle calculated by itself and the second position information of the puncture needle in the puncture image, the second position information can be used as reference information to assist in verifying the accuracy of the first position information. Optionally, the surgical robot can verify by checking the consistency of the first position information and the second position information to obtain a verification result; optionally, the surgical robot can also calculate or process the first position information and the second position information to obtain the difference information between the first position information and the second position information, and then verify based on the difference information to obtain a verification result. When the surgical robot obtains a verification result, and the verification result shows that the first position information is accurate, it means that the second position information previously obtained based on the first position information is also accurate. At this time, the second position information can be determined as the target position information, so that the target position information can be used as the basis for providing guidance information for the subsequent puncture needle positioning.

[0454] This calibration method verifies the needle tip positioning in the puncture image using the puncture needle information provided by the surgical robot. This combines the needle tip positioning of the interventional robot hardware system with that of the image algorithm, significantly improving the precision and accuracy of needle tip positioning. Furthermore, because the surgical robot can provide real-time needle tip positioning, it enables real-time verification of the needle tip position in the puncture image, meeting clinical scenarios requiring real-time performance and improving the efficiency of puncture procedures.

[0455] FIG. 23 is an exemplary flow chart of a method for correcting the position of a puncture needle according to another embodiment of the present specification.

[0456] In one embodiment, a method for positioning a puncture needle in a puncture image is further provided, as shown in FIG23 . The method is performed by a positioning correction module and includes:

[0457] S2301: Crop the puncture image based on the first position information to obtain a cropped image.

[0458] In an embodiment of the present specification, when the surgical robot calculates the first position information, if the first position information is the position information in the robot coordinate system, the first position information in the robot coordinate system can be further converted into the first position information in the image coordinate system according to the coordinate system alignment information, and the model size of the puncture needle can be determined according to the needle tip information and needle tail information in the first position information, and then the puncture image is cropped according to the model size of the puncture needle to obtain a cropped image, so that the cropped image includes a complete puncture needle and rich information about the puncture needle.

[0459] S2302: Based on the cropped image, obtain a segmented image using a segmentation model.

[0460] The segmentation model is a machine learning model. For example, the segmentation model can be a pre-trained neural network model, which is used to segment the target tissue in the input image to obtain a corresponding mask image.

[0461] In the embodiments of this specification, the surgical robot can input the cropped image into a pre-trained segmentation model to perform image segmentation, thereby segmenting the structure of the puncture needle and obtaining a mask image, i.e., a segmented image. It should be noted that the segmentation model can be a lightweight neural network model or a compression model, enabling the segmentation network to complete image segmentation in a very short time, achieving real-time segmentation, thereby improving the real-time performance of subsequent verification.

[0462] S2303: Identify the needle tip position in the segmented image to obtain second position information of the puncture needle in the puncture image.

[0463] In an embodiment of this specification, when the surgical robot obtains a segmented image, it can first restore the size of the segmented image to the size of the puncture image, and identify the position of the needle tip in the restored segmented image to obtain the second position information of the puncture needle in the puncture image. Optionally, the surgical robot can use a pre-trained positioning network or segmentation network to obtain the needle tip position information of the puncture needle, i.e., the second position information of the puncture needle; optionally, the surgical robot can also use a high-dimensional data dimensionality reduction algorithm (Principal Component Analysis, PCA) to identify the needle tip direction in the restored segmented image, thereby obtaining the main puncture direction of the puncture needle in the segmented image to assist in locating the position of the puncture needle.

[0464] The positioning method described in the embodiment of this specification first crops the puncture image based on the first position information of the puncture needle provided by the surgical robot, and then performs image segmentation based on the cropped image, which greatly reduces the amount of calculation required for image segmentation, improves calculation efficiency, and thus can improve the real-time performance of puncture needle positioning.

[0465] FIG. 24 is an exemplary flow chart of a method for correcting the position of a puncture needle according to another embodiment of the present specification.

[0466] In one embodiment, the first position information is obtained by calculation by the robot. Therefore, this specification also provides a method for obtaining the first position information, as shown in FIG24 . The method for obtaining the first position information includes:

[0467] S2401, determining the end coordinates of the last segment of the robotic arm in the surgical robot based on the length and rotation angle of each segment of the robotic arm in the surgical robot.

[0468] The surgical robot may be a multi-degree-of-freedom robotic arm robot, for example, a five-degree-of-freedom robotic arm robot or a six-degree-of-freedom robotic arm robot.

[0469] In an embodiment of the present specification, the surgical robot can first calculate the position coordinates of the robotic arm of the first section based on the length and rotation angle of the robotic arm of the first section, and then calculate the position coordinates of the robotic arm of the second section based on the position coordinates of the robotic arm of the first section and the length and rotation angle of the robotic arm of the second section. The position coordinates of the robotic arm of the next section are calculated in this way until the end coordinates of the robotic arm of the last section are obtained.

[0470] S2402: Determine the position coordinates of the puncture needle in the robot coordinate system according to the end coordinates of the last section of the robotic arm and the size of the puncture needle.

[0471] The size of the puncture needle includes the actual needle tip position and needle tail position of the puncture needle. The position coordinates of the puncture needle in the robot coordinate system include the needle tip coordinates and the needle tail coordinates in the robot coordinate system.

[0472] In the embodiment of this specification, when the surgical robot obtains the end coordinates of the last section of the robotic arm, it can determine the needle tip coordinates of the puncture needle in the robot coordinate system in combination with the actual needle tip position of the puncture needle, and determine the needle tail coordinates of the puncture needle in the robot coordinate system in combination with the actual needle tail position of the puncture needle.

[0473] S2403 : Determine first position information of the puncture needle in the image coordinate system based on the position coordinates in the robot coordinate system and the coordinate system registration information.

[0474] The first position information includes the needle tip position and the needle tail position of the puncture needle. The coordinate system registration information includes a coordinate system registration matrix, and the coordinate system registration matrix represents the conversion relationship between the robot coordinate system and the image coordinate system.

[0475] In the embodiment of this specification, when the robot calculates the position coordinates of the puncture needle in the robot coordinate system, that is, obtains the needle tip coordinates and needle tail coordinates of the puncture needle in the robot coordinate system, the robot can extract the coordinate system registration matrix from the coordinate system registration information, and based on the coordinate system registration matrix, perform coordinate transformation on the needle tip coordinates of the puncture needle in the robot coordinate system, and perform coordinate transformation on the needle tail coordinates of the puncture needle in the robot coordinate system, specifically converting them into first position information including needle tip information and needle tail information in the image coordinate system.

[0476] FIG. 25 is an exemplary flowchart of a method for correcting the position of a puncture needle according to another embodiment of the present specification.

[0477] In one embodiment, a method is provided for correcting the position of a puncture needle in a puncture image based on position information provided by a surgical robot. That is, as shown in FIG. 25 , the above-mentioned S2203 “verifying the first position information based on the second position information to obtain a verification result” includes:

[0478] S2501 : Determine a positional distance between a position of a puncture needle located by a surgical robot and a position of the puncture needle in a puncture image based on first position information and second position information.

[0479] In an embodiment of the present specification, when the surgical robot verifies the first position information, it can first extract the needle tip position from the first position information and extract the needle tip position from the second position information, and then determine the distance between the two needle tip positions, that is, determine the distance between the needle tip position provided by the surgical robot and the needle tip position in the image algorithm, so that the needle tip position provided by the surgical robot can be verified based on the distance later.

[0480] S2502: Verify the first location information based on the location distance to obtain a verification result.

[0481] In the embodiment of this specification, after the surgical robot obtains the distance between the first position information and the second position information based on the aforementioned steps, it can directly verify the first position information based on the distance value of the distance, or it can evaluate the distance and then verify the first position information based on the evaluation result to obtain a verification result.

[0482] FIG. 26 is an exemplary flowchart of a method for correcting the position of a puncture needle according to another embodiment of the present specification.

[0483] As shown in FIG26 , the correction method is executed by the positioning correction module. The above-mentioned S2502 “verifying the first position information based on the position distance to obtain a verification result” includes:

[0484] S2601: Verify the first location information based on the location distance to obtain a verification result.

[0485] If the position distance is greater than the preset distance threshold, step S2602 is executed; if the position distance is not greater than the preset distance threshold, step S2603 is executed.

[0486] S2602: Determine whether the verification result indicates that the first location information is inaccurate.

[0487] S2603: Determine whether the verification result indicates that the first location information is accurate.

[0488] The preset distance threshold may be determined in advance based on the positioning accuracy of the puncture needle, and is a reference parameter for evaluating whether the first position information is accurate.

[0489] In some embodiments, the preset distance threshold can be adjusted according to different puncture needles, different medical imaging devices, and different robotic systems.

[0490] The robotic system refers to the system corresponding to different surgical robots.

[0491] For more information about the puncture needle, please refer to the description related to Figure 18. For more information about the medical imaging device, please refer to the description related to Figure 1.

[0492] In some embodiments, the positioning correction module can determine a corresponding preset distance threshold based on the corresponding puncture needle, medical imaging device, and robotic system by querying a fourth preset table. The preset distance threshold can exist in the fourth preset table for different puncture needles, different medical imaging devices, and different robotic systems.

[0493] In some embodiments, the positioning correction module may determine the preset distance threshold based on device parameters of the medical imaging device and an average value of imaging quality of one or more CT films.

[0494] In some embodiments, the device parameters of the medical imaging device may include scanning time, layer thickness, scanning algorithm, scanning mode, etc.

[0495] In some embodiments, the imaging quality of the CT film can be obtained by automatically performing a comprehensive evaluation on the film obtained after the CT scan by the medical imaging device.

[0496] In some embodiments, the positioning correction module may determine the preset distance threshold based on a ninth preset relationship between the device parameters of the medical imaging device, the average imaging quality of one or more CT films, and the preset distance threshold. For example, the ninth preset relationship may include a relationship between scan time, slice thickness, and the preset distance threshold; the longer the scan time and the thicker the slice, the larger the preset distance threshold may be set. For another example, the ninth preset relationship may include a relationship between the average imaging quality and the preset distance threshold; when the average imaging quality of one or more CT films is higher, it indicates that the CT films more accurately reflect the needle tip position. In this case, the preset distance threshold may be appropriately set larger.

[0497] In some embodiments, the positioning correction module may determine a preset distance threshold based on the type of the puncture needle and system parameters of the robotic system.

[0498] The system parameters of the robot system refer to the relevant parameters of the robot system corresponding to the surgical robot, for example, they may include master-slave control accuracy, theoretical cumulative error, equipment stability, etc.

[0499] In some embodiments, the positioning correction module may determine the preset distance threshold based on a tenth preset relationship between the type of puncture needle and system parameters of the robot system and the preset distance threshold. For example, the tenth preset relationship may include the relationship between master-slave control accuracy, theoretical cumulative error, device stability, and the preset distance threshold. For example, the higher the master-slave control accuracy, the smaller the theoretical cumulative error, and the higher the device stability, the larger the preset distance threshold may be set.

[0500] For another example, the tenth preset relationship may include the relationship between the type of puncture needle and the preset distance threshold.

[0501] In some embodiments of the present specification, by determining a preset distance threshold based on the equipment parameters of the medical imaging equipment and the average imaging quality of one or more CT films, the impact of the medical imaging equipment on the accuracy of the scanned image can be considered to determine a more reasonable preset distance threshold.

[0502] In some embodiments of the present specification, by determining a preset distance threshold based on the type of puncture needle and the system parameters of the robotic system, the impact of the type of puncture needle and the robotic system on the surgical fault tolerance can be taken into account to set a more reasonable preset distance threshold.

[0503] The embodiments of this specification relate to a method for verifying first position information based on a distance value of a position distance. If the position distance is greater than a preset distance threshold, it indicates that the difference between the first position information and the second position information is large, and in this case, the position information of the puncture needle provided by the surgical robot is considered inaccurate. If the position distance is not greater than the preset distance threshold, it indicates that the difference between the first position information and the second position information is small, and in this case, the position information of the puncture needle provided by the surgical robot is considered accurate. In response to the first position information being accurate, the positioning correction module can determine the second position information as the target position information.

[0504] In one embodiment, when the verification result indicates that the first position information is inaccurate, the method shown in Figure 26 above also includes the following steps: determining the cause of the inaccurate first position information, and determining the next action of the surgical robot based on the cause.

[0505] Among them, the causes of inaccurate first position information include cumulative errors of the surgical robot's positioning system, deformation of the puncture needle, mismatch of the puncture needle positioning algorithm, interference from environmental factors, etc.

[0506] In an embodiment of this specification, when the first position information is verified to be inaccurate, the surgical robot may further analyze the cause of the inaccurate first position information and, based on the cause, determine whether to continue the needle puncture, recalibrate the image positioning algorithm, or modify the robot's parameters, thereby determining the next action of the surgical robot. For example, when the surgical robot determines that the first position information is inaccurate, the surgical robot may self-check its own mechanical parameters or its own positioning system to determine whether the accumulated errors of the surgical robot's positioning system are causing the positioning problem, thereby affecting the accuracy of the first position information provided by the surgical robot. Optionally, the surgical robot may also detect the dimensions of the actual puncture needle to determine whether the puncture needle has deformed. If deformation has occurred, it is determined that the deformation of the puncture needle caused the inaccurate first position information provided by the surgical robot. Optionally, the surgical robot may determine that the inaccurate first position information is caused by other reasons besides the accumulated errors of the surgical robot's positioning system and the deformation of the puncture needle.

[0507] In one embodiment, when the surgical robot analyzes that the cause of the inaccurate first position information includes the cumulative error of the surgical robot's positioning system, the positioning correction module can perform the following steps: control the surgical robot to recalibrate, obtain the alignment information of the updated coordinate system, and control the surgical robot to obtain the updated first position information according to the alignment information of the updated coordinate system.

[0508] In the embodiments of this specification, the reasons for the inaccurate first position information include the cumulative error of the positioning system of the surgical robot, indicating that the mechanical state of the surgical robot has changed after long-term use, that is, the hardware may be damaged, resulting in inaccurate positioning system. In this case, the surgical robot can be recalibrated based on the current mechanical state, thereby obtaining new coordinate system registration information for registration, so that when the surgical robot calculates the position information of the next puncture needle in real time, it can calculate based on the new coordinate system registration information, thereby obtaining accurate positioning information of the puncture needle. It should be noted that the recalibration process includes: re-acquiring the first coordinate of the puncture needle in the new robot coordinate system, and re-acquiring the second coordinate of the puncture needle in the image coordinate system, constructing a new coordinate system registration matrix based on the conversion relationship between the first coordinate and the second coordinate, and then obtaining new coordinate system registration information from the newly constructed coordinate system registration matrix.

[0509] In some embodiments, the positioning correction module may adjust the scanning parameters of the medical imaging device based on the verification results and the accumulated error of the positioning system of the surgical robot.

[0510] In some embodiments, the scanning parameters of the medical imaging device may include scanning field of view, exposure dose, scanning type, layer thickness, and the number of acquisition channels used.

[0511] In some embodiments, the positioning correction module may determine scanning parameters for the medical imaging device based on the verification result, the cumulative error of the surgical robot's positioning system, and an eleventh preset relationship between the scanning parameters. For example, the eleventh preset relationship may be such that, when the verification result is inaccurate, the scanning field of view and exposure dose are adjusted based on the cumulative error of the surgical robot's positioning system to ensure that the needle tip is centered in the scanning field of view.

[0512] In some embodiments of the present specification, scanning parameters of the medical imaging device are adjusted according to the verification results and the accumulated error of the positioning system of the surgical robot, so that the medical imaging device can subsequently obtain more accurate scanning images based on the scanning parameters.

[0513] In one embodiment, when the surgical robot analyzes that the cause of the inaccurate first position information includes deformation of the puncture needle, the positioning correction module can perform the following steps: controlling the surgical robot to perform a needle retraction operation.

[0514] In the embodiments of this specification, the reasons for the inaccurate first position information include deformation of the puncture needle, indicating that the puncture needle has been deformed after long-term use, that is, it may be damaged, resulting in a deviation in the positioning of the puncture needle in the image. Therefore, in this case, the puncture cannot continue, and the surgical robot is controlled to withdraw the needle to avoid affecting the puncture. The surgical robot can also display an alarm message to inform the user that there is a puncture risk and stop the puncture operation. The surgical robot can also replace the puncture needle after withdrawing the needle so that the puncture operation can be performed again with a new puncture needle.

[0515] In some embodiments, the positioning correction module can determine the problem of the puncture needle, the cause of the problem, and the next action of the surgical robot based on the first position information and the second position information through the puncture needle monitoring model.

[0516] In some embodiments, the puncture needle monitoring model can be a machine learning model. For example, the puncture needle monitoring model can be a neural network model, a deep neural network model, or any combination thereof. In some embodiments, the input of the puncture needle monitoring model can include first position information and second position information, and the output can be the puncture needle problem, the cause of the problem, and the next action of the surgical robot.

[0517] In some embodiments, the puncture needle monitoring model can be trained based on a large number of ninth training samples with ninth labels. The ninth training sample can be the first position information and the second position information of the sample in the historical records. The ninth label can be the actual historical puncture needle problem corresponding to the ninth training sample, the cause of the problem, and the next action of the surgical robot when the puncture needle has the problem due to the cause. The ninth label can be automatically annotated by the system based on the historical records.

[0518] In some embodiments, the actual historical puncture needle problem and the cause of the problem corresponding to the ninth training sample in the ninth label can be obtained based on post-analysis / detection performed during the historical surgery corresponding to the ninth training sample. For example, the abnormality can be located after the puncture needle is removed, or other equipment can be used to more accurately locate the abnormality.

[0519] In some embodiments, the next action of the surgical robot in the actual history corresponding to the ninth training sample in the ninth label can be determined based on the actual execution action with good subsequent execution effect (obtained based on subsequent evaluation) in the historical surgery corresponding to the ninth training sample.

[0520] In some embodiments, the actual next action of the surgical robot in the history corresponding to the ninth training sample in the ninth label may also be determined based on the next action manually selected by the doctor in the historical surgery corresponding to the sample.

[0521] The training method of the puncture needle monitoring model is similar to that of the exposure range prediction model. For details, please refer to the relevant description of Figure 3.

[0522] In some embodiments of this specification, a puncture needle monitoring model is obtained through training. Through the puncture needle monitoring model, the problem of the puncture needle, the cause of the problem, and the next action of the surgical robot are determined. This can better monitor the condition of the puncture needle during the operation, so as to detect problems in time and provide adjustment ideas to improve surgical efficiency.

[0523] In some embodiments, the puncture needle monitoring model output may further include: whether a reminder needs to be sent to the user and corresponding reminder information.

[0524] In some embodiments, reminder information corresponding to different situations can be preset in advance.

[0525] In some embodiments, when the output of the puncture needle monitoring model includes whether a reminder needs to be sent to the user and the corresponding reminder information, the processor may add the corresponding ninth label when training the puncture needle monitoring model, indicating whether a reminder needs to be sent to the user and the corresponding reminder information corresponding to the ninth training sample. The determination of whether a reminder needs to be sent to the user and the corresponding reminder information corresponding to the ninth training sample can be determined based on the actual situation of the historical surgery corresponding to the ninth training sample. For example, if the puncture needle deformed after a certain moment in the historical surgery, the determination of whether a reminder needs to be sent to the user and the corresponding reminder information can be as follows: determining whether a reminder needs to be sent to the user and sending the corresponding needle withdrawal reminder information to the user.

[0526] In some embodiments of this specification, by including whether a reminder needs to be sent to the user and the corresponding reminder information as part of the output of the puncture needle monitoring model, the puncture needle monitoring process can be made more accurate and effective, which helps to detect problems and provide reminders in a timely manner.

[0527] FIG. 27 is an exemplary flowchart of a method for correcting the position of a puncture needle according to another embodiment of the present specification.

[0528] In summary of all the above embodiments, as shown in FIG27 , a method for correcting the position of a puncture needle is further provided, the method comprising:

[0529] S2701, determining the end coordinates of the last segment of the robotic arm in the surgical robot based on the length and rotation angle of each segment of the robotic arm in the surgical robot.

[0530] S2702: Determine the coordinates of the puncture needle in the robot coordinate system based on the end coordinates of the last section of the robotic arm and the size of the puncture needle.

[0531] S2703 , determining first position information of the puncture needle in the image coordinate system based on the coordinates of the puncture needle in the robot coordinate system and the coordinate system registration information; the first position information includes the needle tip position and the needle tail position of the puncture needle.

[0532] S2704, obtaining a puncture image.

[0533] S2705: Crop the puncture image according to the first position information to obtain a cropped image.

[0534] S2706: Input the cropped image into a segmentation network for segmentation to obtain a segmented image.

[0535] S2707: Adjust the size of the segmented image to the size of the original puncture image to obtain an adjusted segmented image.

[0536] S2708: Identify the needle tip position in the adjusted puncture image according to the first position information to obtain second position information of the puncture needle in the puncture image.

[0537] S2709: Determine the position distance between the puncture needle located by the surgical robot and the puncture needle in the puncture image based on the first position information and the second position information.

[0538] S2710: Verify the first location information according to the location distance to obtain a verification result.

[0539] If the distance is greater than the preset distance threshold, steps S2711 to S2715 are executed.

[0540] If the distance is not greater than the preset distance threshold, step S2716 is executed.

[0541] S2711: Determine if the verification result indicates that the first position information is inaccurate, and further determine whether the cause of the inaccurate first position information is the cumulative error of the positioning system of the surgical robot. If so, execute step S2712. If not, execute step S2713;

[0542] S2712: The surgical robot is recalibrated to obtain new coordinate system registration information, so that the surgical robot can obtain the next first position information according to the new coordinate system registration information.

[0543] S2713: Determine whether the cause of the inaccurate first position information is deformation of the puncture needle. If so, proceed to step S2714.

[0544] If not, execute step S2715;

[0545] S2714, the surgical robot performs the needle removal operation.

[0546] S2715: Display a prompt message to inform the user that there is a risk of puncture and stop the puncture.

[0547] S2716: Determine that the verification result indicates that the first position information is accurate, determine the second position information as the target position information, and continue puncturing according to the target position information.

[0548] The methods described in the above steps are all described in the above content. Please refer to the above description for details and will not be repeated here.

[0549] The method described in the embodiments of this specification effectively verifies the position of a puncture needle in a puncture image. By combining the needle tip position determined by the image algorithm with the needle tip position calculated by the interventional robot hardware system, the needle tip position determined by the image algorithm is accurately verified. Furthermore, if the positioning determined by the image algorithm is inaccurate, the surgical robot's positioning system can be promptly corrected, effectively guiding the surgical robot to perform the correct puncture, thereby improving the efficiency and accuracy of the puncture procedure.

[0550] FIG28 is a schematic structural diagram of a puncture assisting device according to other embodiments of this specification.

[0551] As shown in FIG28 , the puncture assisting device in the embodiment of this specification includes a display module 2803 , a scanning module 2804 and a puncture assisting module 2800 . The puncture assisting module 2800 is used to execute the steps executed by the puncture assisting device in any of the above puncture method embodiments.

[0552] In some embodiments, the puncture assistance module 2800 is used to perform the following steps: obtaining multiple scanned images obtained by the medical imaging device performing a scanning operation on the target object through the scanning module 2804; obtaining the puncture status information of the puncture needle in the surgical robot, and determining the position information of the needle tip point of the puncture needle and the position information of the predicted target point based on the puncture status information; marking the needle tip point and the predicted target point on the target image based on the position information of the needle tip point and the position information of the predicted target point to generate a marked image; displaying the marked image through the display module 2803. For more content, please refer to the relevant description of Figure 2.

[0553] In some embodiments, the puncture assist module 2800 can obtain multiple scan images directly from the medical imaging device through the scanning module, or can obtain multiple scan images indirectly from the medical imaging device; this specification does not limit this.

[0554] In some embodiments, as shown in FIG28 , the puncture assist device in other embodiments further includes a memory 2801 and a computer program 2802 stored in the memory 2801 and executable on the puncture assist module 2800. When the puncture assist module 2800 executes the computer program 2802, the steps executed by the puncture assist device in any of the above-described puncture method embodiments are implemented.

[0555] In some embodiments, the puncture assist module 2800 performs multiple steps when executing the computer program 2802. For more details, please refer to the relevant description of Figure 2.

[0556] FIG28 is merely an example of a puncture assisting device and does not constitute a limitation on the puncture assisting device. The puncture assisting device may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0557] FIG29 is a schematic structural diagram of a surgical robot according to some embodiments of the present specification.

[0558] As shown in Figure 29, the surgical robot in the embodiment of this specification includes a surgical robot main hand 2903, a surgical robot slave hand 2904, at least one processor 2900 (only one processor is shown in Figure 29), a memory 2901, and a computer program 2902 stored in the memory 2901 and executable on the at least one processor 2900. When the processor 2900 executes the computer program 2902, it implements the steps performed by the processor in the puncture assistance system embodiment in any of the above-mentioned embodiments.

[0559] FIG29 is merely an example of a surgical robot and does not constitute a limitation on the surgical robot. The surgical robot may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0560] Some embodiments of the present specification can display the breathing changes of the target object at different breathing amplitude points within the complete breathing interval through multiple first images collected in the aforementioned manner, so that the deformation parameters determined later can more completely reflect the deformation of the target part of the target object within the breathing interval.

[0561] FIG30 is a schematic structural diagram of a puncture needle position correction system according to some embodiments of the present specification.

[0562] The method for correcting the position of the puncture needle provided in the embodiment of this specification can be applied to the verification system shown in Figure 30. In which, the surgical robot 3001 is connected to the medical imaging device 3002 by wired or wireless communication. The surgical robot 3001 can obtain a puncture image from the medical imaging device 3002 and verify the position of the puncture needle in the puncture image based on the positioning information of the puncture needle provided by the surgical robot 3001. In which, the surgical robot 3001 can perform a puncture operation, and during the puncture process, the position of the puncture needle can be located in real time and the real-time positioning information can be obtained. The surgical robot 3001 includes a processing device, which can be but is not limited to various personal computers, laptops, smart phones, tablet computers, servers, etc. The medical imaging device 3002 can be various image scanning devices, such as electronic computed tomography (CT), magnetic resonance imaging (MRI), etc.

[0563] FIG31 is a schematic structural diagram of a device for correcting the position of a puncture needle according to some embodiments of the present specification.

[0564] In one embodiment, as shown in FIG31 , a device for correcting the position of a puncture needle is provided, including a positioning correction module. The positioning correction module includes:

[0565] The acquisition module 3110 is configured to determine first position information of the puncture needle in the image coordinate system based on the positioning position information, and acquire a puncture image.

[0566] The recognition module 3120 is configured to recognize the needle tip position in the puncture image based on the first position information, and obtain second position information of the puncture needle in the puncture image.

[0567] The verification module 3130 is configured to verify the first location information based on the second location information, obtain a verification result, and determine the target location information according to the verification result.

[0568] FIG32 is a schematic structural diagram of a device for correcting the position of a puncture needle according to other embodiments of this specification.

[0569] In one embodiment, as shown in FIG32 , the identification module 3120 includes:

[0570] a cropping unit 3121 configured to crop the puncture image based on the first position information to obtain a cropped image;

[0571] The segmentation unit 3122 is configured to obtain a segmented image based on the cropped image using a segmentation model.

[0572] The recognition unit 3123 is configured to recognize the needle tip position in the segmented image and obtain the second position information in the puncture image.

[0573] FIG33 is a schematic structural diagram of a device for correcting the position of a puncture needle according to other embodiments of this specification.

[0574] In one embodiment, as shown in FIG33 , the acquisition module 3110 includes:

[0575] The first determining unit 3111 is configured to determine the end coordinates of the last segment of the robotic arm in the surgical robot according to the length and rotation angle of each segment of the robotic arm in the surgical robot.

[0576] The second determining unit 3112 is configured to determine the position coordinates of the puncture needle in the robot coordinate system according to the end coordinates of the last section of the robotic arm and the size of the puncture needle.

[0577] The third determining unit 3113 is configured to determine first position information of the puncture needle in the image coordinate system based on the position coordinates in the robot coordinate system and the coordinate system registration information.

[0578] FIG34 is a schematic structural diagram of a device for correcting the position of a puncture needle according to other embodiments of this specification.

[0579] In one embodiment, as shown in FIG34 , the verification module 3130 includes:

[0580] The fourth determining unit 3131 is configured to determine a position distance between the position of the puncture needle located by the surgical robot and the position of the puncture needle in the puncture image based on the first position information and the second position information;

[0581] The verification unit 3132 is configured to verify the first location information based on the location distance to obtain a verification result.

[0582] Each module in the aforementioned puncture needle position verification device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0583] FIG35 is a diagram illustrating the internal structure of a computer device according to some embodiments of the present specification.

[0584] In one embodiment, a computer device is provided, which may be a terminal. A diagram of its internal structure may be shown in FIG35 . The computer device includes a processor, memory, a communication interface, a display module, and an input device connected via a system bus and an I / O interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication. The wireless communication may be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements a method for correcting the position of a puncture needle. The display module of the computer device may be a liquid crystal display or an electronic ink display. The input device of the computer device may be a touchscreen layer covering the display, or may be buttons, a trackball, or a touchpad provided on the computer device housing, or may be an external keyboard, touchpad, or mouse.

[0585] Those skilled in the art will understand that the structure shown in Figure 35 is merely a block diagram of a portion of the structure related to the solution of this specification, and does not constitute a limitation on the computer device to which the solution of this specification is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0586] The above embodiment provides a computer program product, whose implementation principle and technical effects are similar to those of the above method embodiment, and will not be repeated here.

[0587] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database or other media used in the embodiments provided in this specification may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0588] The databases involved in the various embodiments provided in this specification may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a distributed database based on blockchain.

[0589] The present invention also provides a chip within an electronic device, comprising a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin, or a circuit. The processing unit may execute computer instructions to cause the electronic device to perform the steps performed by the puncture assisting device in any of the puncture methods provided in the embodiments of the present invention, or the steps performed by the processor in the aforementioned puncture assisting device embodiments.

[0590] In some embodiments, the computer instructions are stored in a memory unit.

[0591] In some embodiments, the storage unit is a storage unit within the chip, such as a register, cache, etc. The storage unit can also be a storage unit within the terminal located outside the chip, such as ROM or other types of static storage devices that can store static information and instructions, random RAM, etc.

[0592] Among them, the puncture assist device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here. The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only an example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still fall within the spirit and scope of the exemplary embodiments of this specification.

[0593] This specification also uses specific terms to describe the embodiments of this specification. For example, "one embodiment," "an embodiment," and / or "some embodiments" refer to a feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "one embodiment," "an embodiment," or "an alternative embodiment" two or more times in different locations in this specification do not necessarily refer to the same embodiment. Furthermore, certain features, structures, or characteristics of one or more embodiments of this specification may be appropriately combined.

[0594] Similarly, it should be noted that, in order to simplify the presentation of this specification and thus facilitate understanding of one or more embodiments of the invention, the foregoing descriptions of the embodiments of this specification sometimes combine multiple features into a single embodiment, figure, or description thereof. However, this disclosure method does not imply that the subject matter of this specification requires more features than those recited in the claims. In fact, an embodiment may have fewer features than all of the features of a single disclosed embodiment.

[0595] In some embodiments, numbers are used to describe the quantity of components and attributes. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise stated, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the description and claims are approximate values, which may change according to the required characteristics of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the settings of such numerical values ​​are as accurate as possible within the feasible range.

[0596] Finally, it should be understood that the embodiments described in this specification are intended only to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, by way of example and not limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly described and illustrated in this specification.

Claims

1. A puncture assist system, characterized in that: comprising a processor configured to: Obtaining a scanned image of the target object; the scanned image includes at least one of a preoperative image, a first image in a first time period, and a second image in a second time period; wherein the first time period is earlier than the second time period; generating puncture guidance information based on the puncture state information and the scanned image; and guiding the puncture operation based on the puncture guidance information; The puncture guidance information is displayed.

2. The puncture assist system according to claim 1, characterized in that: The processor is further configured to: Acquiring the puncture status information through a surgical robot; Based on the puncture state information, determine the position information of the first needle tip point of the puncture needle and the position information of the predicted target point; the predicted target point refers to the spatial point reached by the needle tip point of the puncture needle after the puncture needle completes the insertion according to the posture in the puncture state information; Based on the position information of the first needle tip point and the position information of the predicted target point, generating a labeled image, wherein the labeled image is an image of a target image labeled with the first needle tip point and the predicted target point; the target image is determined based on the scanned image; The annotated image is displayed.

3. The puncture assist system according to claim 2, characterized in that: The position information of the first needle tip point is calculated based on the relative position relationship between the puncture needle and the surgical robot; the processor is further configured to: Determining a needle track contour of the puncture needle based on the target image; Based on the needle track profile, determining position information of a second needle tip point; Determining whether there is an abnormality in the needle track contour based on the position information of the first needle tip point and the position information of the second needle tip point; In response to the needle track profile being abnormal, a needle track abnormality warning is issued.

4. The puncture assist system according to claim 2, characterized in that: The processor is further configured to: In response to receiving the first posture adjustment instruction, determining lesion position information of the target lesion based on the scanned image; Determining first posture adjustment data based on the puncture state information and the lesion location information; The first posture adjustment data represents posture change data of the puncture needle when the predicted target point is adjusted to the lesion position corresponding to the lesion position information; Based on the first posture adjustment data, a first movement instruction is sent to the surgical robot to control the surgical robot to adjust the posture of the puncture needle.

5. The puncture assist system according to claim 4, characterized in that: Before sending the first movement instruction to the surgical robot, the processor is further configured to: Determine whether at least one of the following needle withdrawal conditions exists, wherein the needle withdrawal conditions include: The puncture depth of the puncture needle is greater than a first threshold; or The distance between the predicted target point and the lesion position is greater than a second threshold; or The posture adjustment based on the first posture adjustment data will cause the puncture needle to interfere with the target tissue organ of the target object; In response to the existence of at least one of the needle withdrawal conditions, a second movement instruction is sent to the surgical robot to control the surgical robot to move the puncture needle for needle withdrawal.

6. The puncture assist system according to claim 2, characterized in that: The processor is further configured to: In response to receiving the second posture adjustment instruction, a third movement instruction is sent to the surgical robot to control the surgical robot to adjust the posture of the puncture needle; the second posture adjustment instruction is generated based on the posture adjustment action of the user on the main hand end of the surgical robot.

7. The puncture assist system according to claim 6, characterized in that: Before sending the third movement instruction to the surgical robot, the processor is further configured to: In response to receiving the needle withdrawal action of the user at the main hand end of the surgical robot, a fourth movement instruction is sent to the surgical robot to control the surgical robot to move the puncture needle for needle withdrawal; the fourth movement instruction is generated based on the needle withdrawal action.

8. The puncture assist system according to claim 2, characterized in that: The scanned image is generated based on the processor automatically controlling the medical imaging device to scan the target object; the processor is further configured to: In response to receiving the signal from the surgical robot to perform the puncture operation, sending a scanning instruction to the processor so that the processor controls the medical imaging device to perform scanning; Wherein, the puncture operation includes a posture adjustment operation and a needle insertion operation; the posture adjustment operation includes adjusting the posture of the puncture needle, and the needle insertion operation includes adjusting the puncture depth of the puncture needle.

9. The puncture assist system according to claim 1, characterized in that: The processor is further configured to: determining deformation parameters based on the preoperative image and the first image; Based on the deformation parameters, the preoperative image and the second image, a target image of the second time period is determined; the target image of the second time period includes a target guide image.

10. The puncture assist system according to claim 9, characterized in that: The target guiding image includes a first guiding image and / or a second guiding image; wherein, The first guide image is represented by a scanned image of at least one slice associated with the second image and the preoperative image; The second guidance image is represented by a three-dimensional model associated with the second image and the pre-operative image.

11. The puncture assist system according to claim 9, characterized in that: The deformation parameters include a plurality of deformation parameters, each of which corresponds to a breathing amplitude parameter, and the processor is further configured to: Acquiring a respiratory amplitude parameter corresponding to the target object when acquiring the second image; Performing deformation processing on the preoperative image based on the deformation parameter corresponding to the respiratory amplitude parameter to obtain the preoperative image after deformation processing; The deformed preoperative image is fused with the second image to determine the first guide image at a time corresponding to the second image.

12. The puncture assist system according to claim 11, characterized in that: The processor is further configured to: Acquire a plurality of first images under different respiratory amplitude parameters; Based on the registration of the multiple first images with the preoperative images respectively, a plurality of deformation parameters are acquired.

13. The puncture assist system according to claim 9, characterized in that: The processor is further configured to: performing segmentation processing on the second image to determine a puncture needle image in the second image; Acquiring a respiratory amplitude parameter corresponding to the target object when acquiring the second image; Segmenting and three-dimensionally reconstructing the preoperative image, and deforming the three-dimensionally reconstructed result based on the deformation parameter corresponding to the respiratory amplitude parameter to obtain an initial model corresponding to the preoperative image after the deformation processing; Based on the puncture needle image and the initial model, the second guide image at the time corresponding to the second image is determined.

14. The puncture assist system according to claim 9, characterized in that: The processor is further configured to: When the respiratory pressure value and / or respiratory amplitude of the target object meets a preset condition, the first image is acquired, wherein when the first image is acquired, the respiratory amplitude point of the target object is within a target respiratory amplitude range.

15. A puncture needle position correction system, characterized in that: The method comprises a positioning correction module, wherein the positioning correction module is configured to: Based on the positioning position information of the puncture needle in the robot coordinate system, determining the first position information of the puncture needle in the image coordinate system, wherein the first position information includes the needle tip position and the needle tail position of the puncture needle; Identify the needle tip position in the puncture image based on the first position information to obtain second position information of the puncture needle in the puncture image; the puncture image includes a second image and / or a target guide image; The first position information is verified based on the second position information to obtain a verification result, and target position information is determined according to the verification result; the target position information includes position information of the first needle tip point.

16. The correction system according to claim 15, characterized in that The positioning correction module is further configured as follows: Cropping the puncture image based on the first position information to obtain a cropped image; Based on the cropped image, obtaining a segmented image through a segmentation model, wherein the segmentation model is a machine learning model; The needle tip position in the segmented image is identified to obtain the second position information in the puncture image.

17. The correction system according to claim 15, characterized in that: The positioning correction module is further configured as follows: Based on the first position information and the second position information, determining a position distance between the position of the puncture needle located by the surgical robot and the position of the puncture needle in the puncture image; The first location information is verified based on the location distance to obtain a verification result.

18. The correction system according to claim 17, characterized in that: The positioning correction module is further configured as follows: If the location distance is greater than a preset distance threshold, determining that the verification result indicates that the first location information is inaccurate; If the location distance is not greater than a preset distance threshold, it is determined that the verification result indicates that the first location information is accurate.

19. The correction system according to claim 18, characterized in that The positioning correction module is further configured as follows: In response to the first location information being accurate, determining the second location information as the target location information; In response to the first position information being inaccurate, a cause of the inaccurate first position information is determined, and a next action of the surgical robot is determined based on the cause.

20. The correction system according to claim 19, characterized in that The positioning correction module is further configured as follows: If the cause is the accumulated error of the positioning system of the surgical robot, controlling the surgical robot to recalibrate to obtain registration information of the updated coordinate system; The surgical robot is controlled to obtain updated first position information based on the registration information of the updated coordinate system.

21. The correction system according to claim 19, wherein the positioning correction module is further configured to: If the cause includes deformation of the puncture needle, the surgical robot is controlled to perform a needle withdrawal operation.

22. The correction system according to claim 15, characterized in that The positioning correction module is further configured as follows: For each of the target guidance images, determining whether the puncture needle is completely displayed in the target guidance image; In response to no, determining a time corresponding to the target guidance image, and obtaining, by a surgical robot, positioning position information of the puncture needle in the robot coordinate system at the time; The puncture needle is mapped to the target guidance image based on the positioning position information, and an updated target guidance image is determined.

23. The correction system according to claim 22, characterized in that The positioning correction module is further configured as follows: For each of the target guide images, based on the second image corresponding to the target guide image, determining a puncture needle image in the second image corresponding to the target guide image; Based on the puncture needle image and the standard puncture needle image in the second image, it is determined whether the puncture needle is completely displayed in the target guide image.

24. The correction system according to claim 22, characterized in that The positioning correction module is further configured as follows: For each of the target guidance images, based on the positioning position information of the puncture needle in the robot coordinate system at a time corresponding to the target guidance image, determine the first position information of the puncture needle in the image coordinate system; Acquire third position information of a second image corresponding to the target guiding image in the image coordinate system; Based on the first position information and the third position information, it is determined whether the puncture needle is completely displayed in the target guide image.

25. A puncture assisting device, characterized in that: include: Scanning module, puncture auxiliary module and display module; The scanning module is configured to obtain a scanned image of the target object; the scanned image includes at least one of a preoperative image, a first image in a first time period, and a second image in a second time period; wherein the first time period is earlier than the second time period; The puncture assisting module is configured to: generate puncture guiding information based on the puncture state information and the scanned image; guiding a puncture operation based on the puncture guidance information; The display module is configured to display the puncture guide information.

26. A surgical robot, characterized in that: The surgical robot comprises: a surgical robot master hand, a first processor and a surgical robot slave hand, the puncture needle is arranged on the surgical robot slave hand, and the surgical robot master hand is communicatively connected with the puncture auxiliary device; wherein: The main hand of the surgical robot is used to send the first puncture operation instruction to the first processor after receiving the first puncture operation instruction, and send a motion instruction for performing the puncture operation to the puncture auxiliary device; The first processor is configured to send a fifth movement instruction corresponding to the first puncture operation instruction to the slave hand of the surgical robot after receiving the first puncture operation instruction; The puncture assisting device is used to send a scanning instruction to the medical imaging device after receiving the movement instruction, so that the medical imaging device performs a medical imaging scan on the target object; The surgical robot's slave hand is used to move the puncture needle according to the fifth movement instruction to perform a puncture operation corresponding to the first puncture operation instruction. The puncture operation includes a posture adjustment operation or a needle insertion operation. The posture adjustment operation is used to adjust the posture of the puncture needle, and the needle insertion operation is used to adjust the puncture depth of the puncture needle.

27. The surgical robot according to claim 26, The invention is characterized in that: The surgical robot main hand is used to send the second puncture operation instruction to the first processor after receiving the second puncture operation instruction; The first processor is used to determine a sixth movement instruction corresponding to the second puncture operation instruction; The first processor is used to send the sixth movement instruction to the surgical robot slave hand, and send a movement instruction for performing a puncture operation to the puncture auxiliary device; The puncture assisting device is used to send a scanning instruction to the medical imaging device after receiving the movement instruction, so that the medical imaging device performs the medical imaging scan on the target object; The surgical robot slave hand is used to move the puncture needle according to the sixth movement instruction to perform a puncture operation corresponding to the second puncture operation instruction; The first processor is used to send the sixth movement instruction to the surgical robot's slave hand, and the time interval between the surgical robot's slave hand executing the sixth movement instruction is used to perform the medical image scan before the puncture operation is performed.

Images