Interventional guidance system, method and device and storage medium

Abstract

The embodiment of the invention provides an intervention guidance system, method and device and a storage medium. The system comprises an image scanning module, an image registration module and an image mapping module. The image scanning module is used for acquiring a preoperative scanning image of a target object, at least one first scanning image in a first time period and at least one second scanning image in a second time period, and the first time period is earlier than the second time period; the image registration module is used for determining at least one deformation parameter based on the preoperative scanning image and the at least one first scanning image; the image mapping module determines at least one target interventional guidance image for a second time period based on the at least one deformation parameter, the preoperative scan image, and the at least one second scan image.

Description

Technical Field

[0001] The present specification relates to the field of medical technology, and in particular to an intervention guidance system, method, device and storage medium. Background Art

[0002] Interventional surgery is currently the most commonly used method for cancer diagnosis and treatment in clinical practice. However, during the operation, the patient needs to be continuously scanned to determine the relative position of the lesion and the interventional needle, which results in multiple pauses during the operation and low intervention accuracy. Long-term surgery will also increase the radiation dose.

[0003] Moreover, during the operation, due to limitations such as radiation dose and imaging time, the scanning range of the scanning equipment is small during real-time interventional guidance, and only a small number of slices can be reconstructed. Real-time interventional guidance is performed on a two-dimensional level, resulting in the doctor's lack of spatial sense during the operation and difficulty in adjusting the interventional needle. Even if the interventional target is clearly known, it is still difficult to quickly adjust the interventional needle to the target.

[0004] Therefore, in order to reduce the radiation dose during surgery, shorten the imaging time, and enhance the doctor's sense of space during the interventional surgery, it is necessary to provide an interventional guidance system, method, device and storage medium. Summary of the invention

[0005] One or more embodiments of the present specification provide an interventional guidance system, the system comprising: an image scanning module, an image registration module and an image mapping module; the image scanning module is used to acquire a preoperative scanning image of a target object, at least one first scanning image of a first time period and at least one second scanning image of a second time period, wherein the first time period is earlier than the second time period; the image registration module is used to determine at least one deformation parameter based on the preoperative scanning image and the at least one first scanning image; the image mapping module determines at least one target interventional guidance image of the second time period based on the at least one deformation parameter, the preoperative scanning image and the at least one second scanning image.

[0006] One or more embodiments of the present specification provide an interventional guidance method, the method comprising: acquiring a preoperative scanning image of a target object, at least one first scanning image of a first time period, and at least one second scanning image of a second time period, wherein the first time period is earlier than the second time period; determining at least one deformation parameter based on the preoperative scanning image and the at least one first scanning image; and determining at least one target interventional guidance image of the second time period based on the at least one deformation parameter, the preoperative scanning image, and the at least one second scanning image.

[0007] One or more embodiments of the present specification provide an interventional guidance device, which includes at least one processor and at least one memory; the at least one memory is used to store computer instructions; the at least one processor is used to execute at least part of the computer instructions to implement the interventional guidance method described in any one of the above embodiments.

[0008] One or more embodiments of the present specification provide a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the intervention guidance method as described in any one of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0010] Figure 1 is an exemplary module diagram of an intervention guidance system according to some embodiments of this specification;

[0011] Figure 2 is an exemplary flow chart of an intervention guidance method according to some embodiments of this specification;

[0012] Figure 3 is a schematic diagram of target breathing amplitude points according to some embodiments of this specification;

[0013] Figure 4 is a schematic diagram of a target breathing amplitude interval according to some embodiments of this specification;

[0014] Figure 5 is a schematic diagram of real-time updating of images according to some embodiments of this specification;

[0015] Figure 6 is a schematic diagram of an ideal breathing amplitude point of a target object during a surgical procedure according to some embodiments of the present specification;

[0016] Figure 7 is an exemplary flow chart of determining a first interventional guidance image according to some embodiments of this specification;

[0017] Figure 8 is an exemplary flow chart of determining an updated target intervention guidance image according to some embodiments of this specification;

[0018] Fig. 9 is a schematic diagram of a surgical instrument in a second image according to some embodiments of the present specification;

[0019] Fig.10 is a schematic diagram of a first interventional guidance image according to some embodiments of this specification;

[0020] Fig.11 is a schematic diagram of a second interventional guidance image according to some embodiments of the present specification. DETAILED DESCRIPTION

[0021] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0022] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0023] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0024] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0025] Interventional surgery is currently the most commonly used method for cancer diagnosis and treatment in clinical practice. However, during the operation, the patient needs to be continuously scanned to determine the relative position of the lesion and the interventional needle, which results in multiple pauses during the operation and low intervention accuracy. Long-term surgery will also increase the radiation dose.

[0026] Some embodiments of the present specification provide an interventional guidance system, method, device and storage medium. The aforementioned interventional guidance system can determine at least one deformation parameter through a preoperative scanning image and at least one first scanning image; based on the at least one deformation parameter, the preoperative scanning image and at least one second scanning image, determine at least one target interventional guidance image, thereby shortening the imaging time, reducing the radiation exposure dose during the operation, and helping to enhance the doctor's sense of space during the execution of the interventional surgery, making the interventional surgery more accurate, safe and fast.

[0027] Figure 1 1 is an exemplary module diagram of an intervention guidance system according to some embodiments of the present specification. In some embodiments, the intervention guidance system 100 includes an image scanning module 110 , an image registration module 120 and an image mapping module 130 .

[0028] The image scanning module 110 may be used to acquire a preoperative scan image of the target object, at least one first scan image of a first time period, and at least one second scan image of a second time period, wherein the first time period is earlier than the second time period.

[0029] In some embodiments, the image scanning module 110 is further used to acquire at least one first scanning image when the respiratory pressure value and / or respiratory amplitude of the target object meets preset conditions, wherein when acquiring at least one first scanning image, the respiratory amplitude point of the target object is within the target respiratory amplitude range.

[0030] The image registration module 120 may be configured to determine at least one deformation parameter based on the pre-operative scan image and the at least one first scan image.

[0031] In some embodiments, the image registration module 120 is further used to process the preoperative scan image and the at least one first scan image based on the registration deep learning model to determine at least one deformation parameter.

[0032] The image mapping module 130 may be configured to determine at least one target intervention guidance image for a second time period based on at least one deformation parameter, a preoperative scan image, and at least one second scan image.

[0033] In some embodiments, for each second scan image, the image mapping module 130 is further used to obtain the respiratory amplitude parameters corresponding to the target object when the second scan image is acquired; deform the preoperative scan image based on the deformation parameters corresponding to the respiratory amplitude parameters to obtain the deformed preoperative scan image; fuse the deformed preoperative scan image with the second scan image to determine the first interventional guidance image at the corresponding moment of the second scan image.

[0034] In some embodiments, for each second scan image, the image mapping module 130 is further used to perform segmentation processing on the second scan image to determine the surgical instrument image in the second scan image; obtain the respiratory amplitude parameters corresponding to the target object when the second scan image is acquired; based on the deformation parameters corresponding to the respiratory amplitude parameters, perform segmentation, three-dimensional reconstruction and deformation processing on the preoperative scan image to obtain the initial model corresponding to the preoperative scan image after deformation processing; based on the surgical instrument image and the initial model, determine the second interventional guidance image at the corresponding moment of the second scan image.

[0035] In some embodiments, for each second scan image, the image mapping module 130 is further used to process the second scan image based on the segmentation deep learning model to determine the surgical instrument image in the second scan image.

[0036] In some embodiments, for each target interventional guidance image, the image mapping module 130 is also used to determine whether the surgical instrument is completely displayed in the target interventional guidance image; in response to no, determine the time corresponding to the target interventional guidance image, and obtain the first spatial position of the surgical instrument in the surgical robot coordinate system through the surgical robot; based on the first spatial position, map the surgical instrument to the target interventional guidance image, and determine an updated target interventional guidance image.

[0037] In some embodiments, the intervention guidance system 100 further includes a prompt module 140 .

[0038] The prompt module 140 may be used to prompt the target object that the breathing amplitude point is within the target breathing amplitude range during the second time period.

[0039] For more information about the preoperative scan image, at least one first scan image of the first time period, at least one second scan image of the second time period, target respiratory amplitude range, deformation parameters, registration deep learning model, target interventional guidance image, first interventional guidance image, second interventional guidance image, and respiratory amplitude parameter, please refer to Figure 2 For more information about image fusion and segmentation deep learning models, see Figure 7 For more information on surgical instruments, see Figure 8 and its related description.

[0040] It should be noted that the above description of the interventional guidance system 100 and its modules is only for convenience of description and does not limit the present specification to the scope of the embodiments. It is understandable that, after understanding the principle of the system, those skilled in the art may arbitrarily combine the modules or form a subsystem to connect with other modules without deviating from the principle. In some embodiments, Figure 1 The image scanning module 110, the image registration module 120 and the image mapping module 130 disclosed in the specification can be different modules in a system, or one module can realize the functions of two or more modules. For example, each module can share a storage module, or each module can have its own storage module. Such variations are within the protection scope of this specification.

[0041] Figure 2 is an exemplary flow chart of an interventional guidance method according to some embodiments of this specification. In some embodiments, process 200 is executed by a processor. The aforementioned processor can be set in a medical device or a computer of a medical device, and can also be set in a computer or server independent of the medical device. The processor is used to process data and / or information obtained from a medical device (such as a scanning device). The processor executes program instructions based on these data, information and / or processing results to perform one or more functions described in this specification. Figure 2 As shown, process 200 may include the following steps.

[0042] Step 210 , obtaining a preoperative scan image of the target object, at least one first scan image in a first time period, and at least one second scan image in a second time period.

[0043] The target object refers to the person who needs to undergo interventional surgery, which includes but is not limited to percutaneous intervention, transvascular intervention, puncture biopsy, etc.

[0044] Preoperative scan images refer to images of the target part (e.g., lesions and target organs) of the target object when the user (e.g., the doctor performing the operation) performs operation planning (e.g., determining the operation path, needle entry point, etc.) before the operation is performed. Preoperative scan images may include images of different slices of the target part of the target object. Among them, before the operation is performed may include any time before the operation is performed on the target object. The user may scan the target object before the operation is performed to obtain a preoperative scan image, thereby facilitating the operation planning of the target object.

[0045] The processor can scan the target part of the target object by scanning equipment (e.g., a computed tomography (CT) scanner) before the operation to obtain a preoperative scan image. In order to determine the specific location of the lesion and the distribution of surrounding organs and tissues, the preoperative scan image can be based on a larger scanning range (e.g., 200mm) to scan the target object and reconstruct the image, and the obtained image includes the target part and the surrounding organs and tissues.

[0046] In some embodiments, when the scanning device acquires the preoperative scanning image, the processor can also collect the respiratory amplitude point corresponding to the target object at that moment through the respiratory gating device, and determine it as the target respiratory amplitude point.

[0047] The respiratory amplitude point refers to the parameter used to reflect the degree of chest fluctuation when the target subject breathes. The target respiratory amplitude point refers to the respiratory amplitude point of the target subject when the preoperative scan image is collected. Figure 3 As shown, the horizontal axis is time, and the vertical axis is the respiratory amplitude point of the target object acquired by the processor. When the processor acquires the preoperative scan image of the target object through the scanning device at time t0, the respiratory amplitude point A0 corresponding to the target object is the target respiratory amplitude point.

[0048] After obtaining the preoperative scan image, it is also necessary to obtain the first scan image in the first time period to align the first scan image with the preoperative scan image, so as to obtain the deformation parameter under the corresponding respiratory amplitude parameter. For more information about the respiratory amplitude parameter and the deformation parameter, please refer to the following of this specification.

[0049] The first time period refers to a period of registration preparation time before and immediately before the operation. For example, the first time period may be the first 3 to 7 seconds before the operation is performed. The operation period refers to the time period from the official start of the operation on the target object to the end of the operation. For example, the operation period refers to the time period from the user officially starting to insert the surgical instrument (such as a puncture needle) used in the interventional operation into the body of the target object to the removal of the aforementioned surgical instrument from the body of the target object.

[0050] The first scan image refers to an image obtained by real-time acquisition of the target part when the operation is about to be performed. For example, the first scan image can be a real-time scan image obtained by acquiring the target part within a complete breathing cycle (for example, 3 to 4 seconds) of the target object when the operation is about to be performed. For another example, the first scan image can be a real-time scan image obtained by acquiring the target part when the target object is performing breathing training when the operation is about to be performed. For another example, the first scan image can be a real-time scan image obtained by acquiring the target part at any time when the target object has breathing changes when the operation is about to be performed. In some embodiments, the first scan image is acquired after the preoperative scan image. For example, the preoperative scan image is acquired 2 to 3 days before the operation is performed, and the first scan image is acquired 3 to 7 seconds before the operation is about to be performed.

[0051] The processor can continuously scan the target part through the scanning device within the first time period, reconstruct in real time, and transmit the reconstructed scanned image to the processor to obtain a first scanned image. In order to accurately reflect the situation of the lesion area, the first scanned image can be an image obtained by scanning the target part and reconstructing the image based on a smaller scanning range (for example, 20mm). It should be understood that in order to subsequently determine multiple target interventional guidance images so as to better guide the interventional surgery, the processor can collect multiple first scanned images within the first time period.

[0052] In order to perform real-time intervention guidance during the subsequent operation, the processor needs to collect at least one first scan image through the scanning device within the first time period. When the scanning device acquires the first scan image, the processor can also collect the respiratory amplitude points corresponding to the target object within the first time period through the respiratory gating device, and each first scan image corresponds to a respiratory amplitude point.

[0053] In some embodiments, the processor acquires a first scanned image through a scanning device at a preset time in the first time period. For example, the processor acquires a first scanned image at a time specified by a user within the first time period. By acquiring the first scanned image at a preset time, the radiation dose of the target object caused by scanning can be reduced.

[0054] In some embodiments, the processor may collect multiple first scan images at preset respiratory amplitude points through a scanning device according to the respiratory amplitude points of the target object. For example, the processor collects a first scan image at every preset respiratory amplitude point change until the collection of a complete respiratory interval of the target object is completed.

[0055] In some embodiments, the processor may also continuously acquire multiple scan images of the target object through a scanning device within a first time period, divide the multiple scan images into multiple intervals, and select a scan image in each interval as the first scan image. The deformation of the target part of the target object in each of the aforementioned intervals is less than a preset deformation threshold. For example, the processor continuously acquires 30 scan images within the complete breathing interval of the target object in the first time period at a preset frequency. The processor may evenly divide the aforementioned 30 scan images into 10 intervals, and select the middle scan image in each interval as the first scan image. For example, the second scan image, the fifth scan image, the seventh scan image...are selected in turn as the first scan image.

[0056] Some embodiments of the present specification can display the breathing changes of the target object at different breathing amplitude points within the complete breathing interval through multiple first scanning images acquired by the aforementioned method, so that the deformation parameters determined later can more completely reflect the deformation of the target part of the target object within the breathing interval.

[0057] In some embodiments, when the processor acquires at least one first scan image, the respiratory pressure value and / or respiratory amplitude of the target object needs to meet a preset condition. Wherein, the preset condition refers to a pre-set condition that triggers the processor to acquire the first scan image. For example, the preset condition may include a respiratory pressure value greater than a first pressure value and less than a second pressure value. The preset condition may also include other conditions. For example, the preset condition may also include a respiratory amplitude greater than a first amplitude and less than a second amplitude. The first pressure value, the second pressure value, the first amplitude and the second amplitude may be default values, preset values, etc. As an example only, the preset condition may include a respiratory pressure value greater than 1.375 kpa and less than 1.782 kpa.

[0058] In some embodiments, when the respiratory pressure value and / or respiratory amplitude of the target object meets the preset conditions, when the processor collects at least one first scan image, the respiratory amplitude point of the target object should be within the target respiratory amplitude range. The target respiratory amplitude range refers to the range that contains the target respiratory amplitude point. The upper and lower bounds of the range can be determined based on the maximum and minimum values ​​of the respiratory amplitude points of the target object within the first time period when the processor collects the first scan image. For example, the processor can determine the maximum value of the respiratory amplitude point of the target object within the first time period when the first scan image is collected as the upper bound of the target respiratory amplitude range. Just as an example, when the respiratory pressure value and / or respiratory amplitude of the target object meets the preset conditions, when at least one first scan image is collected within the first time period t1 to t2, the respiratory amplitude points of the target object collected by the processor can be as follows Figure 4 As shown, when collecting the preoperative scan image, the target respiratory amplitude point of the target object is A0, and in the first time period t1-t2, the maximum value of the respiratory amplitude point of the target object is A1, and the minimum value is A2, then the target respiratory amplitude interval is [A1, A2]. Among them, the target respiratory amplitude point A0 is within the target respiratory amplitude interval [A1, A2].

[0059] In some embodiments of the present specification, the target respiratory amplitude point is included in the target respiratory amplitude range, so that the preoperative scan image and the first scan image can be aligned. In the subsequent interventional surgery, the intervention guidance can be completed as long as the target object's respiratory amplitude point is within the target respiratory amplitude range, thereby avoiding multiple scans of the target object in order to find a specific respiratory amplitude point during the surgery, which may cause the target object to receive excessive radiation dose.

[0060] In some embodiments, the target respiratory amplitude interval includes at least one respiratory amplitude interval segment. Each respiratory amplitude interval segment contains a corresponding respiratory amplitude point, and within the respiratory amplitude interval segment, the deformation of the target part of the target object is less than a preset deformation threshold. The aforementioned at least one respiratory amplitude interval segment may fully or partially cover the target respiratory amplitude interval. The processor may determine the respiratory amplitude interval segment where the respiratory amplitude point is located based on the respiratory amplitude point corresponding to a first scan image, thereby determining the respiratory amplitude interval segment corresponding to the first scan image. The second time period refers to the time period when the operation is performed. It is understandable that the second time period is later than the first time period.

[0061] In some embodiments, the second scan image is an image acquired in real time from the target site during the execution of the surgery.

[0062] In some embodiments, the processor may continuously scan the target site by means of a scanning device within a second time period, and perform image reconstruction to obtain at least one second scan image.

[0063] In some embodiments, during the second time period, the processor can collect the respiratory amplitude points corresponding to the target object in real time through the respiratory gating device, and based on the respiratory amplitude parameters corresponding to each first scan image, collect the second scan image at the same respiratory amplitude parameters. Wherein, the respiratory amplitude parameter is a parameter related to the respiratory information of the target object. In some embodiments, the respiratory amplitude parameter is a respiratory amplitude point or a respiratory amplitude interval to which the respiratory amplitude point belongs. Wherein, the deformation of each respiratory amplitude point of the target part of the target object in the aforementioned respiratory amplitude interval is less than a preset deformation threshold. For example, when there is only one first scan image, the processor can collect the target object based on the respiratory amplitude parameter corresponding to the first scan image, and obtain at least one second scan image to ensure that there are deformation parameters corresponding to the second scan image. For more information about the deformation parameters, please refer to the relevant content below in this specification. It can be understood that the number of first scan images does not need to correspond to the number of second scan images. For example, there may be only one first scan image, but the processor can collect multiple second scan images in the respiratory amplitude point or respiratory amplitude interval corresponding to the aforementioned first scan image through the scanning device.

[0064] It can be understood that when there are multiple first scanning images, and the breathing amplitude parameters (such as breathing amplitude interval segments) corresponding to the aforementioned multiple first scanning images are sufficient to cover the complete breathing interval of the target object, the processor can collect the second scanning image at any time point within the second time period, and the aforementioned second scanning images collected at any time point have corresponding deformation parameters.

[0065] In order to accurately reflect the deformation of the lesion area during the execution of the operation, and to reduce the radiation dose of the target object during the operation, the second scanned image can be an image obtained by scanning the target part based on a smaller scanning range (for example, 20mm) and reconstructing the image. It should be understood that in order to be able to determine multiple target interventional guidance images later, so as to better guide the interventional surgery, the processor can collect multiple second scanned images within the second time period. It should be understood that the second scanned image not only includes the target part, but also includes surgical instruments. However, since the second scanned image is an image acquired based on a smaller scanning range, the surgical instruments may not be fully displayed in the second scanned image, and the distribution of organ tissues around the lesion area cannot be reflected, so it cannot be directly used for interventional guidance of the operation.

[0066] It should be understood that in order to perform real-time interventional guidance during the interventional procedure, the rate at which the processor continuously scans the target site and reconstructs it in real time through the scanning device needs to be greater than the rate threshold. The rate threshold may be a default value (e.g., 10fps). Figure 5 As shown, in order to perform real-time intervention guidance within the time period T, the second scan image needs to be updated at a rate greater than the rate threshold. Image update includes scanning of the target part, image reconstruction, image registration of the second scan image with the preoperative scan image, and image transmission.

[0067] In the second time period, when the processor acquires a plurality of second scanned images, it can prompt the target object that the respiratory amplitude point should be within the target respiratory amplitude range. There are many ways to prompt, such as voice prompts. For example, Figure 6 As shown, during the interventional surgery performed in the second time period t3-t4, the target object's respiratory amplitude point should be within the target respiratory amplitude range [A1, A2] and as close to the target respiratory amplitude point A0 as possible.

[0068] Some embodiments of the present specification can reduce the deformation difference of tissues and organs caused by respiratory movement between the preoperative scan image and the second scan image by making the target object's respiratory amplitude points as close to the target respiratory amplitude points as possible within the second time period, which is beneficial to improving the subsequent registration accuracy of the preoperative scan image and the second scan image.

[0069] Step 220: determining at least one deformation parameter based on the pre-operative scan image and the at least one first scan image.

[0070] The deformation parameter refers to a parameter related to the registration of the target part in the preoperative scan image and the target part in the first scan image. For example, the deformation parameter may include an elastic registration parameter and a rigid registration parameter, etc. Among them, the registration of the target part in the preoperative scan image and the target part in the first scan image may include elastic registration, rigid registration, etc. Elastic registration refers to the registration of image deformation. Rigid registration refers to the registration of translation and rotation of the coordinate axis. As an example only, rigid registration may be a process in which the target part in the first scan image is set as a reference image, and the target part in the preoperative scan image is spatially transformed according to the reference image (for example, after a series of rotations, translations, etc.), so that the corresponding pixel points on the reference image reach the same spatial position. Accordingly, the rigid registration parameter may reflect the spatial transformation relationship between the two images (for example, rotation angle, translation distance, etc.). In some embodiments, the rigid registration parameter may include parameters such as horizontal displacement, vertical translation, rotation angle, etc. in the rigid registration process. In some embodiments, the rigid registration parameter may be represented in the form of a numerical value, a vector, a matrix, etc. Elastic registration can be the process of spatially transforming the target part with irregular internal deformation in the preoperative scan image according to the target part in the first scan image. Accordingly, the elastic registration parameters can refer to the transformation parameters corresponding to the part of the image that has deformed when the medical image is registered (for example, the geometric shape change of the organ tissue or target area of ​​radiotherapy). Radiotherapy or the spontaneous movement of the organ will cause the internal tissue and organ to change in size, shape, volume, etc., so that the collected medical image has some irregular internal deformation. For example, with the change of breathing, the shape and size of the target part of the target object will gradually change.

[0071] The processor may process the preoperative scan image and the at least one first scan image based on the registration software to determine at least one deformation parameter.

[0072] In some embodiments, the processor can process the preoperative scan image and the at least one first scan image based on the registration deep learning model to determine at least one deformation parameter.

[0073] The registration deep learning model may be a machine learning model for determining deformation parameters. For example, the registration deep learning model may be a deep neural network (DNN) model.

[0074] It should be understood that in order to achieve real-time scanning, image reconstruction and image transmission of the target part, the registration deep learning model can be a lightweight model (for example, a simplified network structure), and the processor can also quantize, prune, and perform knowledge distillation on the registration deep learning model to ensure that the registration deep learning model can achieve real-time processing of the preoperative scan image and at least one first scan image. In order to further improve the computing speed and processing performance of the registration deep learning model, high-performance graphics processing hardware (GPU) can be used when deploying the registration deep learning model.

[0075] In some embodiments, the input of the registration deep learning model may include a preoperative scan image and a first scan image; the output may include deformation parameters corresponding to the respiratory amplitude point of the first scan image. When there are multiple first scan images, the registration deep learning model may determine the deformation parameters corresponding to the respiratory amplitude point of each first scan image by inputting the preoperative scan image and different first scan images multiple times.

[0076] In some embodiments, the processor can determine the respiratory amplitude interval corresponding to each respiratory amplitude point, and determine the deformation parameter corresponding to the respiratory amplitude point of a first scan image determined above as the deformation parameter of the respiratory amplitude interval corresponding to the respiratory amplitude point. Wherein, each respiratory amplitude interval contains a corresponding respiratory amplitude point, and within the respiratory amplitude interval, the deformation of the target part of the target object is less than a preset deformation threshold. When a certain respiratory amplitude interval corresponds to a deformation parameter determined based on multiple first scan images, the average of the aforementioned multiple deformation parameters can be determined as the deformation parameter corresponding to the respiratory amplitude interval.

[0077] In some embodiments, the registration deep learning model can be obtained by training multiple first training samples with first labels. For example, multiple first training samples with first labels can be input into the initial registration deep learning model, the value of the loss function is constructed by the first label and the result of the initial registration deep learning model, and the parameters of the initial registration deep learning model are iteratively updated based on the loss function. When the loss function of the initial registration deep learning model meets the iterative preset conditions, the model training is completed and the trained registration deep learning model is obtained. Among them, the iterative preset conditions are that the loss function converges, the number of iterations reaches a threshold, etc.

[0078] In some embodiments, the first training sample includes a sample preoperative scan image and multiple sample first scan images of different sample target objects; the first label includes a sample deformation parameter corresponding to each sample first scan image. In some embodiments, the first training sample can be obtained based on historical data (e.g., a historical preoperative scan image and multiple historical first scan images); the first label can be determined based on manual registration processing of the sample preoperative scan image and the multiple sample first scan images.

[0079] In some embodiments of the present specification, by adopting a lightweight registration deep learning model and performing quantization, pruning, knowledge distillation and other processes on it, not only can real-time processing of preoperative scan images and multiple first scan images be achieved, avoiding occupying more memory and improving calculation speed and processing performance, but also the influence of multiple factors can be considered at the same time, so that the determination of deformation parameters is efficient and accurate, avoiding errors caused by manual determination.

[0080] When the processor only acquires one first scan image within the first time period, the processor can determine a deformation parameter based on the first scan image and the preoperative scan image, and store the correspondence between the respiratory amplitude point or respiratory amplitude interval of the first scan image and the deformation parameter to the storage device.

[0081] In some embodiments of the present specification, only one first scanning image is collected for registration to determine the deformation parameters corresponding to a certain respiratory amplitude point or respiratory amplitude interval segment, which is beneficial to reducing the amount of calculation during early registration and guidance in later interventional surgery, shortening the early preparation time for interventional surgery, and quickly entering interventional surgery.

[0082] When the processor acquires multiple first scan images at different respiratory amplitude points within the first time period, the processor can determine a deformation parameter based on each first scan image and the preoperative scan image, and store the corresponding relationship between the respiratory amplitude point or respiratory amplitude interval of each first scan image and the deformation parameter to the storage device, so as to determine the corresponding deformation parameter directly based on the respiratory amplitude point or respiratory amplitude interval corresponding to the second scan image. For example, the respiratory amplitude points corresponding to the multiple first scan images are C0, D0, and E0, respectively, and the corresponding deformation parameters are X0, X1, and X2, respectively. C , X D , X E When the processor acquires a second scan image, the respiratory amplitude point of the target object is C0, then the deformation parameter corresponding to the second scan image is X C For another example, the respiratory amplitude points corresponding to the first scan images are C0, D0, and E0, and the corresponding deformation parameters are X C , X D , X EIf the respiratory amplitude intervals corresponding to the first scanned images are [C1, C2], [D1, D2], and [E1, E2], respectively, and the respiratory amplitude point C0 is within the respiratory amplitude interval [C1, C2], then the deformation parameter corresponding to the respiratory amplitude interval [C1, C2] is X C ; If the breathing amplitude point D0 is within the breathing amplitude interval [D1, D2], then the deformation parameter corresponding to the breathing amplitude interval [D1, D2] is X D ; If the breathing amplitude point E0 is within the breathing amplitude interval [E1, E2], then the deformation parameter corresponding to the breathing amplitude interval [E1, E2] is X E If the processor acquires a second scan image, the target object's respiratory amplitude point is C3, and the respiratory amplitude point C3 is within the respiratory amplitude interval [C1, C2], then the deformation parameter corresponding to the second scan image is X C .

[0083] In some embodiments of the present specification, collecting multiple first scanning images for registration can determine the deformation parameters corresponding to multiple different respiratory amplitude points or respiratory amplitude intervals, which is beneficial to improving the update frequency of the target interventional guidance image during subsequent surgical execution, ensuring the accuracy of image registration, and making the determined target interventional guidance image more accurate.

[0084] Step 230: Determine at least one target intervention guidance image for a second time period based on at least one deformation parameter, the preoperative scan image, and at least one second scan image.

[0085] The target intervention guidance image refers to a real-time image used to guide interventional surgery. For example, the target intervention guidance image may include surgical instruments, lesions, target organs, etc. The target intervention guidance image may be a real-time image used to guide surgical instruments to perform intervention during the execution of the surgery.

[0086] The processor may determine at least one target intervention guidance image of the second time period based on at least one deformation parameter, the preoperative scan image and the at least one second scan image through image fusion software or other suitable methods.

[0087] In some embodiments, when there is only one deformation parameter X0, the processor can determine the respiratory amplitude point A3 or the respiratory amplitude interval [A4, A5] corresponding to the deformation parameter X0, and the respiratory amplitude point A3 is within the respiratory amplitude interval [A4, A5]. During the operation, when the respiratory amplitude point corresponding to the target object is the respiratory amplitude point A3 or within the respiratory amplitude interval [A4, A5], the processor acquires at least one second scan image and determines the deformation parameter corresponding to the aforementioned at least one second scan image as the deformation parameter X0. After the processor deforms the preoperative scan image based on the deformation parameter X0, it fuses it with the aforementioned at least one second scan image to determine the target interventional guidance image corresponding to the aforementioned at least one second scan image. Some embodiments of the present specification can reduce the amount of calculation during guidance during the interventional surgery by guiding the interventional surgery by determining the target interventional guidance image only within the respiratory amplitude point or respiratory amplitude interval corresponding to one deformation parameter.

[0088] In some embodiments, when there are multiple deformation parameters, the processor obtains the respiratory amplitude parameter corresponding to each deformation parameter. During the operation, when the target object is at the same respiratory amplitude parameter, the processor collects at least one second scan image. For each second scan image, the processor can determine the deformation parameter corresponding to the second scan image based on the respiratory amplitude parameter corresponding to the second scan image, and use the corresponding deformation parameters to deform the preoperative scan image respectively, and then fuse it with the second scan image to determine the target intervention guidance image corresponding to the second scan image. As an example only, the deformation parameter corresponding to the respiratory amplitude interval [C1, C2] is X C , the deformation parameter corresponding to the breathing amplitude interval [D1, D2] is X D , the deformation parameter corresponding to the breathing amplitude interval [E1, E2] is X E If the respiratory amplitude points corresponding to the plurality of second scanned images F1, F2, and F3 acquired by the processor are C3, D3, and E3 respectively, and the respiratory amplitude point C3 is within the respiratory amplitude interval [C1, C2], the respiratory amplitude point D3 is within the respiratory amplitude interval [D1, D2], and the respiratory amplitude point E3 is within the respiratory amplitude interval [E1, E2], then the deformation parameters corresponding to the second scanned images F1, F2, and F3 are X respectively. C , X D , X E The processor is based on the deformation parameter X C , X D , X EThe preoperative scan image G0 is deformed to obtain deformed preoperative scan images G1, G2, and G3, respectively. The deformed preoperative scan images G1, G2, and G3 are fused with the corresponding second scan images F1, F2, and F3 to obtain target intervention guidance images H1, H2, and H3. It can be understood that when multiple deformation parameters cover the complete respiratory interval of the target object, the user can adjust the acquisition frequency of the second scan image (for example, 30Hz) as needed, so that the target intervention guidance image can be continuously output to guide the interventional surgery in real time. Some embodiments of the present specification can ensure the output frequency of the target intervention guidance image through multiple deformation parameters, thereby ensuring the real-time and accuracy of the intervention guidance.

[0089] In some embodiments, the target interventional guidance image includes a first interventional guidance image and / or a second interventional guidance image.

[0090] The first intervention guidance image includes a plurality of slices of scanned images related to at least one second scanned image and a preoperative scanned image. For example, the first intervention guidance image may include a plurality of slices of two-dimensional images of a target part of a target object, and the user may adjust the display of the two-dimensional images of different slices of the target part as needed. As an example only, Fig.10 As shown, the first interventional guidance image 1000 is a two-dimensional image of a certain slice of the target part, and the first interventional guidance image 1000 is a two-dimensional image obtained by fusing a preoperative scan image 1010 after deformation processing of the slice with a second scan image 1020 of the slice.

[0091] The second interventional guidance image includes a three-dimensional model related to at least one second scan image and a preoperative scan image. For example, the second interventional guidance image may be a three-dimensional model including surgical instruments and related parts, wherein the aforementioned related parts may include a target part of a target object and surrounding tissues and organs. Fig.11 As shown, the second intervention guidance image 1100 is a three-dimensional model including a surgical instrument 1120 and a related part 1110 .

[0092] In some embodiments of this specification, the target interventional guidance image is divided into a first interventional guidance image and a second interventional guidance image, and one or more of the images can be displayed according to the needs of the user, so as to meet the operation habits of different users. It should be understood that the target interventional guidance image determined in some embodiments of this specification is only used in the interventional surgery process to provide guidance for the operation of the user or doctor.

[0093] In some embodiments, for each second scan image, the processor can obtain the respiratory amplitude parameter corresponding to the target object when the second scan image is acquired; based on the deformation parameter corresponding to the respiratory amplitude parameter, the preoperative scan image is deformed, and image fused with the second scan image to determine the first intervention guidance image at the time corresponding to the second scan image. For more information about the above embodiments, please refer to Figure 7 and its related description.

[0094] In some embodiments, for each second scan image, the processor can segment the second scan image to determine the surgical instrument image in the second scan image; obtain the respiratory amplitude parameter corresponding to the target object when the second scan image is acquired; based on the deformation parameter corresponding to the respiratory amplitude parameter, segment, three-dimensionally reconstruct and deform the preoperative scan image to obtain the initial model corresponding to the preoperative scan image after deformation processing; based on the surgical instrument image and the initial model, determine the second interventional guidance image at the corresponding moment of the second scan image. For more information about the above embodiments, please refer to Figure 7 and its related description.

[0095] In some embodiments of the present specification, image registration is completed during the preparation time before the interventional surgery is performed, and real-time mapping of the preoperative scan image to the second scan image is performed during the surgery. The preoperative scan image obtained by a large-range scan before the surgery is fused with the second scan image obtained by a narrow-range scan during the surgery, and dynamic updates are performed at the same time. This achieves real-time tracking of organ deformation and a large field of view for surgical guidance, thereby improving the sense of space during the interventional process, controlling the radiation dose, and improving the safety of the surgery.

[0096] Figure 7 FIG. 700 is an exemplary flow chart of determining a first intervention guidance image according to some embodiments of the present specification. In some embodiments, process 700 may be executed by a processor. Figure 7 As shown, process 700 may include the following steps.

[0097] Step 710: For each second scan image, obtain a respiratory amplitude parameter corresponding to the target object when the second scan image is acquired.

[0098] For more information about the respiratory amplitude parameter, see Figure 2 and its related description.

[0099] In some embodiments, when the scanning device collects multiple second scanned images, the processor can also collect the respiratory amplitude points corresponding to the target object in the second time period through the respiratory gating device, and each second scanned image corresponds to a respiratory amplitude point. The processor can determine the respiratory amplitude interval segment where the respiratory amplitude point is located based on the respiratory amplitude point corresponding to a second scanned image, thereby determining the respiratory amplitude interval segment corresponding to the second scanned image.

[0100] Step 720 , deforming the preoperative scan image based on the deformation parameter corresponding to the respiratory amplitude parameter to obtain the deformed preoperative scan image.

[0101] In some embodiments, for each second scan image, the processor can access the storage device based on the respiratory amplitude parameters corresponding to the second scan image, determine the deformation parameters corresponding to the second scan image through the correspondence between the respiratory amplitude parameters and the deformation parameters, perform elastic deformation processing and / or rigid deformation processing on the target part in the preoperative scan image, and obtain the preoperative scan image after deformation processing.

[0102] Step 730: perform image fusion on the deformed preoperative scan image and the second scan image to determine the first intervention guidance image at the time corresponding to the second scan image.

[0103] For more information about the first interventional guidance image, see Figure 2 and its related description.

[0104] In some embodiments, the processor can fuse the target part in the deformed preoperative scan image with the target part in the second scan image to determine the first interventional guidance image when the second scan image is acquired within the second time period.

[0105] During the interventional surgery, the preparation time before the interventional surgery is used to complete image registration, and the deformation parameters corresponding to each respiratory amplitude parameter are saved according to the respiratory amplitude point of the target object. During the surgery, the deformation parameters of the respiratory amplitude parameters corresponding to the respiratory amplitude point of the target object at the moment of collecting the second scanning image are directly used to deform the preoperative scanning image, and the image is fused with the second scanning image to obtain the interventional guidance image when the second scanning image is collected, thereby reducing the amount of calculation during image registration and image fusion and ensuring the real-time nature of the interventional guidance image.

[0106] In some embodiments, for each second scan image, the processor can segment the second scan image to determine the surgical instrument image in the second scan image; obtain the respiratory amplitude parameter corresponding to the target object when the second scan image is acquired; based on the deformation parameter corresponding to the respiratory amplitude parameter, segment, three-dimensionally reconstruct and deform the preoperative scan image to obtain the initial model corresponding to the preoperative scan image after deformation processing; based on the surgical instrument image and the initial model, determine the second interventional guidance image at the corresponding moment of the second scan image. For more information about the second interventional guidance image, please refer to Figure 2 and its related description.

[0107] In some embodiments, for each second scan image, the processor can process the second scan image through image segmentation software to obtain a surgical instrument image.

[0108] In some embodiments, for each second scan image, the processor can process the second scan image based on the segmentation deep learning model to determine the surgical instrument image in the second scan image. Fig. 9 As shown, the processor can segment the deep learning model based on Fig. 9 The second scanned image shown is processed and segmented to obtain an image of surgical instrument B.

[0109] The segmentation deep learning model can be a machine learning model for determining surgical instruments. For example, the segmentation deep learning model can be a deep neural network (DNN) model.

[0110] It should be understood that in order to achieve real-time segmentation of the second scanned image to obtain the surgical instrument image, the segmentation deep learning model can be a lightweight model (for example, a simplified network structure), and the processor can also quantize, prune, and perform knowledge distillation on the segmentation deep learning model to ensure that the segmentation deep learning model can achieve real-time processing of multiple second scanned images. In order to further improve the computing speed and processing performance of the segmentation deep learning model, high-performance image processing hardware can be used when deploying the segmentation deep learning model.

[0111] In some embodiments, the input of the segmentation deep learning model can include a second scan image; the output can include an image of the surgical instrument in the second scan image.

[0112] It should be understood that the processor can process the second scanning images of different slices separately through the segmentation deep learning model to obtain the surgical instrument images in the second scanning images of different slices, and perform three-dimensional reconstruction of the surgical instrument images to obtain a three-dimensional surgical instrument image.

[0113] In some embodiments, the segmentation deep learning model can be trained by a plurality of second training samples with second labels. The training method of the segmentation deep learning model is similar to the training method of the registration deep learning model. For the training of the segmentation deep learning model, please refer to Figure 2 On the training of deep learning models for registration.

[0114] In some embodiments, the second training sample can include a sample second scanned image of a sample target object; the second label can include a sample image of a surgical instrument. In some embodiments, the second training sample can be acquired based on historical data (e.g., a historical second scanned image); the second label can be determined based on manually segmenting the sample second scanned image.

[0115] In some embodiments of the present specification, by adopting a lightweight segmentation deep learning model and performing quantization, pruning, knowledge distillation and other processes on it, not only can real-time processing of multiple second scan images be achieved, avoiding occupying more memory and improving computing speed and processing performance, but also the influence of multiple factors can be considered at the same time, so that the segmented surgical instrument images are more efficient and accurate, avoiding errors determined manually.

[0116] In some embodiments, the processor can segment the organ tissues of different slices of preoperative scan images based on the segmentation deep learning model, and perform three-dimensional reconstruction based on the organ tissues segmented from all slices to obtain an initial model corresponding to the preoperative scan image.

[0117] In some embodiments, the processor can perform deformation processing on the initial model corresponding to the preoperative scan image based on the deformation parameters corresponding to the second scan image to obtain the initial model corresponding to the preoperative scan image after the deformation processing.

[0118] It should be understood that the processor can perform segmentation, three-dimensional reconstruction, and deformation processing of the preoperative scan image in other orders. For example, the processor can perform deformation processing on the preoperative scan images of different slices based on the deformation parameters corresponding to the second scan image to obtain the preoperative scan images of different slices after deformation processing. Furthermore, the processor can segment the organ tissues of the preoperative scan images of different slices after deformation processing based on the segmentation deep learning model, and perform three-dimensional reconstruction based on the organ tissues obtained by segmentation of all slices to obtain the initial model corresponding to the preoperative scan image after deformation processing.

[0119] In some embodiments, the processor can determine the second intervention guidance image at the time corresponding to the second scan image by a formula based on the initial model corresponding to the surgical instrument image and the deformed preoperative scan image. The formula for determining the second intervention guidance image can be V Curent =F Curent +DVFA’ (F Before ), where V Curent is the second intervention guidance image at the corresponding moment of the second scan image, F Curent is the surgical instrument image in the second scanned image, and F Before is the initial model corresponding to the preoperative scan image, DVF A’ (F Before ) is the initial model corresponding to the preoperative scan image after deformation processing. That is, for each second scan image F Curent , through its corresponding respiratory amplitude parameter, determine the corresponding deformation parameter, and use the deformation parameter to transform the initial model F corresponding to the preoperative scan image Before Perform deformation processing to obtain the initial model DVF corresponding to the preoperative scan image after deformation processing A’ (F Before ), and fuse it with the surgical instrument image in the second scan image to obtain a second intervention guidance image V corresponding to the time when the second scan image is collected Curent .

[0120] In some embodiments, by segmenting the second scanned image, a real-time image of the surgical instrument during the interventional surgery can be obtained, which can effectively reflect the deformation of the surgical instrument. The tissues and organs of the preoperative scanned images of different slices are segmented and reconstructed in three dimensions to obtain the initial model corresponding to the preoperative scanned image, and the image is fused with the segmented surgical instrument image after deformation processing, which not only makes the determined interventional guidance image more accurate and better reflects the actual situation of the target part during the surgery, but also allows the second interventional guidance image to be presented in three dimensions, with a stronger sense of three-dimensionality and space.

[0121] Figure 8 FIG. 800 is an exemplary flow chart of determining an updated target intervention guidance image according to some embodiments of the present specification. In some embodiments, process 800 may be executed by a processor. Figure 8 As shown, process 800 may include the following steps.

[0122] Step 810: for each target intervention guidance image, determine whether the surgical instrument is completely displayed in the target intervention guidance image.

[0123] Surgical instruments refer to instruments used in interventional surgery. For example, surgical instruments may include but are not limited to interventional needles, drainage tubes, etc.

[0124] It should be understood that, since the second scan image is acquired based on a smaller scan range, the complete surgical instrument may not be displayed in the second scan image, or there may even be no surgical instrument in the second scan image.

[0125] In some embodiments, the processor can process the second scan image corresponding to the target intervention guidance image through image recognition software to determine whether the surgical instrument is completely displayed in the target intervention guidance image.

[0126] In some embodiments, for each target interventional guidance image, the processor can determine the surgical instrument image in the second scanned image corresponding to the target interventional guidance image; and determine whether the surgical instrument is fully displayed in the target interventional guidance image based on the surgical instrument image and the standard surgical instrument image.

[0127] In some embodiments, the processor can process the second scan image corresponding to the target intervention guidance image based on the segmentation deep learning model to determine the surgical instrument image in the second scan image. For more information about the segmentation deep learning model, please refer to Figure 7 and its related description.

[0128] The standard surgical instrument image refers to a complete image of a surgical instrument used as a reference. The standard surgical instrument image can be pre-stored in a storage device.

[0129] In some embodiments, the processor can obtain a standard surgical instrument image based on the surgical instrument image segmented from the second scanned image corresponding to the interventional guidance image by accessing a storage device, and compare the surgical instrument image with the standard surgical instrument image to determine whether the segmented surgical instrument image is a complete surgical instrument, thereby determining whether the surgical instrument is completely displayed in the target interventional guidance image.

[0130] In some embodiments of the present specification, based on the surgical instrument image obtained by segmenting the second scanned image during the aforementioned image fusion, it is compared with a pre-stored standard surgical instrument image, which can not only determine whether the surgical instrument is completely displayed in the target intervention guidance image, but also reduce the computational complexity of the processor and shorten the update time of the target intervention guidance image.

[0131] In some embodiments, for each target interventional guidance image, the processor can determine the second spatial position of the surgical instrument in the scanning image coordinate system based on the first spatial position of the surgical instrument in the surgical robot coordinate system at the moment corresponding to the target interventional guidance image; obtain the third spatial position of the second scanning image corresponding to the target interventional guidance image in the scanning image coordinate system; and based on the second spatial position and the third spatial position, determine whether the surgical instrument is completely displayed in the target interventional guidance image.

[0132] The first spatial position refers to a parameter related to the position of the surgical instrument in the surgical robot coordinate system. For example, the first spatial position may include the three-dimensional coordinates of the starting point and the end point of the surgical instrument in the surgical robot coordinate system.

[0133] In some embodiments, the processor can obtain the first spatial position of the surgical instrument in the coordinate system of the surgical robot at the moment corresponding to the interventional guidance image by accessing the storage device of the surgical robot.

[0134] The second spatial position refers to a parameter related to the position of the surgical instrument in the scanning image coordinate system.

[0135] In some embodiments, the processor can convert a first spatial position of the surgical instrument in the surgical robot coordinate system into a second spatial position in the scanning image coordinate system based on a mapping relationship between the surgical robot coordinate system and the scanning image coordinate system.

[0136] The third spatial position refers to a parameter related to the position of the second scanned image in the scanned image coordinate system. For example, the third spatial position may include the three-dimensional coordinates of the boundary contour of the second scanned image in the scanned image coordinate system.

[0137] In some embodiments, the processor can determine whether the surgical instrument is completely displayed in the target interventional guidance image by comparing whether the second spatial position of the surgical instrument is entirely within the coverage of the third spatial position of the second scanned image. When the second spatial position of the surgical instrument is entirely within the coverage of the third spatial position of the second scanned image, the processor determines that the surgical instrument is completely displayed in the target interventional guidance image; when the second spatial position of the surgical instrument is partially within the coverage of the third spatial position of the second scanned image, the processor determines that the surgical instrument is partially displayed in the target interventional guidance image; when the second spatial position of the surgical instrument is not within the coverage of the third spatial position of the second scanned image, the processor determines that the surgical instrument is not displayed in the target interventional guidance image.

[0138] In some embodiments of the present specification, the second spatial position of the surgical instrument in the scanning image coordinate system is mapped through the first spatial position of the surgical instrument in the surgical robot coordinate system, and the display situation of the surgical instrument in the target intervention guidance image is judged based on the third spatial position of the second scanning image in the scanning image coordinate system. This is not only applicable to the situation where the surgical instrument is not displayed completely, but also to the situation where the surgical instrument is not displayed at all.

[0139] In some embodiments, if the surgical instrument image segmented in the second scan image completely displays the entire surgical instrument, the processor can directly merge the segmented surgical instrument image into the preoperative scan image after deformation processing to determine the target intervention guidance image at the corresponding moment of acquiring the second scan image.

[0140] Step 820, in response to "no", determine the time corresponding to the target intervention guidance image, and obtain, through the surgical robot, the first spatial position of the surgical instrument in the surgical robot coordinate system at that time.

[0141] In some embodiments, the processor can obtain the first spatial position of the surgical instrument in the coordinate system of the surgical robot at the moment corresponding to the target intervention guidance image by accessing the storage device of the surgical robot.

[0142] Step 830 : Map the surgical instrument to the target intervention guidance image based on the first spatial position, and determine an updated target intervention guidance image.

[0143] In some embodiments, the processor can convert the first spatial position of the surgical instrument in the surgical robot coordinate system into the second spatial position in the scanning image coordinate system based on the mapping relationship between the surgical robot coordinate system and the scanning image coordinate system, and fuse the pre-stored standard surgical instrument image into the second scanning image based on the second spatial position of the surgical instrument in the scanning image coordinate system. It should be understood that since the surgical instrument will deform during the operation, the real-time image of the surgical instrument is different from the standard surgical instrument image pre-stored in the storage device. Therefore, after determining the second spatial position of the surgical instrument in the scanning image coordinate system, if the surgical instrument is not displayed at all in the target interventional guidance image, the processor can directly map the pre-stored standard surgical instrument image to the target interventional guidance image based on the second spatial position to determine the updated target interventional guidance image. If the surgical instrument is partially but not completely displayed in the target interventional guidance image, the pre-stored standard surgical instrument image cannot be directly replaced with the surgical instrument image segmented from the second scanning image. The processor needs to supplement the missing part of the surgical instrument image segmented from the second scanning image based on the standard surgical instrument image to determine the updated interventional guidance image. For more information about image fusion, please refer to Figure 7 and its related description.

[0144] It is worth noting that, since the surgical instrument image may be deformed during the interventional surgery, if the pre-stored complete surgical instrument image is directly used, the deformation of the surgical instrument will not be reflected in the interventional guidance image. However, some embodiments of this specification perform real-time segmentation on the second scan image to obtain the surgical instrument image corresponding to each second scan image and fuse it with the deformed pre-operative scan image, taking into account the deformation of the surgical instrument, and the obtained interventional guidance image is more accurate. When the segmented surgical instrument image is incomplete or not displayed in the interventional guidance image, some embodiments of this specification fuse the pre-stored standard surgical instrument image into the second scan image through the first spatial position of the surgical instrument in the surgical robot coordinate system to obtain the interventional guidance image, thereby ensuring the integrity of the interventional guidance image during the interventional surgery.

[0145] It should be noted that the above descriptions of process 200, process 700 and process 800 are only for illustration and description, and do not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to process 200, process 700 and process 800 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0146] The present specification also provides an intervention guidance device, which includes at least one processor and at least one memory; the at least one memory is used to store computer instructions; the at least one processor is used to execute at least part of the computer instructions to implement the intervention guidance method as described in any embodiment of the present specification.

[0147] The present specification also provides a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the intervention guidance method described in any embodiment of the present specification.

[0148] It should be understood that the aforementioned interventional guidance system, method, device and storage medium can be applied not only to CT guidance, but also to other guidance, such as magnetic resonance guidance.

[0149] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0150] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0151] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0152] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0153] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0154] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0155] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. An intervention guidance system, characterized in that: The system comprises: an image scanning module, an image registration module and an image mapping module; The image scanning module is used to obtain a preoperative scan image of the target object, at least one first scan image of a first time period, and at least one second scan image of a second time period, wherein the first time period is earlier than the second time period; The image registration module is used to determine at least one deformation parameter based on the preoperative scan image and the at least one first scan image; The image mapping module is used to determine at least one target intervention guidance image of the second time period based on the at least one deformation parameter, the pre-operative scan image and the at least one second scan image.

2. The system according to claim 1, characterized in that The target interventional guidance image includes a first interventional guidance image and / or a second interventional guidance image, wherein the first interventional guidance image includes a multi-layer scan image related to the at least one second scan image and the preoperative scan image, and the second interventional guidance image includes a three-dimensional model related to the at least one second scan image and the preoperative scan image.

3. The system according to claim 2, characterized in that Each of the at least one deformation parameter corresponds to a breathing amplitude parameter, and the image mapping module is further used for: For each second scan image, obtaining a respiratory amplitude parameter corresponding to the target object when the second scan image is acquired; Performing deformation processing on the preoperative scan image based on the deformation parameter corresponding to the respiratory amplitude parameter to obtain the preoperative scan image after deformation processing; The deformed preoperative scan image is fused with the second scan image to determine the first intervention guidance image at a time corresponding to the second scan image.

4. The system according to claim 2, characterized in that The image mapping module is further used for: For each second scan image, segment the second scan image to determine a surgical instrument image in the second scan image; Acquiring a respiratory amplitude parameter corresponding to the target object when acquiring the second scan image; Based on the deformation parameters corresponding to the respiratory amplitude parameters, segmenting, three-dimensionally reconstructing and deforming the preoperative scan image, and obtaining an initial model corresponding to the preoperative scan image after the deformation processing; Based on the surgical instrument image and the initial model, the second intervention guidance image at the time corresponding to the second scan image is determined.

5. The system according to claim 4, characterized in that The image mapping module is further used for: For each second scanned image, the second scanned image is processed based on a segmentation deep learning model to determine the surgical instrument image in the second scanned image, and the segmentation deep learning model is a machine learning model.

6. The system according to claim 1, characterized in that The image registration module is further used for: The preoperative scan image and the at least one first scan image are processed based on a registration deep learning model to determine the at least one deformation parameter, and the registration deep learning model is a machine learning model.

7. The system according to claim 1, characterized in that The image scanning module is further used for: When the respiratory pressure value and / or the respiratory amplitude of the target object meets a preset condition, acquiring the at least one first scan image, wherein when acquiring the at least one first scan image, the respiratory amplitude point of the target object is within a target respiratory amplitude range; The system also includes a prompt module, The prompting module is used to prompt that the breathing amplitude point of the target object is within the target breathing amplitude range during the second time period.

8. The system according to claim 1, characterized in that The image mapping module is also used to: For each target intervention guidance image, determining whether the surgical instrument is completely displayed in the target intervention guidance image; In response to no, determining a time corresponding to the target intervention guidance image, and acquiring, by the surgical robot, a first spatial position of the surgical instrument in the surgical robot coordinate system at that time; The surgical instrument is mapped to the target intervention guidance image based on the first spatial position, and an updated target intervention guidance image is determined.

9. The system according to claim 8, characterized in that The image mapping module is further used for: For each target intervention guidance image, based on the second scan image corresponding to the target intervention guidance image, determine a surgical instrument image in the second scan image; Based on the surgical instrument image and the standard surgical instrument image, it is determined whether the surgical instrument is completely displayed in the target intervention guidance image.

10. The system according to claim 8, characterized in that The image mapping module is further used for: For each target intervention guidance image, determining a second spatial position of the surgical instrument in the scanning image coordinate system based on the first spatial position of the surgical instrument in the surgical robot coordinate system at a time corresponding to the target intervention guidance image; Acquire a third spatial position of a second scan image corresponding to the target intervention guidance image in the scan image coordinate system; Based on the second spatial position and the third spatial position, it is determined whether the surgical instrument is completely displayed in the target intervention guidance image.

11. An intervention guidance method, characterized in that: The method comprises: Acquire a preoperative scan image of the target object, at least one first scan image of a first time period, and at least one second scan image of a second time period, wherein the first time period is earlier than the second time period; determining at least one deformation parameter based on the preoperative scan image and the at least one first scan image; At least one target intervention guidance image of the second time period is determined based on the at least one deformation parameter, the pre-operative scan image and the at least one second scan image.

12. An interventional guidance device, characterized in that: The apparatus comprises at least one processor and at least one memory; The at least one memory is used to store computer instructions; The at least one processor is configured to execute at least part of the computer instructions to implement the intervention guidance method as claimed in claim 11.

13. A computer-readable storage medium, characterized in that: The storage medium stores computer instructions. When the computer reads the computer instructions in the storage medium, the computer executes the intervention guidance method as claimed in claim 11.

Images