A registration method and apparatus

Abstract

The specification provides a registration method and device, the method comprising: acquiring a first image of a target part of a target object having surgical planning information; acquiring a second image in a surgical process, and acquiring a first coordinate set collected by a plurality of positioning sensors placed on the target part of the target object when the second image is taken; the second image including images of the plurality of positioning sensors; determining first registration information of the second image and the first image; determining a second coordinate set of the plurality of positioning sensors in a second image coordinate system; determining second registration information of the first coordinate set and the second coordinate set; and determining surgical planning information in a virtual endoscope field of view of the endoscope in a corresponding pose in the first image according to the first registration information and the second registration information. The scheme can automatically realize registration of an intraoperative real coordinate system to a preoperative image coordinate system, and acquire local surgical planning information, thereby reducing the amount of manual operation.

Description

Technical Field

[0001] This application relates to the field of medical device technology, and in particular to a registration method and apparatus. Background Technology

[0002] Bronchial navigation based on preoperative CT (Computed Tomography) scans is a novel method for assisting pulmonary nodule puncture. Compared to traditional percutaneous puncture, it offers less trauma. This is because the pathway planning is based on preoperative CT images, while the intraoperative image obtained is a real endoscopic image. Figure 1 As shown, A is the preoperative CT image and B is the intraoperative actual endoscopic image. Matching the coordinate systems of the two is a very important step.

[0003] The current common method is to obtain multiple marker points and their coordinates on preoperative CT images; then, in a real-world scenario, the doctor manually moves the endoscope to the position of the marker points to obtain the corresponding coordinates of the marker points in the real coordinate system, thereby performing point cloud registration.

[0004] This registration method has a low degree of automation, resulting in low registration efficiency and is largely affected by the doctor's skill level and experience. Summary of the Invention

[0005] The purpose of this application is to provide a registration method and apparatus to solve the problems of low registration efficiency and significant dependence on the doctor's skill level and experience in existing registration methods.

[0006] To address the aforementioned technical problems, this specification provides a registration device in a first aspect, comprising: a first acquisition unit for acquiring a first image of a target body part of a target object, the first image containing surgical planning information; a second acquisition unit for acquiring a second image of the target body part of the target object during surgery, and acquiring a first coordinate set collected by multiple positioning sensors placed on the target body part of the target object when the second image is captured; the second image includes images of the multiple positioning sensors; the first coordinate set is the coordinates of the positioning sensors themselves in a real coordinate system; a first determination unit for determining first registration information for registering the second image with the first image; a second determination unit for determining a second coordinate set of the multiple positioning sensors in the coordinate system of the second image; a third determination unit for determining second registration information for registering the first coordinate set and the second coordinate set; a third acquisition unit for acquiring the pose of an endoscope in a real coordinate system; and a fourth determination unit for determining, based on the first registration information and the second registration information, the corresponding pose of the endoscope in the first image, and determining surgical planning information in the virtual endoscopic field of view under the corresponding pose in the first image.

[0007] In some embodiments, the first determining unit inputs the second image and the first image into a pre-trained registration model to obtain first registration information output by the registration model; wherein, the registration model is trained by the following method: acquiring sample images, the sample images including a first sample image taken before surgery and a second sample image taken during surgery; and repeating the following operations until the loss function converges: inputting the first sample image and the second sample image into the registration model to obtain the first registration information; transforming the second sample image using the first registration information to obtain a third sample image; calculating the first dissimilarity between the first sample image and the third sample image, and using the first dissimilarity as the loss function; and adjusting the parameters of the registration model if the loss function does not converge.

[0008] In some embodiments, the training method of the registration model further includes: segmenting the target tissue from the first sample image to obtain a first target tissue image, and segmenting the first target tissue from the second sample image to obtain a second target tissue image; transforming the second target tissue image using the first registration information to obtain a third target tissue image; calculating a second dissimilarity between the first target tissue image and the third target tissue image; and accordingly, using the first dissimilarity and the second dissimilarity as a loss function.

[0009] In some embodiments, the third determining unit includes: a first obtaining subunit, configured to obtain the coordinate correspondence between the first coordinate set and the second coordinate set; and a calculation subunit, configured to calculate the registration matrix between the first coordinate set and the second coordinate set based on the one-to-one correspondence between the first coordinate set and the second coordinate set, and use the registration matrix as second registration information.

[0010] In some embodiments, the second acquisition unit includes: a second acquisition subunit, configured to acquire second images of the target object in multiple different respiratory states during a respiratory phase; correspondingly, the first determination unit includes: a first registration subunit, configured to register each second image with the first image respectively to obtain first registration information corresponding to multiple different respiratory states during a respiratory phase; a first determination subunit, configured to monitor respiratory state parameters of the target object during surgery and determine the corresponding first registration information based on the respiratory state parameters; and / or, the second acquisition unit includes: a third acquisition subunit, configured to acquire second images of the target object in multiple different respiratory states during a respiratory phase, and acquire images captured by the second image. The first set of coordinates is collected by multiple positioning sensors placed on the target part of the target object; correspondingly, the second determining unit determines the second set of coordinates in the second image coordinate system when the second positioning sensor is captured in each second image, thereby obtaining the first set of coordinates and the second set of coordinates corresponding to different respiratory states of a respiratory phase; the third determining unit includes: a second registration subunit, used to register the first set of coordinates and the second set of coordinates corresponding to the same respiratory state in each respiratory state, respectively, to obtain the second registration information corresponding to each respiratory state; the second determining subunit is used to monitor the respiratory state parameters of the target object during the operation, and determine the corresponding second registration information according to the respiratory state parameters.

[0011] A second aspect of this specification provides a registration method, comprising: acquiring a first image of a target body part of a target object, the first image containing surgical planning information; acquiring a second image of the target body part of the target object during the surgical process, and acquiring a first set of coordinates collected by multiple positioning sensors placed on the target body part of the target object when the second image is captured; the second image includes images of the multiple positioning sensors; the first set of coordinates is the coordinates of the positioning sensors themselves in a real coordinate system; determining first registration information for registering the second image with the first image; determining a second set of coordinates of the multiple positioning sensors in the coordinate system of the second image; determining second registration information for registering the first set of coordinates and the second set of coordinates; acquiring the pose of an endoscope in a real coordinate system; and determining, based on the first registration information and the second registration information, the corresponding pose of the endoscope in the first image, and determining surgical planning information in the virtual endoscopic field of view under the corresponding pose in the first image.

[0012] In some embodiments, determining first registration information for registering the second image with the first image includes: inputting the second image and the first image into a pre-trained registration model to obtain the first registration information output by the registration model; wherein the registration model is trained by: acquiring sample images, the sample images including a first sample image taken preoperatively and a second sample image taken intraoperatively; and repeating the following operations until the loss function converges: inputting the first sample image and the second sample image into the registration model to obtain the first registration information; transforming the second sample image using the first registration information to obtain a third sample image; calculating a first dissimilarity between the first sample image and the third sample image, and using the first dissimilarity as the loss function; and adjusting the parameters of the registration model if the loss function does not converge.

[0013] In some embodiments, the training method of the registration model further includes: segmenting the target tissue from the first sample image to obtain a first target tissue image, and segmenting the first target tissue from the second sample image to obtain a second target tissue image; transforming the second target tissue image using the first registration information to obtain a third target tissue image; calculating a second dissimilarity between the first target tissue image and the third target tissue image; and accordingly, using the first dissimilarity and the second dissimilarity as a loss function.

[0014] In some embodiments, a pre-trained registration model is used as an initial model, and a registration model is trained based on the initial model using sample images of the target object itself.

[0015] In some embodiments, determining second registration information for registering the first coordinate set and the second coordinate set includes: obtaining the coordinate correspondence in the first coordinate set and the second coordinate set; calculating the registration matrix of the first coordinate set and the second coordinate set based on the one-to-one correspondence in the first coordinate set and the second coordinate set, and using the registration matrix as the second registration information.

[0016] In some embodiments, acquiring a second image of the target body part of the target object during surgery and determining first registration information for registering the second image with the first image includes: acquiring multiple second images of the target object in different respiratory states during a respiratory phase; registering each second image with the first image to obtain first registration information corresponding to multiple different respiratory states during a respiratory phase; monitoring respiratory state parameters of the target object during surgery and determining the corresponding first registration information based on the respiratory state parameters; and / or, acquiring a second image of the target body part of the target object during surgery, and acquiring a first set of coordinates collected by multiple positioning sensors placed on the target body part of the target object when the second image is captured, and determining a second set of coordinates in which the positioning sensors are located in the second image coordinate system. Determining the second registration information for registering the first coordinate set and the second coordinate set includes: acquiring second images of the target object in multiple different respiratory states during a respiratory phase, and acquiring a first coordinate set collected by multiple positioning sensors placed on the target part of the target object when the second images are captured; determining the second coordinate set in which the second positioning sensor is located in the second image coordinate system when each second image is captured, thereby obtaining the first coordinate set and the second coordinate set corresponding to each different respiratory state during a respiratory phase; registering the first coordinate set and the second coordinate set corresponding to the same respiratory state in each respiratory state to obtain the second registration information corresponding to each respiratory state; monitoring the respiratory state parameters of the target object during the operation, and determining the corresponding second registration information based on the respiratory state parameters.

[0017] In some embodiments, the target site is the lungs or abdomen.

[0018] A third aspect of this specification provides a surgical system, comprising: a first imaging system for acquiring a first image of a target body part; a second imaging system for acquiring a second image of the target body part; the second imaging system including a plurality of positioning sensors for positioning the target body part, the plurality of positioning sensors being used to position the target body part, each positioning sensor acquiring a value representing the coordinates of its own location in a real coordinate system; an endoscope for acquiring real-time images of the interior of the target body part during surgery; and a processing device for determining surgical planning information in the first image and determining first registration information for registering the second image with the first image; determining a second set of coordinates of the positioning sensors in the coordinate system of the second image; determining second registration information for registering the first set of coordinates and the second set of coordinates; and, based on the first registration information and the second registration information, determining the corresponding pose of the endoscope's real pose in the first image and determining surgical planning information in the endoscopic field of view under the corresponding pose in the first image.

[0019] A fourth aspect of this specification provides a processing apparatus, comprising: a memory and a processor, the processor and the memory being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to implement the steps of the method of any of the second aspects.

[0020] A fifth aspect of this specification provides a computer storage medium storing computer program instructions that, when executed by a processor, implement the steps of the method described in any of the second aspects.

[0021] The registration method, apparatus, and processing device provided in this specification determine surgical planning information in a first image of the target body part of the target object. During the surgery, the target body part of the target object is imaged in real time to obtain a real-time second image. First registration information between the second image and the first image is determined; this first registration information is the conversion information from the real-time image during the surgery to the preoperative image carrying the surgical plan. During real-time imaging during the surgery, multiple positioning sensors are placed on the target body part of the target object. Therefore, the real coordinates of the target object's location in the real coordinate system can be obtained in real time from the positioning sensors. The second image also contains images of these positioning sensors. Second registration information between the real space and the second image can be determined based on the real coordinates of the positioning sensors and their coordinates in the second image. This second registration information is the conversion information from the real space to the second image. Based on the second and first registration information, the real space can be registered with the preoperative first image. This allows the corresponding coordinates of the endoscope's real coordinates to be found in the first image, and the image and surgical planning information that the virtual endoscope at the corresponding coordinate position in the first image can be determined. This solution can automatically register the intraoperative real coordinate system to the preoperative image coordinate system and obtain the surgical planning information corresponding to the actual endoscopic pose. The entire process can be fully automated, reducing manual operation. This solution registers the preoperative image coordinate system with the real coordinate system, not just the intraoperative image coordinate system. It can automatically obtain the local surgical planning information corresponding to the actual endoscopic pose without the need for manual positioning of the local surgical planning information corresponding to the actual endoscopic pose. This solution uses positioning sensors to achieve the registration from the real coordinate system to the intraoperative image coordinate system, eliminating the need for manually marking the coordinates of points. Therefore, the registration method has high accuracy and is not affected by the doctor's skill level or experience. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 A schematic diagram of a bronchial surgery scene is shown;

[0024] Figure 2 A flowchart of the registration method provided in this specification is shown;

[0025] Figure 3 A schematic diagram illustrating the determination of surgical planning information for bronchial surgery is shown.

[0026] Figure 4 A schematic diagram of the registration method provided in this specification is shown;

[0027] Figure 5 Another schematic diagram of the registration method provided in this specification is shown;

[0028] Figure 6 A diagram showing the placement of three positioning sensors on the human body and the corresponding second image is provided.

[0029] Figure 7 A schematic diagram of an electromagnetic positioning sensor is shown.

[0030] Figure 8 A flowchart of a training method for a registration model is shown;

[0031] Figure 9 This diagram illustrates a training method for the registration model.

[0032] Figure 10 A flowchart illustrating another training method for the registration model is shown;

[0033] Figure 11 Another flowchart of the registration method provided in this specification is shown;

[0034] Figure 12 This specification shows a schematic diagram of a surgical system provided.

[0035] Figure 13 An internal structural diagram of the processing apparatus provided in this specification is shown. Detailed Implementation

[0036] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0037] This specification provides a registration method that determines surgical planning information in a first image obtained preoperatively, performs real-time imaging during the operation to obtain a second image corresponding to a given moment, uses the second image as a transition to determine registration information from the real surgical space to the first preoperative image, registers the real coordinates of the endoscope to the virtual endoscope on the first image based on this registration information, and determines the image seen by the virtual endoscope and the surgical planning information on the first image. In other words, the real position of the endoscope is automatically and in real-time registered to the path planned in the first preoperative image, so that the doctor can perform the surgical operation under the guidance of the surgical planning information.

[0038] like Figure 2 As shown, the method includes the following steps:

[0039] S10: Obtain the first image of the target body part of the target object. The first image contains surgical planning information.

[0040] The target can be a person or an animal.

[0041] The target site can be any area requiring endoscopy, such as the lungs or abdomen. The endoscope can be connected to other surgical equipment via physical cables or wirelessly, such as a capsule endoscope. The accompanying drawings in this manual only illustrate the registration scheme using the lungs as the target site and the bronchus as the target tissue; this does not mean that this scheme is only applicable to situations where the target site is the lungs.

[0042] Surgical planning information can refer to the path of surgical instruments to the target site, and what operation to perform at which point in the target site. Figure 3 A schematic diagram illustrating the determination of surgical planning information for bronchial surgery is shown. For example... Figure 3 As shown, preoperative surgical planning information may include first performing bronchial segmentation and lesion segmentation, then locating the puncture point of the tracheal branch closest to the lesion, then finding the shortest path from the main trachea to the puncture point, and finally being confirmed or adjusted by the doctor.

[0043] Bronchial segmentation can be accomplished using deep learning algorithms, such as the common encoder-decoder network Unet; it can also be achieved using traditional algorithms, such as region growing.

[0044] Lesion segmentation can be accomplished using semi-automatic segmentation algorithms, such as the GraphCut algorithm; it can also be done fully automatically using traditional or deep learning algorithms, but manual confirmation and adjustment by doctors are required to improve segmentation accuracy and increase the positive rate of puncture.

[0045] It should be noted that the term "surgery" in this instruction manual includes not only medical procedures such as cutting and suturing of the patient's body using medical instruments, but also procedures such as cutting, forcepsing, or puncturing to remove diseased tissue from the patient's body for pathological examination (i.e., biopsy). In other words, the term "surgery" in this instruction manual refers to the means of processing the patient's body as needed for diagnosis or treatment.

[0046] The first image is determined preoperatively. For example, the first image can be a CT image or an MRI image.

[0047] Since surgical planning needs to be based on the first image, the first image usually requires high precision. Imaging methods with low precision or accuracy are usually not suitable for determining the first image.

[0048] S20: Acquire a second image of the target body part of the target object during the surgical process, and acquire a first set of coordinates collected by multiple positioning sensors placed on the target body part of the target object when the second image is captured; the second image includes images of multiple positioning sensors; the first set of coordinates is the coordinates of the positioning sensor itself in the real coordinate system.

[0049] The second image is determined in real time during the operation (i.e., intraoperatively). Therefore, the imaging technology used to obtain the second image should be suitable for use during the operation and be an imaging method with high imaging precision and accuracy.

[0050] For example, the second image could be a CBCT (Cone beam CT) image.

[0051] The first image and the second image can both be two-dimensional images, or they can both be three-dimensional images.

[0052] The first image and the second image can be two-dimensional or three-dimensional. The first image and the second image are usually images of the same dimension.

[0053] Can be combined Figure 4 and Figure 5 To understand the registration method provided in this manual. Figure 4 "Preoperative images" and Figure 5 The image marked with "①" is the first image. Figure 4 "Intraoperative images" and Figure 5 The image marked with “④” is the second image.

[0054] To register the real-time intraoperative images with the target area in the real coordinate system, this method places multiple positioning sensors at the target area of ​​the object during real-time intraoperative imaging. These sensors acquire their own coordinates in the real coordinate system. Furthermore, these positioning sensors are included in the real-time imaging results during the procedure. Therefore, a first set of coordinates for the positioning sensors in the second image coordinate system can be obtained from the real-time intraoperative imaging results (i.e., the second image). This first set of coordinates includes the first coordinates of each positioning sensor in the second image.

[0055] When placing multiple positioning sensors on a target location, it is important to ensure that the sensors are not on a straight line or on a plane, so that their positions can be uniquely and accurately determined using a spatial coordinate system.

[0056] Figure 4 The diagram shows a schematic of an "intraoperative image" (i.e., the second image) that includes multiple positioning sensor images, where the four gray dots represent images from positioning sensors. Figure 6 The left side of the image shows a schematic diagram illustrating the placement of three positioning sensors on the human body. Figure 6 The right side of the text shows the order according to Figure 6 The diagram shows the real-time images (i.e., the second image) captured when three positioning sensors are placed on the left side of the image. The three circles represent the images from the three positioning sensors, respectively.

[0057] A positioning sensor is a sensor that can obtain the coordinates of its own location in a real coordinate system. Positioning sensors are typically used in conjunction with positioning systems. For example, a positioning sensor can be an electromagnetic positioning sensor used in conjunction with an electromagnetic positioning system. Positioning sensors can be disc-shaped or spherical. Figure 7 A schematic diagram of an electromagnetic positioning sensor is shown.

[0058] Of course, the positioning sensor can also be a positioning sensor that uses other positioning technologies. For example, the positioning sensor can be a target of an optical positioning system, which can accurately determine the coordinates of each target's location in the real coordinate system.

[0059] The true coordinate system in this application refers to the coordinate system of the actual surgical space, not the image coordinate system used when capturing the second image. Typically, the image coordinate system and the true coordinate system are different. True coordinates refer to coordinates within that true coordinate system.

[0060] S30: Determine the first registration information for registering the second image with the first image.

[0061] The first registration information can be a registration matrix that converts the second image into the first image, or it can be a deformation field that converts the second matrix into the first matrix.

[0062] A deformation field is a matrix formed by the displacement vectors of each pixel (for a two-dimensional image) or each voxel (for a three-dimensional image).

[0063] Of course, the first registration information can also be the registration matrix that converts the first image into the second image. Based on the "registration matrix that converts the first image into the second image", the inverse operation can usually be performed to obtain the "registration matrix that converts the second image into the first image".

[0064] Figure 4 The diagram illustrates the function of the first registration information, which involves registering the intraoperative image to the first image coordinate system of the preoperative image (i.e., the first image). The first image coordinate system refers to the coordinate system established when the first image is captured or created.

[0065] Figure 5 The deformation field diagram marked by "⑤" is also the first registration information mentioned above.

[0066] S40: Determine the second set of coordinates of multiple positioning sensors in the second image coordinate system.

[0067] This second coordinate set includes the coordinates of each positioning sensor in the second image coordinate system. The second image coordinate system refers to the coordinate system established when the second image is captured or formed.

[0068] By using the first set of coordinates of multiple positioning sensors in the real coordinate system and the second set of coordinates in the second image coordinate system, a transformation relationship can be established between the real coordinate system and the second image coordinate system to obtain the second registration information.

[0069] S50: Determine the second registration information for registering the first coordinate set and the second coordinate set.

[0070] The second registration information can be a registration matrix that converts the second coordinate set into the first coordinate set. Alternatively, the second registration matrix can also be a registration matrix that converts the first coordinate set into the second coordinate set. The inverse operation of converting the first coordinate set into the second coordinate set usually yields the registration matrix that converts the second coordinate set into the first coordinate set.

[0071] Figure 4 The diagram illustrates the function of the second registration information, which involves registering the target area in the real coordinate system to the second image coordinate system of the intraoperative image (i.e., the second image).

[0072] Figure 5The "transformation matrix" marked by "⑥" is the second registration information mentioned above.

[0073] S60: Obtain the pose of the endoscope in the real coordinate system.

[0074] The position of the endoscope can be the position coordinates and orientation of the target point on the endoscope in the real coordinate system. Among them, the orientation can refer to the direction of extension of the endoscope's field of view axis, etc.

[0075] A position sensor can be installed on the endoscope, enabling it to communicate with the processor in the surgical system. This allows the endoscope to acquire its position coordinates and orientation in the real coordinate system, as determined by the position sensor. In existing technologies, some endoscopes have built-in position sensors that allow them to report their own orientation information.

[0076] S70: Based on the first registration information and the second registration information, determine the corresponding pose of the endoscope in the first image, and determine the surgical planning information in the virtual endoscopic field of view under the corresponding pose in the first image.

[0077] Step S70 can be found in [reference]. Figure 1 To understand A and B in the text.

[0078] like Figure 4 As shown and Figure 5 As shown, by using the first and second registration information, the target area in the real coordinate system can be transformed into the preoperative image (i.e., the first image). This allows the real pose of the endoscope to be transformed into the preoperative image (i.e., the first image). A virtual endoscope can be constructed in the first image, and target tissue images and surgical planning information can be obtained within the virtual endoscope's field of view. Figure 5 The dots marked with “②” in the image represent the true pose of the endoscope in the real coordinate system.

[0079] The registration method provided in this manual determines surgical planning information in a first image of the target body part of the target object. During the surgery, the target body part of the target object is imaged in real time to obtain a real-time second image. First registration information is determined between the second image and the first image; this first registration information is the conversion information from the real-time image during the surgery to the preoperative image carrying the surgical plan. During real-time imaging during the surgery, multiple positioning sensors are placed on the target body part of the target object. Therefore, the real coordinates of the target object's location in the real coordinate system can be obtained in real time from the positioning sensors. The second image also contains images of these positioning sensors. Second registration information is determined between the real space and the second image based on the real coordinates of the positioning sensors and their coordinates in the second image. This second registration information is the conversion information from the real space to the second image. Based on the second and first registration information, the real space can be registered with the preoperative first image. This allows the corresponding coordinates of the endoscope's real coordinates to be found in the first image, and the image and surgical planning information that the virtual endoscope at the corresponding coordinate position in the first image can be determined. This solution can automatically register the intraoperative real coordinate system to the preoperative image coordinate system and obtain the surgical planning information corresponding to the actual endoscopic pose. The entire process can be fully automated, reducing manual operation. This solution registers the preoperative image coordinate system with the real coordinate system, not just the intraoperative image coordinate system. It can automatically obtain the local surgical planning information corresponding to the actual endoscopic pose without the need for manual positioning of the local surgical planning information corresponding to the actual endoscopic pose. This solution uses positioning sensors to achieve the registration from the real coordinate system to the intraoperative image coordinate system, eliminating the need for manually marking the coordinates of points. Therefore, the registration method has high accuracy and is not affected by the doctor's skill level or experience.

[0080] In some embodiments, step S30 can be performed by directly selecting a first target point cloud from the first image based on the characteristics of the target tissue, selecting a second target point cloud from the second image, and calculating the first registration information using the least squares method.

[0081] In other embodiments, step S30 may involve inputting the second image and the first image into a pre-trained registration model to obtain the first registration information output by the registration model.

[0082] like Figure 8 As shown, the registration model is trained through the following steps S31 to S36:

[0083] S31: Obtain sample images, including a first sample image taken before surgery and a second sample image taken during surgery.

[0084] S32: Input the first sample image and the second sample image into the registration model to obtain the first registration information.

[0085] S33: The second sample image is transformed using the first registration information to obtain the third sample image.

[0086] S34: Calculate the first dissimilarity between the first sample image and the third sample image, and use the first dissimilarity as the loss function.

[0087] S35: Determine if the loss function has converged. If the loss function has not converged, proceed to step S36; otherwise, end the training of the registration model using this sample image.

[0088] S36: Adjust the parameters of the registration model and jump to S32 to continue execution.

[0089] Through the above steps S31 to S36, a registration model can be trained using a sample image. Based on this, a large number of sample images can be used to train the registration model, and each sample image is trained in the manner described in steps S31 to S36.

[0090] Figure 9 The diagram shows a training schematic of the registration model, where A is the first sample image taken before surgery, B is the second sample image taken during surgery, X is the registration model, Y is the first registration information, and C is the third sample image mentioned above.

[0091] Furthermore, such as Figure 10 As shown, the training method for the registration model may also include the following steps S37 to S39:

[0092] S37: Segment the target tissue from the first sample image to obtain a first target tissue image, and segment the target tissue from the second sample image to obtain a second target tissue image.

[0093] The target tissue here can be a distinctive tissue within the target area. For example, the target tissue could be the bronchi of the lungs.

[0094] S38: The second target tissue image is transformed using the first registration information to obtain the third target tissue image.

[0095] S39: Calculate the second dissimilarity between the first target tissue image and the third target tissue image.

[0096] Accordingly, step S34 uses the first dissimilarity and the second dissimilarity as the loss function.

[0097] A registration model can be pre-trained based on a large number of sample images of a large number of target objects (i.e., through big data), making the registration model applicable to each target object. That is, during the surgery of each target object, the first registration information is obtained through the pre-trained registration model.

[0098] In some embodiments, the first registration information can be obtained through big data training as described above. Then, using a pre-trained registration model as the initial model, the initial model is further trained by combining sample images of the target object itself to obtain a registration model with a higher matching degree to the target object. That is, the initial model for multiple target objects can be the same.

[0099] In some embodiments, the registration model trained during the previous registration process can be used as the initial model, and the registration model can be trained using sample images of the target object itself. That is, the initial models for multiple target objects are different.

[0100] In some embodiments, step S50 above can be performed by directly calculating the registration matrix between the first coordinate set and the second coordinate set. For example, the ICP algorithm in point cloud registration methods can be used. Specifically, after obtaining the coordinate correspondence between the first coordinate set and the second coordinate set, the registration matrix between the first coordinate set and the second coordinate set can be calculated based on the one-to-one correspondence between the first coordinate set and the second coordinate set, and the registration matrix can be used as the second registration information.

[0101] Because the tissues in the thoracic and abdominal cavities have a certain degree of elasticity, the target subject's breathing process usually causes some deformation of the tissues. Therefore, the target subject's breathing has a significant impact on the registration effect.

[0102] To compensate for the impact of respiration on registration results, such as Figure 11 As shown, the registration method proposed in this specification may include the following steps:

[0103] S10: Obtain the first image of the target body part of the target object. The first image contains surgical planning information.

[0104] S1101: Obtain a second image of the target object in multiple different respiratory states during a respiratory phase.

[0105] S1102: Register each second image with the first image to obtain first registration information corresponding to multiple different respiratory states of a respiratory phase.

[0106] S1103: Monitor the respiratory status parameters of the target subject during the operation and determine the corresponding first registration information based on the respiratory status parameters.

[0107] S1104: Acquire second images of the target object in multiple different respiratory states during a respiratory phase, and acquire the first coordinate set collected by multiple positioning sensors placed on the target part of the target object when the second image is captured, determine the second coordinate set of the second positioning sensor in the second image coordinate system when each second image is captured, thereby obtaining the first coordinate set and the second coordinate set corresponding to each different respiratory state during a respiratory phase.

[0108] S1105: Register the first coordinate set and the second coordinate set corresponding to the same breathing state in each breathing state to obtain the second registration information corresponding to each breathing state.

[0109] S1106: Monitor the respiratory status parameters of the target subject during the operation and determine the corresponding second registration information based on the respiratory status parameters.

[0110] S60: Obtain the pose of the endoscope in the real coordinate system.

[0111] S70: Based on the first registration information and the second registration information, determine the corresponding pose of the endoscope in the first image, and determine the surgical planning information in the virtual endoscopic field of view under the corresponding pose in the first image.

[0112] Figure 11 In the registration method shown, the first registration information and the second registration information corresponding to multiple respiratory states of a respiratory phase are predetermined. During the operation, the respiratory state parameters of the target object are detected in real time, and the first registration information and the second registration information are determined according to the respiratory state parameters. That is, the first registration information and the second registration information used in the registration process are corresponding to the respiratory state and change with the respiratory state. Therefore, the registration result is more accurate.

[0113] Figure 11 The registration method shown can determine the first and second registration information corresponding to multiple respiratory states of a respiratory phase before surgery. During the operation, the first and second registration information corresponding to the respiratory state can be found directly based on the real-time monitored respiratory state parameters, without the need to calculate the first and second registration information in real time during the operation. Therefore, the amount of calculation during the operation is reduced, and the acquisition of the first and second registration information is more timely, so as to provide more timely feedback of local surgical planning information.

[0114] This specification provides a registration apparatus that can be used to implement the registration method described above. The registration apparatus includes a first acquisition unit, a second acquisition unit, a first determination unit, a second determination unit, a third determination unit, a third acquisition unit, and a fourth determination unit.

[0115] The first acquisition unit is used to acquire a first image of the target part of the target object, wherein the first image contains surgical planning information.

[0116] The second acquisition unit is used to acquire a second image of the target body part of the target object during the surgical process, and to acquire a first set of coordinates collected by multiple positioning sensors placed on the target body part of the target object when the second image is captured; the second image includes the images of the multiple positioning sensors; the first set of coordinates is the coordinates of the position of the positioning sensor itself in the real coordinate system.

[0117] The first determining unit is used to determine the first registration information for registering the second image with the first image.

[0118] The second determining unit is used to determine the second set of coordinates of the plurality of positioning sensors in the second image coordinate system.

[0119] The third determining unit is used to determine the second registration information for registering the first coordinate set and the second coordinate set.

[0120] The third acquisition unit is used to acquire the pose of the endoscope in the real coordinate system.

[0121] The fourth determining unit is used to determine the corresponding pose of the endoscope in the first image based on the first registration information and the second registration information, and to determine the surgical planning information in the virtual endoscopic field of view under the corresponding pose in the first image.

[0122] In some embodiments, the first determining unit inputs the second image and the first image into a pre-trained registration model to obtain first registration information output by the registration model;

[0123] The registration model is trained using the following method: acquiring sample images, including a first sample image taken preoperatively and a second sample image taken intraoperatively; and repeating the following operations until the loss function converges: inputting the first sample image and the second sample image into the registration model to obtain first registration information; transforming the second sample image using the first registration information to obtain a third sample image; calculating the first dissimilarity between the first sample image and the third sample image, and using the first dissimilarity as the loss function; and adjusting the parameters of the registration model if the loss function does not converge.

[0124] In some embodiments, the training method of the registration model further includes: segmenting the target tissue from the first sample image to obtain a first target tissue image, and segmenting the first target tissue from the second sample image to obtain a second target tissue image; transforming the second target tissue image using the first registration information to obtain a third target tissue image; calculating a second dissimilarity between the first target tissue image and the third target tissue image; and accordingly, using the first dissimilarity and the second dissimilarity as a loss function.

[0125] In some embodiments, the third determining unit includes: a first obtaining subunit, configured to obtain the coordinate correspondence between the first coordinate set and the second coordinate set; and a calculation subunit, configured to calculate the registration matrix between the first coordinate set and the second coordinate set based on the one-to-one correspondence between the first coordinate set and the second coordinate set, and use the registration matrix as second registration information.

[0126] In some embodiments, the second acquisition unit includes a second acquisition subunit, configured to acquire second images of a target object in multiple different respiratory states during a respiratory phase.

[0127] Accordingly, the first determining unit includes: a first registration subunit, used to register each second image with the first image respectively to obtain first registration information corresponding to multiple different respiratory states of a respiratory phase; and a first determining subunit, used to monitor the respiratory state parameters of the target object during the operation and determine the corresponding first registration information based on the respiratory state parameters.

[0128] In some embodiments, the second acquisition unit includes: a third acquisition subunit, configured to acquire second images of the target object in multiple different respiratory states during a respiratory phase, and acquire a first set of coordinates collected by multiple positioning sensors placed on the target part of the target object when the second image is captured.

[0129] Accordingly, the second determining unit determines the second coordinate set in the second image coordinate system where the second positioning sensor is located when each second image is captured, thereby obtaining the first coordinate set and the second coordinate set corresponding to each different breathing state of a breathing phase; the third determining unit includes: a second registration subunit, used to register the first coordinate set and the second coordinate set corresponding to the same breathing state in each breathing state respectively, to obtain the second registration information corresponding to each breathing state; and a second determining subunit, used to monitor the breathing state parameters of the target object during the operation, and determine the corresponding second registration information based on the breathing state parameters.

[0130] This manual provides a surgical system, such as Figure 12As shown, it includes: a first imaging system 1201, a second imaging system 1202, an endoscope 1203, and a processing device 1204.

[0131] The first imaging system 1201 is used to acquire a first image of the target part of the target object. The first imaging system 1201 can be a CT imaging system or an MRI imaging system.

[0132] The second imaging system 1202 is used to acquire a second image of the target area of ​​the target object. The second imaging system includes multiple positioning sensors, which are used to capture images of the target area of ​​the target object. The acquisition value of each positioning sensor is the coordinate of its own position in the real coordinate system. The second imaging system can be a CBCT imaging system.

[0133] Endoscope 1203 is used to acquire real-time images of the inside of a target site during surgery.

[0134] The processing device 1204 is used to determine surgical planning information in the first image and to determine first registration information for registering the second image with the first image; to determine the second coordinate set in the coordinate system of the positioning sensor in the second image; to determine the second registration information for registering the first coordinate set and the second coordinate set; and, based on the first registration information and the second registration information, to determine the corresponding pose of the endoscope in the first image and to determine the surgical planning information in the endoscopic field of view under the corresponding pose in the first image.

[0135] The above surgical system can be found in [reference]. Figure 1 To understand.

[0136] This invention also provides a processing device, such as... Figure 13 As shown, the processing device may include a processor 1301 and a memory 1302, wherein the processor 1301 and the memory 1302 may be connected by a bus or other means.

[0137] Processor 1301 may be a central processing unit (CPU). Processor 1301 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0138] The memory 1302, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the brain puncture assist method in the embodiments of the present invention. The processor 1301 executes various functional applications and data classification of the processing device by running the non-transitory software programs, instructions, and modules stored in the memory 1302, thereby realizing the brain puncture assist method in the above method embodiments.

[0139] The memory 1302 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor 1301, etc. Furthermore, the memory 1302 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1302 may optionally include memory remotely located relative to the processor 1301, and these remote memories may be connected to the processor 1301 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0140] The one or more modules are stored in the memory 1302 and, when executed by the processor 1301, perform the brain puncture-assisted method provided in this specification.

[0141] For details regarding the processing device described above, please refer to the relevant descriptions and effects in the corresponding embodiments of the above method for further understanding; they will not be repeated here.

[0142] This specification also provides a computer storage medium storing computer program instructions, which, when executed, implement the steps of the corresponding embodiments of the above-described method.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0144] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, hardware + program embodiments are relatively simple in description because they are fundamentally similar to method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0145] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0146] Those skilled in the art will also know that, besides implementing a processor in the form of purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the processor function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic processors (PLCs), and embedded microprocessors. Therefore, such a processor can be considered a hardware component, and the devices within it used to implement various functions can also be considered structures within that hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the method and structures within a hardware component.

[0147] The above description is merely an embodiment of the present specification and is not intended to limit the embodiments of the present specification. For those skilled in the art, various modifications and variations can be made to the embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the embodiments of the present specification should be included within the scope of the claims of the embodiments of the present specification.

Claims

1. A registration device, characterized in that, include: The first acquisition unit is used to acquire a first image of the target part of the target object, wherein the first image contains surgical planning information; The second acquisition unit is used to acquire a second image of the target body part of the target object during the surgical process, and to acquire a first set of coordinates collected by multiple positioning sensors placed on the target body part of the target object when the second image is captured; the second image includes the images of the multiple positioning sensors; the first set of coordinates is the coordinates of the position of the positioning sensor itself in the real coordinate system; the multiple positioning sensors are not on a straight line or on a plane, so that the position of the multiple positioning sensors can be accurately and uniquely determined to a spatial coordinate system; The first determining unit is used to determine first registration information for registering the second image with the first image; The second determining unit is used to determine the second set of coordinates of the plurality of positioning sensors in the second image coordinate system; The third determining unit is used to determine the second registration information for registering the first coordinate set and the second coordinate set; The third acquisition unit is used to acquire the pose of the endoscope in the real coordinate system; The fourth determining unit is used to determine the corresponding pose of the endoscope in the first image based on the first registration information and the second registration information, and to determine the surgical planning information in the virtual endoscopic field of view under the corresponding pose in the first image.

2. The registration device according to claim 1, characterized in that, The first determining unit inputs the second image and the first image into a pre-trained registration model to obtain the first registration information output by the registration model; The registration model is trained using the following method: Acquire sample images, including a first sample image taken preoperatively and a second sample image taken intraoperatively; and repeat the following operations until the loss function converges: The first sample image and the second sample image are input into the registration model to obtain the first registration information; The second sample image is transformed using the first registration information to obtain the third sample image; Calculate the first dissimilarity between the first sample image and the third sample image, and use the first dissimilarity as the loss function; If the loss function does not converge, adjust the parameters of the registration model.

3. The registration device according to claim 2, characterized in that, The training method for the registration model also includes: The target tissue is segmented from the first sample image to obtain a first target tissue image, and the first target tissue is segmented from the second sample image to obtain a second target tissue image; The second target tissue image is transformed using the first registration information to obtain a third target tissue image; Calculate the second dissimilarity between the first target tissue image and the third target tissue image; Accordingly, the first dissimilarity and the second dissimilarity are used as loss functions.

4. The registration device according to claim 1, characterized in that, The third determining unit includes: The first acquisition subunit is used to acquire the coordinate correspondence between the first coordinate set and the second coordinate set; The calculation subunit is used to calculate the registration matrix of the first coordinate set and the second coordinate set according to the one-to-one correspondence in the first coordinate set and the second coordinate set, and use the registration matrix as the second registration information.

5. The registration device according to claim 1, characterized in that, The second acquisition unit includes: a second acquisition subunit, used to acquire second images of the target object in multiple different respiratory states during a respiratory phase; Accordingly, the first determining unit includes: The first registration subunit is used to register each second image with the first image to obtain first registration information corresponding to multiple different respiratory states of a respiratory phase. The first determining subunit is used to monitor the respiratory state parameters of the target object during the operation and determine the corresponding first registration information based on the respiratory state parameters; And / or, The second acquisition unit includes: a third acquisition subunit, used to acquire second images of the target object in multiple different respiratory states during a respiratory phase, and to acquire a first set of coordinates collected by multiple positioning sensors placed on the target part of the target object when the second image is captured; Accordingly, the second determining unit determines the second coordinate set in the second image coordinate system when the second positioning sensor is captured in each second image, thereby obtaining the first coordinate set and the second coordinate set corresponding to each different breathing state of a breathing phase; The third determining unit includes: The second registration subunit is used to register the first coordinate set and the second coordinate set corresponding to the same breathing state in each breathing state, so as to obtain the second registration information corresponding to each breathing state. The second determining subunit is used to monitor the respiratory state parameters of the target object during the operation and determine the corresponding second registration information based on the respiratory state parameters.

6. A registration method, characterized in that, include: Acquire a first image of the target body part of the target object, wherein the first image contains surgical planning information; A second image of the target body part of the target object is acquired during the surgical procedure, and a first set of coordinates is acquired from multiple positioning sensors placed on the target body part of the target object when the second image is captured; the second image includes the images of the multiple positioning sensors; the first set of coordinates is the coordinates of the position of the positioning sensor itself in the real coordinate system; the multiple positioning sensors are not on a straight line or on a plane, so that the position of the multiple positioning sensors can accurately and uniquely determine a spatial coordinate system; Determine the first registration information for registering the second image with the first image; Determine the second set of coordinates of the plurality of positioning sensors in the second image coordinate system; Determine the second registration information for registering the first coordinate set and the second coordinate set; Obtain the pose of the endoscope in the real coordinate system; Based on the first registration information and the second registration information, the corresponding pose of the endoscope in the first image is determined, and the surgical planning information in the virtual endoscopic field of view under the corresponding pose in the first image is determined.

7. The registration method according to claim 6, characterized in that, Determining the first registration information for registering the second image with the first image includes: The second image and the first image are input into a pre-trained registration model to obtain the first registration information output by the registration model; The registration model is trained using the following method: Acquire sample images, including a first sample image taken preoperatively and a second sample image taken intraoperatively; and repeat the following operations until the loss function converges: The first sample image and the second sample image are input into the registration model to obtain the first registration information; The second sample image is transformed using the first registration information to obtain the third sample image; Calculate the first dissimilarity between the first sample image and the third sample image, and use the first dissimilarity as the loss function; If the loss function does not converge, adjust the parameters of the registration model.

8. The registration method according to claim 7, characterized in that, The training method for the registration model also includes: The target tissue is segmented from the first sample image to obtain a first target tissue image, and the first target tissue is segmented from the second sample image to obtain a second target tissue image; The second target tissue image is transformed using the first registration information to obtain a third target tissue image; Calculate the second dissimilarity between the first target tissue image and the third target tissue image; Accordingly, the first dissimilarity and the second dissimilarity are used as loss functions.

9. The registration method according to claim 6, characterized in that, The second registration information for registering the first coordinate set and the second coordinate set includes: Obtain the coordinate correspondence between the first coordinate set and the second coordinate set; Based on the one-to-one correspondence between the first coordinate set and the second coordinate set, a registration matrix is ​​calculated between the first coordinate set and the second coordinate set, and the registration matrix is ​​used as the second registration information.

10. The registration method according to claim 6, characterized in that, Acquire a second image of the target body part during the surgical procedure, and determine first registration information for registering the second image with the first image, including: Acquire a second image of the target object in multiple different respiratory states during a respiratory phase; Each second image is registered with the first image to obtain first registration information corresponding to multiple different respiratory states of a respiratory phase; During the surgery, the respiratory status parameters of the target subject are monitored, and the corresponding first registration information is determined based on the respiratory status parameters; And / or, Acquire a second image of the target body part of the target object during the surgical procedure, and acquire a first set of coordinates collected by multiple positioning sensors placed on the target body part of the target object when the second image is captured. Determine a second set of coordinates in which the positioning sensors are located in the coordinate system of the second image, and determine second registration information for registering the first set of coordinates and the second set of coordinates, including: Acquire second images of the target object in multiple different respiratory states during a respiratory phase, and acquire the first set of coordinates collected by multiple positioning sensors placed on the target part of the target object when the second image is captured. Determine the second set of coordinates of the second positioning sensor in the second image coordinate system when each second image is captured, thereby obtaining the first set of coordinates and the second set of coordinates corresponding to each different respiratory state during a respiratory phase. The first coordinate set and the second coordinate set corresponding to the same breathing state in each breathing state are registered to obtain the second registration information corresponding to each breathing state. During the surgery, the respiratory status parameters of the target subject are monitored, and the corresponding second registration information is determined based on the respiratory status parameters.

Images