Real-time non-rigid registration method and system for surgical navigation image based on deep learning

Abstract

The application discloses a kind of real-time non-rigid registration method and system of surgical navigation image based on deep learning, comprising: calibration the space transformation matrix between RGB-D camera and medical imaging equipment;RGB-D camera real-time capture patient whole body optical image and depth information;Medical imaging is carried out to patient, while reconstructing the initial model of patient based on SMPL based on human posture estimation neural network of deep learning;Medical image is point cloud, and the point-to-plane spatial position matching relationship between medical image and the initial model of patient is established;Neural network reconstructs the dynamic change of patient model in operation in real time, and the non-rigid deformation field of medical image is solved by matching relationship;Registration medical image, update display.The application can realize the real-time non-rigid registration of surgical navigation medical image to different body posture patient, solve the movement, deformation of organ, soft tissue caused by patient posture change in surgical navigation, thereby cause the problem of medical image failure, help to improve the accuracy of surgical navigation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer-aided medical treatment, and particularly relates to a real-time non-rigid registration method and system for surgical navigation images based on deep learning. BACKGROUND

[0002] Surgical navigation technology is one of the important developments in the field of modern medicine, and its main purpose is to provide accurate and reliable spatial information in real time during surgery to guide surgeons to perform high-precision surgical operations. Surgical navigation systems usually use medical imaging devices such as CT, MRI, PET, and ultrasound to obtain information about the patient's internal organs and lesion areas. Then, through a specific hardware and software platform, these medical image data are converted into a three-dimensional model, and the surgeon is guided to operate in real time during the operation.

[0003] However, in the clinical application of surgical navigation, the organs and soft tissues will move and deform with the change of the patient's posture. In order to obtain accurate target anatomical structure information, the surgeon needs to perform frequent scanning during the operation to update the medical images, which will cause the patient to be exposed to a higher level of radiation. Another solution is to use a custom-made mold made of plastic or plaster to fix the patient, keeping the patient in the same posture before and during the operation, but this method is usually suboptimal, limiting the flexibility of the surgeon's operation, increasing the difficulty of the operation, and causing additional time and financial costs to the patient. SUMMARY

[0004] The purpose of the application is to solve the problem of organ and soft tissue movement and deformation caused by the change of the patient's posture in surgical navigation, which leads to the failure of medical images. A real-time non-rigid registration method and system for surgical navigation images based on deep learning are proposed to achieve real-time non-rigid registration of medical images for different patients in surgical navigation.

[0005] In order to achieve the above purpose, the application adopts the following technical solutions:

[0006] In a first aspect, the application provides a real-time non-rigid registration method for surgical navigation images based on deep learning, comprising the following steps:

[0007] Calibrating the spatial transformation matrix between the RGB-D camera and the medical imaging device;

[0008] The RGB-D camera captures real-time optical images and depth information of the patient's whole body;

[0009] The patient is imaged for surgical navigation, and at the same time, the patient's whole body optical image and depth information are input into the pre-established human posture estimation neural network based on deep learning, and the patient's initial model based on the SMPL human model is reconstructed;

[0010] Medical images are cloudified into points, and the spatial transformation matrix is ​​used to establish a point-to-surface spatial position matching relationship between the medical images and the initial patient model.

[0011] A neural network is used to reconstruct a patient model that is dynamically changing during surgery in real time, and the non-rigid deformation field of the medical image is solved by matching relationships.

[0012] Register medical images and update the display.

[0013] As a preferred technical solution, the spatial relationship between the calibrated RGB-D camera and the medical imaging equipment is specifically as follows:

[0014] Select a marker that is clearly visible on both the RGB-D camera and the medical imaging device, ensuring that the marker is within the field of view of both devices; use the RGB-D camera and the medical imaging device to photograph the patient and the marker respectively, acquiring data from both imaging devices; identify the marker and extract its spatial coordinates on both the RGB-D camera and the medical imaging device; calculate the spatial transformation matrix between the RGB-D camera and the medical imaging device using the least squares method.

[0015] As a preferred technical solution, when a patient completes medical imaging, the video frames captured by the RGB-D camera are marked as initial frames. The initial frames are input into a deep learning-based human pose estimation neural network, which outputs SMPL parameters that conform to the patient's body shape and pose. The SMPL then reconstructs the patient's initial model based on the parameters.

[0016] As a preferred technical solution, the human pose estimation neural network has an encoder-decoder structure, including an optical image encoding module, a depth image encoding module, and a decoding module, specifically:

[0017] The optical image encoding module is used to extract texture and color features from RGB images. These features include the patient's body contour and skin color, which help the human pose estimation neural network to identify the relationship between joint positions and body parts.

[0018] The depth image encoding module is used to extract human geometric features in three-dimensional space from depth images. These features help determine the relative positions of the patient's joints and body parts in three-dimensional space.

[0019] The decoding module is used to decode information about human posture and body shape from the encoded and merged features and map it into the parameter space of the SMPL model.

[0020] The optical image coding module and the depth image coding module have the same structure but do not share parameters. The image features extracted by the two coding modules are merged and input into the fully connected layer of the decoding module, and then output the parameters W = [φ, θ, β] of the SMPL human body model, where φ is the patient's global pose parameter, θ is the pose parameter of each part, and β is the body posture parameter including height, weight and body shape.

[0021] The SMPL human body model is a parametric human body model. Based on the parameter W, SMPL outputs a mesh human body model M composed of vertices.

[0022] As a preferred technical solution, both the optical image encoding module and the depth image encoding module use ResNet-50, and the decoding module is a fully connected layer.

[0023] As a preferred technical solution, the training method of the human pose estimation neural network is as follows:

[0024] The training dataset is input into the human pose estimation neural network to obtain the estimated SMPL parameters W = [φ, θ, β]. The SMPL reconstructs the human model M based on the parameters.

[0025] The attitude estimation loss function is calculated as follows:

[0026]

[0027] in, These are the pose parameters labeled in the dataset;

[0028] The posture estimation loss function is calculated as follows:

[0029]

[0030] in, These are the postural parameters labeled in the dataset;

[0031] The three-dimensional joint loss is calculated as follows:

[0032]

[0033] Among them, J 3D The three-dimensional human joint positions are calculated from the reconstructed SMPL model. The 3D joint positions labeled for the dataset;

[0034] The two-dimensional joint projection loss is calculated as follows:

[0035]

[0036] Among them, J 2DThe two-dimensional human joint positions are represented by the orthogonal projection of the three-dimensional joint positions onto the camera plane. Two-dimensional joint locations labeled for the dataset;

[0037] The complete loss function is calculated as follows:

[0038] L T =L p +L s +L 3D +L 2D ;

[0039] The Adam optimizer completes backpropagation and updates the network parameters.

[0040] As a preferred technical solution, the medical image point cloudification utilizes the spatial transformation matrix to establish a point-to-surface spatial position matching relationship between the medical image and the initial patient model, specifically as follows:

[0041] The medical image data is converted into a medical image point cloud P, and then transformed into the spatial coordinate system of the RGB-D camera through the spatial transformation matrix calibrated by the RGB-D camera and the medical imaging equipment. The nearest patient model mesh surface m of each point p in the medical image point cloud P is searched, and a pairwise matching relationship is established.

[0042] As a preferred technical solution, the neural network reconstructs the patient model that dynamically changes during surgery in real time, and solves the non-rigid deformation field of the medical image by matching relationships, specifically as follows:

[0043] An RGB-D camera captures real-time optical images and depth information of the patient's whole body during surgery, and a human pose estimation neural network reconstructs a dynamically changing patient model in real time.

[0044] The non-rigid deformation field of the medical image is calculated by solving the position change of the mesh surface m matched by each point p in the medical image point cloud P. The relative pose transformation matrix of all matched meshes m from the initial patient model to the new patient model is calculated. The relative pose transformation matrix of all matched meshes forms the deformation field, which is represented as a vector field, where each vector represents the rotation and displacement of a spatial point during the shape change process.

[0045] As a preferred technical solution, the registration and updating of medical images specifically includes:

[0046] The displacement and rotation vectors in the calculated non-rigid deformation field are multiplied by the corresponding points in the medical image point cloud to transform each point in the medical image point cloud to a new coordinate system. The medical image point cloud after applying the non-rigid deformation field is consistent with the new patient model in space. During the surgical navigation process, the doctor performs precise operations based on the real-time updated medical images.

[0047] Secondly, the present invention provides a real-time non-rigid registration system for surgical navigation images based on deep learning, for implementing the real-time non-rigid registration method for surgical navigation images based on deep learning, including a calibration module, a human pose estimation module, a registration initialization module, a non-rigid registration module and a display module;

[0048] The calibration module is used to calibrate the spatial transformation matrix between the RGB-D camera and the medical imaging equipment;

[0049] The human pose estimation module is used to capture the patient's whole-body optical image and depth information in real time by the RGB-D camera, and the human pose estimation neural network reconstructs the patient model based on SMPL in real time.

[0050] The registration initialization module is used to perform medical imaging of the patient for surgical navigation, and the patient model at the same time is marked as the initial model; and to establish the point-to-surface spatial position matching relationship between the medical image point cloud and the patient initial model;

[0051] The non-rigid registration module is used to solve the non-rigid deformation field of the medical image based on the matching relationship between the patient model reconstructed in real time during the operation and the medical image, and to register the medical image.

[0052] The display module is used to register medical images and update the display.

[0053] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0054] 1. The method and system of this invention are mainly applied in surgical navigation. There is no need to frequently update medical images or use instruments to fix the patient during the operation. By estimating the patient's posture and attitude in real time through a registration method based on deep learning, the changes in the patient's posture and the movement and deformation of organs and soft tissues can be tracked, which improves the accuracy and practicality of surgical navigation.

[0055] 2. The method and system of the present invention have the advantages of being lightweight and high-performance, and can realize real-time calculation. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart of the real-time non-rigid registration method for surgical navigation images based on deep learning, as described in this invention.

[0058] Figure 2 This is a flowchart of the real-time deployment process of the present invention. Detailed Implementation

[0059] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0060] Example

[0061] The method and system proposed in this invention enable real-time, non-rigid registration of surgical navigation medical images for patients in different body postures, solving the problem of medical image failure caused by organ and soft tissue movement and deformation due to changes in patient posture during surgical navigation. (See reference...) Figure 1 , Figure 2 The present invention provides a real-time non-rigid registration method for surgical navigation images based on deep learning, comprising the following steps:

[0062] S1. Determine the spatial relationship, specifically:

[0063] Select a marker that is clearly visible on both the RGB-D camera and the medical imaging equipment, ensuring that the marker is within the field of view of both devices. Use the RGB-D camera and the medical imaging equipment (such as a CT scanner, MRI machine, photoacoustic imager, etc.) to photograph the patient and the marker, acquiring data from both imaging devices. Identify the marker and use computer vision algorithms, such as deep learning-based object detection methods, to extract the spatial coordinates of the marker on both the RGB-D camera and the medical imaging equipment. Calculate the spatial transformation matrix between the two imaging devices using the least squares method.

[0064] The S2 and RGB-D cameras capture real-time optical images and depth information of the patient's whole body, which are then used as input to the human pose estimation neural network in subsequent steps.

[0065] S3. Perform medical imaging on the patient and simultaneously reconstruct the initial patient model. The specific steps are as follows:

[0066] S31. Performing medical imaging on the patient for surgical navigation, including but not limited to: computed tomography, magnetic resonance imaging, photoacoustic imaging, ultrasound imaging, and optical coherence tomography.

[0067] S32. When medical imaging is completed, mark the video frame captured by the RGB-D camera as the initial frame;

[0068] S33. The initial frame input is a human pose estimation neural network based on deep learning, and the network output is SMPL parameters that conform to the patient's body shape and pose.

[0069] The human pose estimation neural network has an encoder-decoder structure, including an optical image encoding module, a depth image encoding module, and a decoding module, specifically:

[0070] The optical image coding module uses ResNet-50, and the depth image coding module also uses ResNet-50. They have the same structure but do not share parameters. The image features extracted by the two coding modules are merged and input into a fully connected layer that serves as the decoding module. The output is the SMPL parameters W = [φ, θ, β], where... For the patient's global pose parameters, These are the attitude parameters for each part. It includes body shape parameters such as height, weight, and build.

[0071] Based on the parameter W, a skinned multi-person linear model (SMPL) is reconstructed, resulting in a mesh human body model consisting of 10475 vertices. This model is the initial model, corresponding to the patient's posture when completing medical imaging.

[0072] Furthermore, the training steps for the human pose estimation neural network are as follows:

[0073] The training dataset is input into the network to obtain the estimated SMPL parameters W = [φ, θ, β]. SMPL then reconstructs the human body model based on these parameters.

[0074] The attitude estimation loss function is calculated as follows:

[0075]

[0076] in, These are the pose parameters labeled in the dataset;

[0077] The posture estimation loss function is calculated as follows:

[0078]

[0079] in, These are the postural parameters labeled in the dataset;

[0080] The three-dimensional joint loss is calculated as follows:

[0081]

[0082] in, The three-dimensional human joint positions are calculated from the reconstructed SMPL model. The 3D joint positions labeled for the dataset;

[0083] The two-dimensional joint projection loss is calculated as follows:

[0084]

[0085] in, The two-dimensional human joint positions are represented by the orthogonal projection of the three-dimensional joint positions onto the camera plane. Two-dimensional joint locations labeled for the dataset;

[0086] The complete loss function is calculated as follows:

[0087] L T =L p +L s +L 3D +L 2D ;

[0088] The Adam optimizer completes backpropagation and updates the network parameters;

[0089] S34. Reconstruct the SMPL human body model based on the parameters. This model is the initial model, corresponding to the patient's posture when completing medical imaging.

[0090] S4. Medical image point cloudification and spatial matching with the patient model are established, specifically as follows:

[0091] S41. Using medical image processing libraries such as ITK and VTK, extract voxel data, map voxel coordinates to three-dimensional spatial coordinates, and transform medical image data into three-dimensional point cloud data P.

[0092] S42. Using the spatial transformation matrix obtained from S1 calibration, transform the 3D point cloud P into the spatial coordinate system of the RGB-D camera.

[0093] S43. Using a KD-tree (K-dimensional tree) as the spatial search data structure, search for the patient model grid m with the smallest Euclidean distance for each point p in the medical image point cloud P, and establish a pairwise matching relationship.

[0094] S5. Intraoperative real-time reconstruction of the patient model, solving for the non-rigid deformation field of the medical image, specifically:

[0095] S51. Repeat steps S2 and S3. The RGB-D camera captures real-time optical images and depth information of the patient's whole body during the operation. The human pose estimation neural network reconstructs the dynamically changing patient model in real time.

[0096] S52. Solve the deformation field of the medical image. Calculate the relative pose transformation matrix of all matching meshes m from the initial patient model to the new patient model; the relative pose transformation matrices of all matching meshes form the deformation field, which can be represented as a vector field, where each vector represents the rotation and displacement of a spatial point during the shape change process.

[0097] S6. Non-rigid registration and update display of medical images, specifically:

[0098] The calculated displacement and rotation vectors in the non-rigid deformation field are multiplied with the corresponding points in the medical image point cloud using matrix multiplication. This transforms each point in the medical image point cloud to a new coordinate system, ensuring that the medical image point cloud after applying the non-rigid deformation field maintains spatial consistency with the new patient model. During surgical navigation, surgeons can perform precise operations based on real-time updated medical images.

[0099] Another embodiment of the present invention provides a real-time non-rigid registration system for surgical navigation images based on deep learning, including a calibration module, a human pose estimation module, a registration initialization module, a non-rigid registration module, and a display module;

[0100] The calibration module is used to calibrate the spatial transformation matrix between the RGB-D camera and the medical imaging equipment;

[0101] The human pose estimation module is used to capture the patient's whole-body optical image and depth information in real time by the RGB-D camera, and the human pose estimation neural network reconstructs the patient model based on SMPL in real time.

[0102] The registration initialization module is used to perform medical imaging of the patient for surgical navigation, and the patient model at the same time is marked as the initial model; and to establish the point-to-surface spatial position matching relationship between the medical image point cloud and the patient initial model;

[0103] The non-rigid registration module is used to solve the non-rigid deformation field of the medical image based on the matching relationship between the patient model reconstructed in real time during the operation and the medical image, and to register the medical image.

[0104] The display module is used to register medical images and update the display.

[0105] Furthermore, it also includes a network training module to train the human pose estimation neural network. The specific training process is as follows:

[0106] The training dataset is input into the network to obtain the estimated SMPL parameters W = [φ, θ, β]. SMPL then reconstructs the human body model based on these parameters.

[0107] The attitude estimation loss function is calculated as follows:

[0108]

[0109] in, These are the pose parameters labeled in the dataset;

[0110] The posture estimation loss function is calculated as follows:

[0111]

[0112] in, These are the postural parameters labeled in the dataset;

[0113] The three-dimensional joint loss is calculated as follows:

[0114]

[0115] in, The three-dimensional human joint positions are calculated from the reconstructed SMPL model. The 3D joint positions labeled for the dataset;

[0116] The two-dimensional joint projection loss is calculated as follows:

[0117]

[0118] in, The two-dimensional human joint positions are represented by the orthogonal projection of the three-dimensional joint positions onto the camera plane. Two-dimensional joint locations labeled for the dataset;

[0119] The complete loss function is calculated as follows:

[0120] L T =L p +L s +L 3D +L 2D ;

[0121] The Adam optimizer completes backpropagation and updates the network parameters.

[0122] It should also be noted that, in this specification, terms such as "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include those elements, but also includes other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0123] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0124] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A real-time non-rigid registration method for surgical navigation images based on deep learning, characterized in that, Includes the following steps: Calibrate the spatial transformation matrix between the RGB-D camera and the medical imaging equipment; RGB-D cameras capture real-time optical images and depth information of the patient's entire body; Medical imaging is performed on the patient for surgical navigation. At the same time, the patient's whole-body optical images and depth information are input into a pre-established deep learning-based human pose estimation neural network to reconstruct an initial patient model based on the SMPL human model. The medical image point cloud is converted into a point cloud. The spatial transformation matrix is ​​used to establish the point-to-surface spatial position matching relationship between the medical image and the initial patient model. Specifically, the medical image data is converted into a medical image point cloud P, and then transformed into the spatial coordinate system of the RGB-D camera through the spatial transformation matrix calibrated by the RGB-D camera and the medical imaging equipment. The nearest patient model mesh surface m of each point p in the medical image point cloud P is searched, and a pairwise matching relationship is established. The neural network reconstructs the dynamic patient model during surgery in real time, and solves the non-rigid deformation field of medical images by matching relationships. Specifically, the RGB-D camera captures the patient's whole-body optical images and depth information in real time during surgery, and the human pose estimation neural network reconstructs the dynamic patient model in real time. Medical image point cloud Each point in Matched mesh surface The positional change solution calculates the non-rigid deformation field of the medical image and computes all matching meshes. The relative pose transformation matrix from the initial patient model to the new patient model; the relative pose transformation matrices of all matched meshes constitute the deformation field, which is represented as a vector field, where each vector represents the rotation and displacement of a spatial point during the shape transformation process; Register medical images and update the display.

2. The real-time non-rigid registration method for surgical navigation images based on deep learning according to claim 1, characterized in that, The spatial transformation matrix between the calibrated RGB-D camera and the medical imaging equipment is specifically as follows: Select a marker that is clearly visible on both the RGB-D camera and the medical imaging device, ensuring that the marker is within the field of view of both devices; use the RGB-D camera and the medical imaging device to photograph the patient and the marker separately, acquiring data from both imaging devices; The markers are identified, and their spatial coordinates on the RGB-D camera and medical imaging equipment are extracted. The spatial transformation matrix between the RGB-D camera and the medical imaging equipment is calculated using the least squares method.

3. The real-time non-rigid registration method for surgical navigation images based on deep learning according to claim 1, characterized in that, When a patient completes medical imaging, the video frames captured by the RGB-D camera are marked as initial frames. The initial frames are input into a deep learning-based human pose estimation neural network, which outputs SMPL parameters that conform to the patient's body shape and pose. The SMPL then reconstructs the patient's initial model based on the parameters.

4. The real-time non-rigid registration method for surgical navigation images based on deep learning according to claim 1, characterized in that, The human pose estimation neural network has an encoder-decoder structure, including an optical image encoding module, a depth image encoding module, and a decoding module, specifically: The optical image encoding module is used to extract texture and color features from RGB images. These features include the patient's body contour and skin color, which help the human pose estimation neural network to identify the relationship between joint positions and body parts. The depth image encoding module is used to extract human geometric features in three-dimensional space from depth images. These features help determine the relative positions of the patient's joints and body parts in three-dimensional space. The decoding module is used to decode information about human posture and body shape from the encoded and merged features and map it into the parameter space of the SMPL model. The optical image encoding module and the depth image encoding module have the same structure but do not share parameters. The image features extracted by the two encoding modules are merged and input into the fully connected layer of the decoding module, and then the parameters of the SMPL human body model are output. ,in For the patient's global pose parameters, These are the attitude parameters for each part. It includes body shape parameters such as height, weight, and build; The SMPL human body model is a parametric human body model, based on the parameters. SMPL outputs a mesh human body model composed of vertices. .

5. The real-time non-rigid registration method for surgical navigation images based on deep learning according to claim 4, characterized in that, Both the optical image encoding module and the depth image encoding module use ResNet-50, and the decoding module is a fully connected layer.

6. The real-time non-rigid registration method for surgical navigation images based on deep learning according to claim 4, characterized in that, The training method for the human pose estimation neural network is as follows: The training dataset is input into the human pose estimation neural network to obtain the estimated SMPL parameters. SMPL reconstructs the human body model based on parameters. ; The attitude estimation loss function is calculated as follows: in, These are the pose parameters labeled in the dataset; The posture estimation loss function is calculated as follows: in, These are the postural parameters labeled in the dataset; The three-dimensional joint loss is calculated as follows: in, The three-dimensional human joint positions are calculated from the reconstructed SMPL model. The 3D joint positions labeled for the dataset; The two-dimensional joint projection loss is calculated as follows: in, The two-dimensional human joint positions are represented by the orthogonal projection of the three-dimensional joint positions onto the camera plane. Two-dimensional joint locations labeled for the dataset; The complete loss function is calculated as follows: ； The Adam optimizer completes backpropagation and updates the network parameters.

7. The real-time non-rigid registration method for surgical navigation images based on deep learning according to claim 1, characterized in that, The registered medical images are updated and displayed as follows: The displacement and rotation vectors in the calculated non-rigid deformation field are multiplied by the corresponding points in the medical image point cloud to transform each point in the medical image point cloud to a new coordinate system. The medical image point cloud after applying the non-rigid deformation field is consistent with the new patient model in space. During the surgical navigation process, the doctor performs precise operations based on the real-time updated medical images.

8. A real-time non-rigid registration system for surgical navigation images based on deep learning, characterized in that, The method for implementing the real-time non-rigid registration of surgical navigation images based on deep learning as described in any one of claims 1-7 includes a calibration module, a human pose estimation module, a registration initialization module, a non-rigid registration module, and a display module. The calibration module is used to calibrate the spatial transformation matrix between the RGB-D camera and the medical imaging equipment; The human pose estimation module is used to capture real-time optical images and depth information of the patient's whole body using an RGB-D camera, and to reconstruct a patient model based on SMPL in real-time using a human pose estimation neural network. The registration initialization module is used to perform medical imaging of the patient for surgical navigation, and the patient model at the same time is marked as the initial model; and to establish the point-to-surface spatial position matching relationship between the medical image point cloud and the patient initial model; The non-rigid registration module is used to solve the non-rigid deformation field of the medical image based on the matching relationship between the patient model reconstructed in real time during the operation and the medical image, and to register the medical image. The display module is used to register medical images and update the display.

Images