Target organ model registration method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a target organ model registration method and system, electronic equipment and a storage medium, and belongs to the technical field of surgical navigation, and the method comprises the steps: obtaining target organ image data of a patient before an operation, and constructing a corresponding preoperative three-dimensional point cloud model based on the target organ image data; the method comprises the following steps: scanning target organ data of a patient in an operation to obtain image data in the operation, and constructing a corresponding three-dimensional point cloud model in the operation based on the image data in the operation; rigid body transformation parameters of the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model are determined through an ICP registration algorithm, and the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model are registered based on the rigid body transformation parameters. The embodiment of the invention aims at improving the real-time performance and the accuracy of target organ model registration, and further improving the accuracy and the real-time performance of surgical navigation.

Description

Technical Field

[0001] The present application relates to the field of surgical navigation technology, and in particular to a target organ model registration method and system, electronic equipment, and storage medium. Background Art

[0002] Model registration plays a crucial role in medical image analysis, robotic navigation, and intraoperative navigation. Intraoperative navigation involves constructing a 3D model containing detailed information about the patient's organs and tissues through preoperative simulation and planning of the surgical process. During surgery, these models are precisely aligned with the actual surgical scene. This process, known as model registration, is key to achieving precise surgical navigation. As vital organs in the human body, target organs have complex structural characteristics, and their morphology and structure are easily affected by various physiological activities, such as respiratory movement and heartbeat. Therefore, extremely high demands are placed on registration algorithms.

[0003] Traditional registration algorithms struggle to cope with the rapid changes in the target organ's morphology and structure. Consequently, registering the target organ model requires processing large amounts of data and complex computational processes. This results in low registration efficiency, high difficulty, and limited registration accuracy, hindering real-time surgical navigation. Therefore, addressing the complex structural characteristics of the target organ, achieving real-time registration of the target organ model, and improving the accuracy and real-time nature of surgical navigation are critical challenges that need to be addressed in the field of surgical navigation. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a target organ model registration method and system, electronic equipment and storage medium, aiming to improve the real-time and accuracy of target organ model registration, thereby improving the accuracy and real-time performance of surgical navigation.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a target organ model registration method, comprising:

[0006] Acquire preoperative target organ imaging data of the patient, and construct a corresponding preoperative three-dimensional point cloud model based on the target organ imaging data;

[0007] Scanning the target organ data of the patient during surgery to obtain intraoperative imaging data, and constructing a corresponding intraoperative three-dimensional point cloud model based on the intraoperative imaging data;

[0008] The rigid body transformation parameters of the preoperative 3D point cloud model and the intraoperative 3D point cloud model are determined by an ICP registration algorithm, and the preoperative 3D point cloud model and the intraoperative 3D point cloud model are registered based on the rigid body transformation parameters.

[0009] In some embodiments, constructing a corresponding preoperative three-dimensional point cloud model based on the target organ image data is specifically:

[0010] Extracting information from the target organ image data to obtain first image data;

[0011] performing image preprocessing on the first image data to obtain second image data;

[0012] Based on the second image data, a corresponding preoperative three-dimensional point cloud model is generated using the MC algorithm.

[0013] In some embodiments, the scanning of the target organ data during the patient's surgery is specifically: scanning the target organ data during the patient's surgery by a laser radar scanning method or an infrared scanning method.

[0014] In some embodiments, the rigid body transformation parameters of the pre-operative 3D point cloud model and the intra-operative 3D point cloud model are determined by the ICP registration algorithm, specifically:

[0015] Searching for the closest point of each point in the intraoperative three-dimensional point cloud model in the preoperative three-dimensional point cloud model to obtain a plurality of corresponding closest point pairs;

[0016] The average distance between each point in each of the closest point pairs is calculated, and optimization is performed with minimization of the average distance as the objective function to obtain the rigid body transformation parameters of the pre-operative three-dimensional point cloud model and the intra-operative three-dimensional point cloud model, wherein the rigid body transformation parameters include translation parameters and rotation parameters.

[0017] In some embodiments, registering the pre-operative 3D point cloud model with the intra-operative 3D point cloud model based on the rigid body transformation parameters is specifically performed as follows:

[0018] transforming the intraoperative three-dimensional point cloud model based on the rigid body transformation parameters to obtain a corresponding initial target organ registration model;

[0019] Iteratively calculate the average distance between the initial target organ registration model and the preoperative three-dimensional point cloud model until the average distance is less than a preset threshold or a preset number of iterations is met, and output the registered target organ model.

[0020] In some embodiments, after outputting the registered target organ model, the method further includes: projecting the target organ model to the actual target organ position of the patient through AR or MR.

[0021] In some embodiments, the target organ model registration method further includes: auxiliary registration based on preset anatomical landmarks of the target organ itself, QR code marks, and external marks.

[0022] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a target organ model registration system, the system comprising: a first building module, a second building module and a registration module;

[0023] The first construction module is used to obtain preoperative target organ imaging data of the patient and construct a corresponding preoperative three-dimensional point cloud model based on the target organ imaging data;

[0024] The second construction module is configured to scan the target organ data during the patient's surgery to obtain intraoperative imaging data, and to construct a corresponding intraoperative three-dimensional point cloud model based on the intraoperative imaging data;

[0025] The registration module is used to determine the rigid body transformation parameters of the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model through the ICP registration algorithm, and to align the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model based on the rigid body transformation parameters.

[0026] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the target organ model registration method described in the first aspect above.

[0027] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the target organ model registration method described in the first aspect above.

[0028] The target organ model registration method and system, electronic device and storage medium proposed in the present application can convert a two-dimensional image into a three-dimensional image with an intuitive stereoscopic effect by acquiring the target organ imaging data of the patient before surgery and constructing a corresponding preoperative three-dimensional point cloud model based on the target organ imaging data, and can show the three-dimensional structure and morphology of the target organ, so as to more accurately understand the patient's anatomical structure and lesion location; by scanning the target organ data during surgery of the patient to obtain intraoperative imaging data, and constructing a corresponding intraoperative three-dimensional point cloud model based on the intraoperative imaging data, the patient during surgery can be The two-dimensional situation of the target organ is converted into a three-dimensional image with an intuitive three-dimensional effect, which can show the three-dimensional results and morphology of the target organ during surgery, making it convenient to understand the blood vessels that will be encountered in real time; the rigid body transformation parameters of the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model are determined by the ICP registration algorithm, and the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model are registered based on the rigid body transformation parameters. The preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model can be accurately registered, thereby improving the real-time and accuracy of the target organ model registration, thereby improving the accuracy and real-time of surgical navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of a target organ model registration method provided in an embodiment of the present application;

[0030] Figure 2 yes Figure 1 Flowchart of step S101 in FIG.

[0031] Figure 3 yes Figure 1 Flowchart of step S103 in FIG.

[0032] Figure 4 yes Figure 3 Flowchart of step S301 in FIG.

[0033] Figure 5 yes Figure 3 Flowchart of step S302 in FIG.

[0034] Figure 6 Schematic diagram of the structure of a target organ model registration system provided in an embodiment of the present application;

[0035] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0037] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0039] First, let’s analyze some of the terms used in this application:

[0040] Registration refers to the matching of geographic coordinates of different image graphics obtained by different imaging methods within the same area. Specifically in the medical field, this technology can achieve precise alignment of virtual 3D models with the patient's actual anatomy.

[0041] LiDAR is a sensor that measures distance and acquires the geometric shape of objects. It works by scanning a target with a laser beam, recording the return signal after the laser beam interacts with the target, and then calculating the laser beam's time of flight (TOF) and the intensity of the reflected signal to determine the target's position and shape, thereby constructing a point cloud.

[0042] The ICP algorithm is commonly used for point cloud registration. It achieves precise alignment between two point cloud datasets by iteratively minimizing errors. Data analysis involves analyzing, summarizing, understanding, and digesting large amounts of collected data using appropriate statistical analysis methods to maximize its functionality and effectiveness. It is the process of purposefully collecting, organizing, processing, and analyzing data for commercial or other specific purposes to extract valuable information and insights.

[0043] Based on this, the embodiments of the present application provide a target organ model registration method and system, an electronic device and a storage medium, aiming to improve the real-time and accuracy of target organ model registration, thereby improving the accuracy and real-time performance of surgical navigation.

[0044] The target organ model registration method and system, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the target organ model registration method in the embodiments of the present application is described.

[0045] The target organ model registration method provided in the embodiment of the present application relates to the field of surgical navigation technology. The target organ model registration method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can realize the application of the target organ model registration method, etc., but is not limited to the above forms.

[0046] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0047] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0048] Figure 1 This is an optional flowchart of the target organ model registration method provided in an embodiment of the present application, which may include but is not limited to steps S101 to S103.

[0049] Step S101, obtaining preoperative target organ image data of the patient, and constructing a corresponding preoperative three-dimensional point cloud model based on the target organ image data;

[0050] Step S102: Scanning the target organ data of the patient during surgery to obtain intraoperative imaging data, and constructing a corresponding intraoperative three-dimensional point cloud model based on the intraoperative imaging data;

[0051] Step S103 : determining rigid body transformation parameters of the preoperative 3D point cloud model and the intraoperative 3D point cloud model through an ICP registration algorithm, and registering the preoperative 3D point cloud model and the intraoperative 3D point cloud model based on the rigid body transformation parameters.

[0052] The present application proposes a target organ model registration method and system, electronic device and storage medium, which can convert a two-dimensional image into a three-dimensional image with an intuitive stereoscopic effect by acquiring the target organ imaging data of the patient before surgery and constructing a corresponding preoperative three-dimensional point cloud model based on the target organ imaging data, and can display the three-dimensional structure and morphology of the target organ, and can more accurately understand the patient's anatomical structure and lesion location; by scanning the target organ data during surgery of the patient, obtaining intraoperative imaging data, and constructing a corresponding intraoperative three-dimensional point cloud model based on the intraoperative imaging data, the patient's intraoperative The three-dimensional situation of the target organ is converted into a three-dimensional image with an intuitive three-dimensional effect, which can show the three-dimensional results and morphology of the target organ during the operation, and facilitate real-time understanding of the blood vessels to be encountered; the rigid body transformation parameters of the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model are determined by the ICP registration algorithm, and the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model are registered based on the rigid body transformation parameters. The preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model can be accurately registered, thereby improving the real-time and accuracy of the target organ model registration, thereby improving the accuracy and real-time performance of surgical navigation.

[0053] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S203:

[0054] Step S201, extracting information from the target organ image data to obtain first image data;

[0055] Step S202: performing image preprocessing on the first image data to obtain second image data;

[0056] Step S203: Based on the second image data, generate a corresponding preoperative three-dimensional point cloud model using the MC algorithm.

[0057] In step S201 of some embodiments, the patient's target organs, portal veins, arteries, veins, tumors and other parts can be scanned by medical imaging technologies such as, but not limited to, ray projection imaging (SPR, CR, DDR, DSA etc.), CT imaging, ultrasound imaging, magnetic resonance imaging (MRI, MRSI), radionuclide imaging (PET, SPECT, γCamera), and interventional imaging to obtain the patient's target organ image data. After the target organ image data is obtained, image information related to the target organ needs to be extracted from the target organ image data, which may include selecting a specific layer or area containing the target organ, and adjusting image parameters such as contrast and brightness to obtain first image data.

[0058] In step S202 of some embodiments, noise in the first image data may be removed using, but is not limited to, filtering techniques. The filtering techniques may include, but are not limited to, mean filtering, median filtering, and hybrid median filtering. Subsequently, contrast enhancement and sharpening techniques may be used to further clarify the boundary between the target organ and surrounding tissue. Image segmentation algorithms may also be used to separate the target organ from the background. Image segmentation algorithms may include, but are not limited to, threshold segmentation, region growing, and level set methods.

[0059] In step S203 of some embodiments, first, the preprocessed second image data (usually a two-dimensional tomographic image sequence) is used as the input of the MC algorithm. Next, the MC algorithm traverses each voxel (three-dimensional pixel) and determines the position of the isosurface inside the voxel based on the grayscale value of the voxel and the grayscale value change of its adjacent voxels. Then, a three-dimensional mesh model is generated by connecting these isosurfaces. Finally, the generated three-dimensional mesh model is smoothed to eliminate possible step effects or non-smooth areas. At the same time, color, texture and other rendering processing can also be performed as needed to finally obtain the corresponding preoperative three-dimensional point cloud model.

[0060] In step S101 shown in the embodiment of the present application, by acquiring the patient's preoperative target organ imaging data and constructing a corresponding preoperative three-dimensional point cloud model based on the target organ imaging data, the two-dimensional image can be converted into a three-dimensional image with an intuitive three-dimensional effect, the three-dimensional structure and morphology of the target organ can be displayed, and the patient's anatomical structure and lesion location can be understood more accurately.

[0061] In some embodiments, in step S102, after obtaining the preoperative 3D point cloud model, it can be imported into AR / MR development software (OpenGL, Open Inventor, Unity) to render and display 3D images of the target organ, portal vein, artery, vein, and tumor. This eliminates the need for the surgeon to perform complex image synthesis and spatial visualization of 2D scanned images of the target organ. Furthermore, the location of the target organ lesion and its spatial proximity to the vessels within the target organ can be intuitively, clearly, and multi-dimensionally displayed. Subsequently, the patient's intraoperative target organ data can be scanned using, but not limited to, a laser radar scanning method or an infrared scanning method to obtain intraoperative image data, and a corresponding intraoperative 3D point cloud model is constructed based on the intraoperative image data. The laser radar scans the patient's intraoperative target organ with a laser beam, records the echo signal after the laser beam interacts with the target organ, and then determines the position and shape of the target organ by calculating the laser beam's propagation time and the intensity of the reflected signal. This point cloud is then constructed to obtain the true 3D point cloud data of the target organ, effectively resolving the problem of inaccurate registration caused by the movement of organs during surgery with the patient's respiration. Among them, the infrared laser and the binocular camera can be combined into a scanning pillar. First, during the scanning process, the laser emitter projects the invisible infrared laser onto the patient's target organ, and then the binocular camera synchronously collects the line laser reflected on the surface of the target organ being measured. Then, the relevant algorithm is used to calculate the depth information of the target organ surface, and further generate a three-dimensional point cloud model of the target organ during surgery.

[0062] In step S102 shown in the embodiment of the present application, intraoperative imaging data is obtained by scanning the target organ data of the patient during surgery, and a corresponding intraoperative three-dimensional point cloud model is constructed based on the intraoperative imaging data. The two-dimensional situation of the patient's target organ during surgery can be converted into a three-dimensional image with an intuitive three-dimensional effect, which can display the three-dimensional results and morphology of the target organ during surgery, making it convenient to understand the blood vessels that are about to be encountered in real time.

[0063] See also Figure 3 In some embodiments, step S103 may include but is not limited to steps S301 to S302:

[0064] Step S301, determining rigid body transformation parameters of the pre-operative 3D point cloud model and the intra-operative 3D point cloud model by using an ICP registration algorithm;

[0065] Step S302 : registering the pre-operative 3D point cloud model with the intra-operative 3D point cloud model based on the rigid body transformation parameters.

[0066] See also Figure 4 In some embodiments, step S301 may include but is not limited to steps S401 to S402:

[0067] Step S401, searching for the closest point of each point in the intraoperative 3D point cloud model to the preoperative 3D point cloud model, and obtaining a plurality of corresponding closest point pairs;

[0068] Step S402, calculate the average distance between each point in each of the nearest point pairs, and optimize with minimization of the average distance as the objective function to obtain the rigid body transformation parameters of the pre-operative three-dimensional point cloud model and the intra-operative three-dimensional point cloud model, wherein the rigid body transformation parameters include translation parameters and rotation parameters.

[0069] In step S401 of some embodiments, first, it is possible but not limited to select the axis with the largest variance as the splitting axis of the pre-operative three-dimensional point cloud model through the variance of each point in the pre-operative three-dimensional point cloud model in the three directions of x, y, and z, and perform quick sorting in the direction of the splitting axis, and then find the median, use the median as the boundary, recursively generate the left subtree for points less than the median, and recursively generate the right subtree for points greater than the median, and finally return to the root node, and determine the corresponding KD tree based on the left and right subtrees and the root node. Then, search the KD tree for the nearest point of each point in the pre-operative three-dimensional point cloud model, and obtain several corresponding nearest point pairs based on all the nearest points found. It should be noted that the nearest point of each point in the pre-operative three-dimensional point cloud model can also be searched through data structures such as octrees, and this application does not limit this.

[0070] In step S402 of some embodiments, the average distance between each point in each of the closest point pairs is calculated, and optimization is performed with minimization of the average distance as the objective function, where the objective function is: Where R is the rotation parameter, T is the translation parameter, pi is the point position in the preoperative 3D point cloud model, {Pi|Pi∈R3, i=1, 2, ..., N}, q i For the point positions in the intraoperative three-dimensional point cloud model, the rigid body transformation parameters of the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model are obtained by optimizing the average distance in the objective function as the target, wherein the rigid body transformation parameters include translation parameters and rotation parameters.

[0071] It should be noted that the rigid body transformation parameters of the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model can also be determined by the following method. Specifically: first, the center of mass of the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model are calculated respectively, and the center of mass is the average position of all points in the point set; secondly, each point in the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model is subtracted from its corresponding center of mass to obtain a new point set, and the center of mass of these new point sets is located at the origin. Finally, the corresponding covariance matrix is ​​constructed using the first centered point set and the second centered point set, wherein the covariance matrix reflects the rotation relationship between the two point sets, and the covariance matrix is ​​subjected to singular value decomposition (SVD) to obtain the corresponding rotation parameters, and the translation parameters are obtained based on the rotation parameters, the first center of mass and the second center of mass.

[0072] In step S301 shown in the embodiment of the present application, the rigid body transformation parameters of the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model are determined through the ICP registration algorithm, so as to facilitate the subsequent precise registration of the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model based on the rigid body transformation parameters.

[0073] See also Figure 5 In some embodiments, step S302 may include but is not limited to steps S501 to S502:

[0074] Step S501, transforming the intraoperative three-dimensional point cloud model based on the rigid body transformation parameters to obtain a corresponding initial target organ registration model;

[0075] Step S502 , iteratively calculating the average distance between the initial target organ registration model and the preoperative three-dimensional point cloud model until the average distance is less than a preset threshold or a preset number of iterations is met, and outputting the registered target organ model.

[0076] In step S501 of some embodiments, after obtaining the rigid body transformation parameters, i.e., the rotation parameters and the translation parameters, each point in the intraoperative three-dimensional point cloud model is rotated using the rotation parameters, and the translation parameters are added to the rotated point set to obtain the corresponding initial target organ registration model.

[0077] In step S502 of some embodiments, during the iterative process of the ICP algorithm, an upper limit on the number of iterations and a preset error threshold are typically customized. After each iteration, the average distance between the newly transformed model, i.e., the initial target organ registration model, and the preoperative three-dimensional point cloud model is calculated. If the average distance is less than the preset error threshold, or the number of iterations reaches the upper limit, the algorithm stops iterating and outputs the final rigid body transformation parameters and the target organ model transformed based on the final rigid body transformation parameters, thereby achieving accurate registration of the preoperative three-dimensional point cloud model with the intraoperative three-dimensional point cloud model.

[0078] In step S302 shown in the embodiment of the present application, the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model can be accurately aligned through rigid body transformation parameters, thereby improving the real-time and accuracy of the target organ model alignment, and further improving the accuracy and real-time performance of surgical navigation.

[0079] In some embodiments, after step S502, the target organ model is projected onto the actual target organ position of the patient through AR or MR. Specifically, the projection device corresponding to AR or MR is calibrated, and the real-time tracking and positioning functions of AR or MR are used to ensure that the successfully registered target organ model is accurately projected onto the patient's actual organ and surgical site. The doctor can interact with the target organ model through the AR / MR device, such as rotating, scaling, and labeling, so as to better understand the situation of the surgical site.

[0080] The embodiment of the present application also includes auxiliary registration based on preset anatomical landmark points of the target organ itself, QR code marks and external marks. Specifically, by selecting points with obvious anatomical features on the target organ as anatomical landmark points, such as the liver portal, gallbladder bed, inferior vena cava, etc., in the subsequent registration process, the position coordinates of the anatomical landmark points of the real target organ are detected through image recognition technology, and the virtual anatomical landmark points can be aligned with the anatomical landmark points at the corresponding positions of the real target organ with the help of the mark points. Through calculation and optimization, the precise registration of the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model is achieved. In addition, by anchoring the preoperative 3D point cloud model on a QR code and having the surgeon hold the QR code during the operation, when the surgeon moves the QR code, the position of the preoperative 3D point cloud model will also change accordingly. By continuously moving the position of the QR code, the preoperative 3D point cloud model anchored thereon is made to coincide with the position of the intraoperative 3D point cloud model. The surgeon then confirms the successful registration through voice, gestures, buttons, and other instructions. After success, the QR code can be removed. At this time, the preoperative 3D point cloud model and the intraoperative 3D point cloud model are in the same spatial position, and the superposition and fusion of the preoperative 3D point cloud model and the intraoperative 3D point cloud model can be achieved. Of course, a special registration marker, such as a cross, can also be installed outside the patient's body. Before the operation, the marker also needs to be used for 2D image data acquisition and 3D model reconstruction. Then, during the operation, the preoperative 3D point cloud model and the intraoperative 3D point cloud model marker can be matched to complete the registration of the preoperative 3D point cloud model and the intraoperative 3D point cloud model. In this way, by detecting the position coordinates of one or more anatomical landmark points of the real target organ, one or more virtual anatomical landmark points are aligned to one or more anatomical landmark points at the corresponding positions of the real target organ, thereby assisting in the alignment of the virtual three-dimensional model and the real target organ and improving the efficiency of the alignment.

[0081] This embodiment also includes that during the specific implementation of orthopedic surgery, first, CT is used to perform fine imaging of the bone structure, MRI is used to assess soft tissue and ligament injuries, and X-rays are used to achieve rapid positioning during the operation to obtain multimodal imaging data. These image data are input into a processing system equipped with a deep learning algorithm (such as Mask R-CNN), and the fracture line, bone fragments, blood vessels and nerve bundles are automatically segmented to generate a dynamic and interactive bone three-dimensional model. During the operation, the image data obtained by the C-arm / O-arm are integrated in real time, and the above-mentioned three-dimensional model is dynamically updated to correct the bone displacement or reduction deviation caused by the operation during the operation. Using the bony landmarks on the patient's body, such as the iliac crest and the greater trochanter of the femur, in conjunction with the intraoperative optical / electromagnetic tracking system, the generated three-dimensional model is aligned with the patient's real anatomical structure in real time to ensure that the alignment accuracy reaches the millimeter level. Based on the aligned model, a digital twin system is constructed, and methods such as finite element analysis are used to simulate the biomechanical stability after fracture reduction, evaluate the stress distribution of internal fixation schemes such as steel plates and screws, and predict and avoid the risk of secondary fractures after surgery in advance. With the help of an algorithm, the fracture reduction path is optimized by comprehensively considering multiple parameters such as the spatial position of bone fragments, soft tissue tension, and neurovascular avoidance, generating the optimal reduction sequence and internal fixation implantation angle. In a virtual environment, reduction operations such as traction and rotation are simulated, and the feasibility of reduction and joint matching, such as the fit of the ankle joint surface, are verified using a physics engine. The doctor issues gesture commands through the AR / MR device to control an orthopedic robotic arm of a type with 7 degrees of freedom. The robotic arm guides the reduction of the bone fragments in real time and locks them in the target position. During the reduction process, the robotic arm provides tactile feedback to the doctor, enabling him to sense the reduction resistance. After the reduction is completed, the robotic arm assists in high-precision internal fixation operations, such as screw placement. At the same time, the AR interface synchronously displays the screw track angle and depth information, as well as risk warnings for adjacent nerves and blood vessels, to ensure the safety and accuracy of the surgical operation.

[0082] The target organ model registration method and system, electronic device and storage medium proposed in the present application can convert a two-dimensional image into a three-dimensional image with an intuitive stereoscopic effect by acquiring the target organ imaging data of the patient before surgery and constructing a corresponding preoperative three-dimensional point cloud model based on the target organ imaging data, and can show the three-dimensional structure and morphology of the target organ, so as to more accurately understand the patient's anatomical structure and lesion location; by scanning the target organ data during surgery of the patient to obtain intraoperative imaging data, and constructing a corresponding intraoperative three-dimensional point cloud model based on the intraoperative imaging data, the patient during surgery can be The two-dimensional situation of the target organ is converted into a three-dimensional image with an intuitive three-dimensional effect, which can show the three-dimensional results and morphology of the target organ during surgery, making it convenient to understand the blood vessels that will be encountered in real time; the rigid body transformation parameters of the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model are determined by the ICP registration algorithm, and the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model are registered based on the rigid body transformation parameters. The preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model can be accurately registered, thereby improving the real-time and accuracy of the target organ model registration, thereby improving the accuracy and real-time of surgical navigation.

[0083] See also Figure 6 , the embodiment of the present application also provides a target organ model registration system, which can implement the above-mentioned target organ model registration method, and the system further includes: a first construction module 100, a second construction module 200 and a registration module 300;

[0084] The first construction module 100 is used to obtain preoperative target organ image data of the patient and construct a corresponding preoperative three-dimensional point cloud model based on the target organ image data;

[0085] The second construction module 200 is used to scan the target organ data of the patient during surgery to obtain intraoperative imaging data, and to construct a corresponding intraoperative three-dimensional point cloud model based on the intraoperative imaging data;

[0086] The registration module 300 is used to register the pre-operative 3D point cloud model with the intra-operative 3D point cloud model using an ICP registration algorithm.

[0087] The specific implementation of the target organ model registration system is basically the same as the specific embodiment of the above-mentioned target organ model registration method, and will not be repeated here.

[0088] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the target organ model registration method. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0089] See also Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0090] The processor 701 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0091] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 702 and are called by the processor 701 to execute the target organ model registration method of the embodiments of this application.

[0092] Input / output interface 703, used to implement information input and output;

[0093] Communication interface 704, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0094] Bus 705 , which transmits information between various components of the device (e.g., processor 701 , memory 702 , input / output interface 703 , and communication interface 704 );

[0095] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0096] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned target organ model registration method is implemented.

[0097] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0098] The target organ model registration method and system, electronic device and storage medium proposed in the present application can convert a two-dimensional image into a three-dimensional image with an intuitive stereoscopic effect by acquiring the target organ imaging data of the patient before surgery and constructing a corresponding preoperative three-dimensional point cloud model based on the target organ imaging data, and can show the three-dimensional structure and morphology of the target organ, so as to more accurately understand the patient's anatomical structure and lesion location; by scanning the target organ data during surgery of the patient to obtain intraoperative imaging data, and constructing a corresponding intraoperative three-dimensional point cloud model based on the intraoperative imaging data, the patient during surgery can be The two-dimensional situation of the target organ is converted into a three-dimensional image with an intuitive three-dimensional effect, which can show the three-dimensional results and morphology of the target organ during surgery, making it convenient to understand the blood vessels that will be encountered in real time; the rigid body transformation parameters of the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model are determined by the ICP registration algorithm, and the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model are registered based on the rigid body transformation parameters. The preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model can be accurately registered, thereby improving the real-time and accuracy of the target organ model registration, thereby improving the accuracy and real-time of surgical navigation.

[0099] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0100] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0102] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0103] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0104] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0106] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0107] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0109] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A target organ model registration method, characterized in that: include: Acquire preoperative target organ imaging data of the patient, and construct a corresponding preoperative three-dimensional point cloud model based on the target organ imaging data; Scanning the target organ data of the patient during surgery to obtain intraoperative imaging data, and constructing a corresponding intraoperative three-dimensional point cloud model based on the intraoperative imaging data; The rigid body transformation parameters of the preoperative 3D point cloud model and the intraoperative 3D point cloud model are determined by an ICP registration algorithm, and the preoperative 3D point cloud model and the intraoperative 3D point cloud model are registered based on the rigid body transformation parameters.

2. The target organ model registration method according to claim 1, characterized in that: The constructing of the corresponding preoperative three-dimensional point cloud model based on the target organ image data is specifically as follows: Extracting information from the target organ image data to obtain first image data; performing image preprocessing on the first image data to obtain second image data; Based on the second image data, a corresponding preoperative three-dimensional point cloud model is generated using the MC algorithm.

3. The target organ model registration method according to claim 1, characterized in that: The scanning of the target organ data during the patient's operation is specifically: scanning the target organ data during the patient's operation by a laser radar scanning method or an infrared scanning method.

4. The target organ model registration method according to claim 1, characterized in that: The rigid body transformation parameters of the pre-operative 3D point cloud model and the intra-operative 3D point cloud model are determined by the ICP registration algorithm, specifically: Searching for the closest point of each point in the intraoperative three-dimensional point cloud model in the preoperative three-dimensional point cloud model to obtain a plurality of corresponding closest point pairs; The average distance between each point in each of the nearest point pairs is calculated, and optimization is performed with minimization of the average distance as the objective function to obtain the rigid body transformation parameters of the pre-operative three-dimensional point cloud model and the intra-operative three-dimensional point cloud model, wherein the rigid body transformation parameters include translation parameters and rotation parameters.

5. The target organ model registration method according to claim 1, characterized in that: The registering of the pre-operative 3D point cloud model with the intra-operative 3D point cloud model based on the rigid body transformation parameters is specifically as follows: transforming the intraoperative three-dimensional point cloud model based on the rigid body transformation parameters to obtain a corresponding initial target organ registration model; Iteratively calculate the average distance between the initial target organ registration model and the preoperative three-dimensional point cloud model until the average distance is less than a preset threshold or a preset number of iterations is met, and output the registered target organ model.

6. The target organ model registration method according to claim 5, characterized in that: After outputting the registered target organ model, the method further includes: projecting the target organ model to the actual target organ position of the patient through AR or MR.

7. The target organ model registration method according to claim 1, characterized in that: Also includes: Assisted registration is based on preset anatomical landmarks of the target organ, QR code markers and external markers.

8. A target organ model registration system, characterized in that: The system comprises: a first building module, a second building module and a registration module; The first construction module is used to obtain preoperative target organ imaging data of the patient and construct a corresponding preoperative three-dimensional point cloud model based on the target organ imaging data; The second construction module is configured to scan the target organ data during the patient's surgery to obtain intraoperative imaging data, and to construct a corresponding intraoperative three-dimensional point cloud model based on the intraoperative imaging data; The registration module is used to determine the rigid body transformation parameters of the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model through the ICP registration algorithm, and to align the preoperative three-dimensional point cloud model and the intraoperative three-dimensional point cloud model based on the rigid body transformation parameters.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the target organ model registration method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the target organ model registration method according to any one of claims 1 to 7 is implemented.

Images