Target registration method, device, equipment, storage medium and program product

Abstract

The present application discloses a target registration method, apparatus, device, storage medium and program product, which relates to the field of artificial intelligence technology. The method comprises: obtaining an image to be registered and a reference image containing the same target; obtaining an initial registration relationship between the image to be registered and the reference image based on the pose matrix of the target in the image to be registered and the pose matrix of the target in the reference image; obtaining a preliminary registration result corresponding to the target in the registration image based on the initial registration relationship and the reference image; obtaining an optimized registration relationship between the image to be registered and the reference image based on the effective area of ​​the depth image corresponding to the image to be registered and the preliminary registration result; adjusting the preliminary registration result based on the optimized registration relationship to obtain an optimized registration result corresponding to the target in the registration image, and the optimized registration result is used to guide the object to process the target. The present application optimizes the preliminary registration result through the depth image, which can improve the registration accuracy of the target.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a target registration method, apparatus, device, storage medium, and program product. Background Art

[0002] In the field of surgical navigation, there are surgical navigation applications targeting various human body parts, such as the head, brain, thigh, femur, abdomen, lungs, etc. Among them, the registration of the target (such as a human body part) has a significant impact on surgical navigation.

[0003] Taking the head as an example, existing technologies manually mark key points on the head in preoperative images and on the actual head during surgery. Then, a registration algorithm is used to align the head based on these key points to achieve navigation during head surgery. However, this manual operation introduces human error, resulting in insufficient registration accuracy. Summary of the Invention

[0004] The embodiments of the present application provide a target registration method, apparatus, device, storage medium, and program product, which can improve the accuracy of target registration. The technical solution may include the following contents.

[0005] According to one aspect of an embodiment of the present application, a target registration method is provided, the method comprising:

[0006] Acquire an image to be registered and a reference image containing the same object;

[0007] Detecting the image to be registered to obtain a pose matrix of the target in the image to be registered, wherein the pose matrix is ​​used to represent the position and posture of the target in the image;

[0008] Based on the pose matrix of the target in the image to be registered and the pose matrix of the target in the reference image, obtaining an initial registration relationship between the image to be registered and the reference image; wherein the initial registration relationship is used to preliminarily characterize the transformation relationship between the image to be registered and the reference image;

[0009] Obtaining a preliminary registration result corresponding to the target in the image to be registered according to the initial registration relationship and the reference image;

[0010] Based on the effective area of ​​the depth image corresponding to the image to be registered and the preliminary registration result, obtaining an optimized registration relationship between the image to be registered and the reference image; wherein the effective area of ​​the depth image includes the target;

[0011] According to the optimized registration relationship, the preliminary registration result is adjusted to obtain an optimized registration result corresponding to the target in the registered image; wherein the optimized registration result is used to guide the subject to process the target.

[0012] According to one aspect of an embodiment of the present application, a target registration device is provided, the device comprising:

[0013] A registration image acquisition module is used to acquire an image to be registered and a reference image containing the same object;

[0014] A pose matrix acquisition module is used to detect the image to be registered and obtain a pose matrix of the target in the image to be registered, wherein the pose matrix is ​​used to represent the position and posture of the target in the image;

[0015] an initial relationship acquisition module, configured to acquire an initial registration relationship between the image to be registered and the reference image based on the pose matrix of the target in the image to be registered and the pose matrix of the target in the reference image; wherein the initial registration relationship is used to preliminarily characterize the transformation relationship between the image to be registered and the reference image;

[0016] A preliminary result acquisition module, configured to acquire a preliminary registration result corresponding to the target in the image to be registered based on the initial registration relationship and the reference image;

[0017] an optimized relationship acquisition module, configured to acquire an optimized registration relationship between the image to be registered and the reference image based on a valid area of ​​the depth image corresponding to the image to be registered and the preliminary registration result; wherein the valid area of ​​the depth image includes the target;

[0018] The optimization result acquisition module is used to adjust the preliminary registration result according to the optimized registration relationship and obtain the optimized registration result corresponding to the target in the registered image; wherein the optimized registration result is used to guide the object to process the target.

[0019] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned target registration method.

[0020] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned target registration method.

[0021] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described object registration method.

[0022] The technical solution provided by the embodiments of the present application includes at least the following beneficial effects.

[0023] The target is initially registered based on its pose matrix in the image to be registered and its pose matrix in the reference image. This initial registration result is then optimized based on the effective area of ​​the depth image corresponding to the image to be registered and the initial registration result. This results in an optimized registration result corresponding to the target, achieving fully automated target registration and avoiding the introduction of human error, thereby improving target registration accuracy. Furthermore, the initial registration result is optimized based on the effective area of ​​the depth image, effectively reducing the registration error in the initial registration result and further improving target registration accuracy.

[0024] In addition, since automatic registration of the target can be achieved based on the image to be registered and the reference image, and then navigation of the target can be achieved, there is no need to use expensive large-scale equipment to navigate the target (such as surgical navigation of the target), which greatly improves the ease and convenience of target navigation and reduces the cost of target navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 This is a schematic diagram of an implementation environment for a solution provided by an embodiment of the present application;

[0027] Figure 2 is a flow chart of a target registration method provided by one embodiment of the present application;

[0028] Figure 3 This is a flowchart of a method for obtaining a pose matrix provided by an embodiment of the present application;

[0029] Figure 4 is a schematic diagram of common head postures provided by one embodiment of the present application;

[0030] Figure 5This is a schematic diagram of a coordinate diagram and a heat map of key points provided by an embodiment of the present application;

[0031] Figure 6 Schematic diagram of a rotation matrix detection module provided by one embodiment of the present application;

[0032] Figure 7 is a flow chart of a target registration method provided by another embodiment of the present application;

[0033] Figure 8 is a schematic diagram of effective area segmentation provided by an embodiment of the present application;

[0034] Figure 9 is a schematic diagram of a semi-automatic registration method provided by an embodiment of the present application;

[0035] Figure 10 This is a schematic diagram of an implementation environment for a surgical navigation scenario provided by an embodiment of the present application;

[0036] Figure 11 is a schematic diagram of a method for using a surgical navigation system in a surgical navigation scenario provided by one embodiment of the present application;

[0037] Figure 12 is a schematic diagram of a target registration method in a surgical navigation scenario provided by an embodiment of the present application;

[0038] Figure 13 is a schematic diagram of a preliminary registration head model provided by an embodiment of the present application;

[0039] Figure 14 is a schematic diagram of an optimized registration head model provided by one embodiment of the present application;

[0040] Figure 15 This is a schematic diagram of rendering views under different viewing angles and display options provided by an embodiment of the present application;

[0041] Figure 16 is a block diagram of an object registration device provided by one embodiment of the present application;

[0042] Figure 17 is a block diagram of a target registration device provided by another embodiment of the present application;

[0043] Figure 18 It is a structural diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0045] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0046] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0047] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying and measuring objects, and then further processing the images to make them more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, and map construction.

[0048] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0049] The technical solution provided in the embodiments of the present application involves computer vision technology and machine learning technology of artificial intelligence. Computer vision technology is used to obtain the effective area, target, key points of the target and rotation matrix of the target corresponding to the image, and then machine learning technology is used to train the effective area segmentation model, target detection model, key point detection model and rotation matrix detection model based on the effective area, target, key points of the target and rotation matrix of the target.

[0050] In the methods provided in the embodiments of the present application, the execution entity of each step may be a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. The computer device may be a terminal such as a PC (Personal Computer), a tablet computer, a smartphone, a wearable device, an intelligent robot, or a vehicle-mounted device; it may also be a server. The server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0051] The technical solutions provided by the embodiments of this application are applicable to any scenario requiring target registration, such as surgical navigation, target authentication, target registration, smart transportation, assisted driving, image analysis, and safety detection. The technical solutions provided by the embodiments of this application can improve the accuracy of target registration.

[0052] For an example, see Figure 1 , which shows a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application. The implementation environment of the solution can be realized as the architecture of a target registration system. The implementation environment may include: a terminal 10 and a server 20.

[0053] The terminal 10 may be an electronic device such as a mobile phone, tablet computer, PC (Personal Computer), wearable device, intelligent robot, etc. A client of a target application may be installed in the terminal 10. The target application may be a surgical navigation application, an object registration application, an image registration application, a navigation application, a simulation learning application, a safety detection application, or any other application that can be used for target registration, and the present embodiment of the application does not limit this.

[0054] The server 20 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. The server 20 is used to provide backend services for the client of the target application in the terminal 10. For example, the server 20 may be the backend server of the target application (e.g., a surgical navigation application).

[0055] The terminal 10 and the server 20 can communicate with each other via the network 30 .

[0056] For example, refer to Figure 1 After acquiring the image to be registered and the reference image, the client of the target application in terminal 10 detects the image to be registered, obtains the pose matrix corresponding to the target in the image to be registered, and obtains the pose matrix of the reference model corresponding to the target constructed based on the reference image. Then, a preliminary registration is performed based on the two pose matrices to obtain a preliminary registration result corresponding to the target. Then, an effective area of ​​the depth image corresponding to the image to be registered is obtained, and optimized registration is performed based on the effective area and the preliminary registration result to obtain an optimized registration result corresponding to the target in the image to be registered. This optimized registration result is then displayed to the subject (i.e., the user).

[0057] Optionally, the above-mentioned target registration process can also be executed in the server 20. After obtaining the image to be registered, the depth image corresponding to the image to be registered, and the reference image, the server 20 performs the above-mentioned two-stage target registration to obtain the optimized registration result corresponding to the target in the image to be registered, and then sends the optimized registration result to the client of the target application for display to the object.

[0058] Please refer to Figure 2 , which shows a flow chart of a target registration method provided by an embodiment of the present application. The execution subject of each step of the method can be Figure 1 In the terminal 10 or server 20 in the implementation environment of the solution shown, the method may include the following steps (201-206).

[0059] Step 201: Acquire an image to be registered and a reference image containing the same object.

[0060] The embodiments of this application do not limit the target, and different scenarios may correspond to different targets. For example, in a surgical navigation scenario, the target may indicate the head, brain, face, femur, abdomen, lungs, etc. In a smart transportation scenario, the target may refer to a license plate, traffic sign, vehicle, road, bridge, etc. In a target registration scenario, the target may refer to a person, a part of a person, an animal, a part of an animal, an object, etc.

[0061] The image to be registered refers to an image that includes a target and the target needs to be registered. The image to be registered can be acquired in real time or in advance, and the embodiments of the present application do not limit this. For example, taking the surgical navigation scene as an example, the image to be registered can refer to an intraoperative image including the target acquired in real time during surgery. For example, a video capture device or an image acquisition device can be used to capture the head in real time to obtain the image to be registered. Optionally, target registration can be performed in real time for a temporally continuous sequence of images to be registered (such as a surgical navigation scene), or target registration can be performed only for a set of images including the target (such as a target authentication scene), and the embodiments of the present application do not limit this. The image to be registered can refer to an RGB (Red-Green-Blue, red, green and blue) image, that is, a color image.

[0062] A reference image refers to an image that includes a target and is used to register with the target in the image to be registered. The target in the reference image can serve as a registration reference for the target in the image to be registered. For example, the reference image can refer to an image that includes the target and is acquired before the image to be registered. For example, in a surgical navigation scenario, the reference image can refer to a preoperative image that includes the target and is acquired before surgery, such as a CT (Computed Tomography) scan, an MRI (Magnetic Resource Imaging), or an RGB image extracted from a preoperative image.

[0063] Optionally, the image to be registered can be an image of the target taken at any shooting angle, and the reference image can refer to an image corresponding to the target in a standard posture, that is, the embodiment of the present application supports registration of targets at any angle.

[0064] In step 202 , the image to be registered is detected to obtain a pose matrix of the target in the image to be registered. The pose matrix is ​​used to represent the position and pose of the target in the image.

[0065] The pose of the target can include two parts of transformation: translation and rotation. In the embodiment of the present application, the pose matrix can include two parts: translation vector and rotation matrix. Among them, the translation vector is used to represent the position of the target in the image, and the rotation matrix is ​​used to represent the pose of the target in the image. For example, P1∈R 3×4 Represents the pose matrix of the target, P1∈R 3×4 The elements in are composed of translation vectors and rotation matrices. The translation vector can be represented by a three-dimensional vector (tdx, tdy, tdz). For example, the translation vector can refer to the translation vector of the target's coordinates relative to the origin of the world coordinate system. The rotation matrix can be represented by R 3×3Indicates that the elements in R may refer to the rotation angles corresponding to the target. Optionally, the pose matrix of the target may be constructed based on the world coordinate system.

[0066] In an example, the above detection may include three detection processes: target detection, key point detection, and rotation matrix detection. Then step 202 may further include the following steps.

[0067] Step 202a: perform target detection on the image to be registered to obtain a bounding box corresponding to the target.

[0068] Optionally, the target detection model can be used to perform target detection on the image to be registered to obtain a bounding box (i.e., a Bounding Box) corresponding to the target. The bounding box is used to represent the predicted area of ​​the target in the image to be registered.

[0069] For example, by inputting the image to be registered into the target detection model, multiple bounding boxes corresponding to the image to be registered and the confidence level corresponding to each bounding box can be obtained. The multiple bounding boxes can be first screened based on the confidence level to obtain candidate bounding boxes, and then the bounding box corresponding to the target can be obtained from the candidate bounding boxes. For example, taking single-target registration as an example, since only the bounding box of one target needs to be obtained, the bounding boxes with a confidence level greater than 0.5 can be first screened out, and then the bounding box with the maximum confidence level among the candidate bounding boxes can be determined as the bounding box corresponding to the target. For multiple targets, the bounding boxes corresponding to each target can be obtained first, and then the screening process can be performed.

[0070] The target detection model is a neural network for detecting targets. Optionally, the target detection model can be constructed based on a lightweight SSD (Single Shot Detector, a target detection network), or based on other detection models such as YOLO (You Only Look Once, a target detection network), RetinaFace (a face detection network), CNN (Convolutional Neural Network, convolutional neural network), R-CNN (Region-CNN, region-based convolutional neural network), Faster R-CNN, etc., which are not limited in this embodiment of the present application.

[0071] In one example, the target detection model can be trained using a larger dataset (such as a large number of sample images of targets at any perspective) and data enhancement to obtain a target detection model that can be used to detect targets at any perspective.

[0072] For example, let’s take the surgical navigation scenario as an example. Since surgical navigation is mainly performed during the operation in the hospital, and the surgical navigation scenario is very different from the common natural scene, in the natural scene, the target is generally facing the image acquisition device, so the conventional target detection algorithm can be used. However, in the surgical navigation scenario, the relative position between the target and the camera is more complicated. For example, Figure 4 Taking the head as an example, common head postures include posture 401: head facing the camera from the side, posture 402: head facing the camera from the top, posture 403: head facing the camera from the front, and posture 404: head facing the camera from the back. Depending on the location of the surgery, the head may be rotated at any angle during surgery.

[0073] After acquiring a large number of sample images of targets at arbitrary perspectives, random data augmentation is first performed on each image sample to obtain a data-augmented image sample. This data augmentation may include random flipping, random 360-degree rotation, random stretching, and random scaling. The target detection model then obtains the bounding boxes corresponding to the data-augmented image samples. Based on the bounding boxes and / or confidence levels, a training loss for the target detection model is constructed. The target detection model is then iteratively trained based on the training loss to obtain a trained target detection model.

[0074] Step 202b: based on the bounding box corresponding to the target, a captured image including the target is captured from the image to be registered.

[0075] Optionally, the bounding box may be enlarged to obtain an enlarged bounding box; and based on the enlarged bounding box, a captured image including the target is captured from the image to be registered.

[0076] For example, the side length of the bounding box can be proportionally enlarged first, such as 1.1 times, 1.2 times, 1.3 times, etc. Then, the area corresponding to the enlarged bounding box is cut out from the image to be registered to obtain a cutout image. During the training process of the target detection model, the target detection model can also be trained with the enlarged bounding box, which is conducive to improving the robustness of the target detection model. In addition, key point detection and rotation matrix detection based on the screenshot image can improve the accuracy of key point detection and rotation matrix detection, thereby improving the registration accuracy of the target.

[0077] Step 202c: perform key point detection on the captured image to obtain target key points corresponding to the target.

[0078] Optionally, at least one key point is required to calculate the translation vector corresponding to the target. For example, taking the head in a surgical navigation scenario as an example, since the head posture varies greatly between different surgeries or within the same surgery, to avoid situations where key points are not detected, this embodiment of the application detects at least the following five key points: the left and right corners of the eyes, the tip of the nose, and the left and right corners of the mouth. Optionally, additional key points such as the ears, chin, and lower jaw may be detected, but this embodiment of the application does not limit this.

[0079] The keypoint with the highest confidence among the multiple keypoints can be determined as the target keypoint. For example, when the head is facing the camera head-on, the confidence corresponding to the corner of the eye is higher, so the corner of the eye can be determined as the target keypoint. When the head is facing the camera sideways, the confidence corresponding to the tip of the nose is higher, so the tip of the nose can be determined as the target keypoint.

[0080] In one example, a key point detection model can be used to perform key point detection on a captured image to obtain multiple key points corresponding to the target, and then a target key point corresponding to the target can be determined from the multiple key points based on confidence.

[0081] The keypoint detection model is a neural network used to detect keypoints. This model can be built based on segmentation networks such as the Keypoint Heatmap network (with a 2D Unet as the backbone), the DeepLab network (a semantic segmentation network), and the Hourlass network.

[0082] For example, the key point detection model can output a heat map of key points corresponding to the target. Figure 5 The key point detection model can output a key point map 501 in the form of coordinates corresponding to the target, and / or a key point heat map 502. The target key points are obtained based on the key point map 501 or the key point heat map 502. Compared to performing a regression network algorithm on only designated feature points, heat map-based key point detection can obtain the confidence level of each key point. By comparing the confidence levels, it can be determined whether the key point or target key point exists.

[0083] Optionally, during the training process of the key point detection model, random data augmentation can be performed on each image sample to obtain a data-enhanced image sample. The data augmentation can include random flipping, random 360-degree rotation, random stretching, and scaling at any time. The key point detection model is then used to obtain a predicted key point heat map corresponding to the data-enhanced image sample. The training loss of the key point detection model is then calculated. Finally, the key point detection model is iteratively trained using the training loss of the key point detection model to obtain a trained key point detection model.

[0084] Optionally, the mean square error algorithm can be used to calculate the training loss of the key point detection model. The training loss of the key point detection model can be expressed as follows:

[0085] Loss=(H y -H gt ) 2 ;

[0086] Among them, H y H is the predicted key point heat map output by the key point detection model. gt is the true key point heat map corresponding to the image sample (i.e., the gold standard).

[0087] Step 202d: Perform posture detection on the captured image to obtain a rotation matrix corresponding to the target. The rotation matrix is ​​used to represent the posture of the target.

[0088] In the embodiment of the present application, the rotation matrix corresponding to the target is obtained by directly regressing the rotation matrix of the model. For example, the rotation matrix detection model can be used to perform posture detection on the captured image to obtain the rotation matrix corresponding to the target.

[0089] The rotation matrix detection model is a neural network for rotation matrix detection. For example, Figure 6 The rotation matrix detection model may include a backbone network 601 (i.e., backbone), a global average pooling layer 602 (i.e., Global Average Pooling, GAP layer), and a linear layer 603. The backbone network 601 may be constructed based on Resnet18, MobileNet (a lightweight convolutional neural network), DenseNet (a dense convolutional neural network), etc., and the linear layer 603 includes two linear layers.

[0090] The backbone network 601 is used to extract features from the captured image to obtain a feature map, which is then transformed through the global average pooling layer 602 to obtain an intermediate feature vector. Finally, the intermediate feature vector is linearly transformed through the linear layer 603 to obtain an output vector (such as a 6-dimensional vector), and a rotation matrix is ​​obtained based on the output vector.

[0091] In an example, the specific method for obtaining the rotation matrix can be as follows:

[0092] 1. Perform posture detection on the captured image to obtain the output vector corresponding to the target. The output vector is used to represent the posture of the target in the captured image.

[0093] For example, reference Figure 6 , detect the model through the rotation matrix and obtain the 6-dimensional vector corresponding to the target.

[0094] 2. Split the output vector to obtain a first orthogonal rotation vector and a second orthogonal rotation vector.

[0095] Optionally, the output vector may be split in half to obtain a first orthogonal rotation vector and a second orthogonal rotation vector. Figure 6 , the first 3-dimensional elements of the 6-dimensional vector can be determined as the first orthogonal rotation vector, and the last 3-dimensional elements of the 6-dimensional vector can be determined as the second orthogonal rotation vector.

[0096] 3. Transform the first orthogonal rotation vector to obtain the first submatrix.

[0097] Optionally, the process of obtaining the first submatrix can be expressed as follows:

[0098] Wherein, x1 is the first orthogonal rotation vector.

[0099] 4. Based on the first submatrix and the second orthogonal rotation vector, obtain the second submatrix.

[0100] Optionally, the process of obtaining the second sub-matrix can be expressed as follows:

[0101] v2=x2-(v1·x2)v1, where x2 is the second orthogonal rotation vector.

[0102] 5. Transform the second orthogonal rotation vector to obtain the third submatrix.

[0103] Optionally, the process of obtaining the third submatrix can be expressed as follows:

[0104]

[0105] 6. Based on the first sub-matrix and the third sub-matrix, obtain the fourth sub-matrix.

[0106] Optionally, the process of obtaining the fourth sub-matrix can be expressed as follows: v4=v1·v3.

[0107] 7. Based on the first submatrix, the third submatrix and the fourth submatrix, obtain the rotation matrix corresponding to the target.

[0108] Alternatively, the process of obtaining the rotation matrix can be expressed as follows:

[0109] M=|v1 v3 v4|;where M is the rotation matrix, M∈R 3×3 .

[0110] Optionally, the training loss of the rotation matrix detection model can be calculated based on the rotation matrix corresponding to the target, and then the rotation matrix detection model can be iteratively trained based on the training loss of the rotation matrix detection model to obtain a trained rotation matrix detection model, which can directly regress the rotation matrix. Exemplarily, the geodesic distance loss can be used to calculate the training loss of the rotation matrix detection model based on the rotation matrix corresponding to the target. The training loss can be expressed as follows:

[0111]

[0112] Among them, M p is the rotation matrix output by the rotation matrix detection model, M gt is the true rotation matrix (i.e., the gold standard), and tr() is the trace of the matrix.

[0113] According to the angle to represent the posture of the target, it can be divided into Euler angles, quaternions and rotation matrices. The relevant technology mainly uses convolutional neural networks to regress Euler angles and uses the center of the target's bounding box to calculate the 2D translation vector (tdx, tdy). When the target is facing the camera from the side, this method will not cause a large error. However, when the target is facing the camera from the side, the target is facing the camera from the back, the target is facing the camera from the top, etc., the error will become very large. At the same time, there is a gimbal block problem in Euler angles, and there are multiple solutions for the same posture. The regression network based on Euler angles in the relevant technology cannot solve the problem of Large Pose (excessive rotation relative to the positive side of the target) at any viewing angle. However, the present application can reduce the error in posture acquisition and avoid the gimbal lock and LargePose problems that exist when regressing Euler angles by directly regressing the rotation matrix, thereby improving the accuracy of posture acquisition and further improving the accuracy of target registration.

[0114] Optionally, in order to enable the rotation matrix detection model to achieve a higher posture prediction effect, when training the rotation matrix detection model, the embodiment of the present application uses a posture enhancement training scheme. This scheme improves the prediction ability of the rotation matrix detection model by simulating the 2D translation and rotation of the target relative to the vertical photographic plane of the image acquisition device. Exemplarily, the image acquisition device can be simulated to be close to or away from the target by Crop (random cropping), Resize (random scaling), etc. The posture change of the target can be simulated by random flipping, random 360-degree rotation, etc.

[0115] Among them, data augmentation methods such as Crop and Resize can translate the target without changing the relative rotation angle of the target, which can greatly simulate the movement of the image acquisition device in real scenes.

[0116] In one example, the posture enhancement algorithm (i.e., rotation, flipping, etc.) can be as follows: the 6DoF of the target's posture can be converted into a matrix expression, and the corresponding transformation is performed to obtain the pose matrix. The 6DoF and pose matrix can be converted to each other, so that the real rotation matrix corresponding to the image sample can be updated.

[0117] The process can be expressed by the following formula:

[0118] 6DoF→Matrix M0;

[0119] M t =TM0;

[0120] Matrix M t →6DoF;

[0121] Among them, M0 is the rotation matrix before updating, M t is the updated rotation matrix.

[0122] For example, taking the image center as the rotation center, the update process of the true rotation matrix (i.e., the gold standard) can be as follows: Rotation involves the conversion between Euler angles and rotation matrices. Assuming the three rotation angles are α, β, and γ, the process of converting Euler angles to rotation matrices can be expressed as follows:

[0123]

[0124]

[0125]

[0126] M=R y (β)*R x (α)*R z (γ).

[0127] The conversion process from rotation matrix to Euler angle can be expressed as follows:

[0128] α=αtan2(M 21 , M 22 );

[0129] β=αrcsin(-M 23 );

[0130] γ=αtan2(M 21 , M 22 ).

[0131] The 2D rotation enhancement formula of the posture is as follows:

[0132] M t =TM0;

[0133] Wherein, T=to_rotation_matrix(0,0,γ1), γ1=γ0+θ, 0 is the rotation angle of the image acquisition device, (0,0,γ0) is the initial rotation angle corresponding to the target, and (0,0,γ1) is the updated rotation angle corresponding to the target.

[0134] Step 202e: Based on the target key points and the rotation matrix, obtain the pose matrix of the target in the image to be registered.

[0135] Optionally, the translation vector corresponding to the target can be obtained based on the target key points, and then the target pose matrix can be constructed based on the translation vector and the rotation matrix.

[0136] Exemplarily, based on the depth image corresponding to the image to be registered, the depth information of the target key point is obtained; based on the plane coordinates of the target key point in the image to be registered and the depth information of the target key point, the translation vector of the target key point in the world coordinate system is obtained; wherein the translation vector is used to characterize the position of the target key point in the image to be registered; based on the translation vector and the rotation matrix, the pose matrix of the target in the image to be registered is constructed.

[0137] In an embodiment of the present application, the depth image corresponding to the image to be registered is aligned with the image to be registered, that is, the pixels in the depth image corresponding to the image to be registered are aligned one by one with the pixels in the image to be registered. It is only necessary to obtain the pixel points corresponding to the target key points in the depth image to obtain the depth information of the target key points. The plane coordinates of the target in the image to be registered are used as the x-axis coordinates and y-axis coordinates of the target, and the depth information of the target key points is used as the z-axis coordinates. The three-dimensional coordinates of the target key points in the world coordinate system can be obtained, and then based on the three-dimensional coordinates and the origin of the world coordinate system, the translation vector of the target key points in the world coordinate system can be obtained. The data corresponding to the translation vector and the data corresponding to the rotation matrix are used as elements to construct the pose matrix of the target.

[0138] Step 203: Based on the pose matrix of the target in the image to be registered and the pose matrix of the target in the reference image, an initial registration relationship between the image to be registered and the reference image is obtained; wherein the initial registration relationship is used to preliminarily characterize the transformation relationship between the image to be registered and the reference image.

[0139] In an embodiment of the present application, the pose of the target in the reference image and the pose of the target in the image to be registered may be the same or different.

[0140] In one example, an initial registration relationship may be acquired based on a reference model of the target. The process may be as follows:

[0141] 1. Based on the reference image, build a reference model corresponding to the target.

[0142] Optionally, based on the surface data of the target in the reference image, a binary segmentation image corresponding to the target can be obtained; the binary segmentation image is transformed to obtain a reference model corresponding to the target; wherein the reference model is composed of multiple triangular facets.

[0143] Among them, the surface data is used to characterize the appearance of the target. For example, in a surgical navigation scenario, the surface data may refer to data corresponding to the skin. In a target authentication scenario, the surface data may refer to data on the surface of an object. The binary segmented image corresponding to the target can be used to reflect the characteristics of the target and the area in the reference image. Optionally, the Marching Cube algorithm can be used to convert the binary segmented image into a reference model composed of multiple triangular facets, and the reference model can be a three-dimensional model.

[0144] 2. Based on the standard data corresponding to the target, perform rigid body transformation on the reference model to obtain the transformed reference model.

[0145] Since the target's pose may vary at different times, and the reference model constructed based on the reference image may contain errors, a rigid body transformation can be performed on the reference model's pose using standard data to reduce the impact of errors. Standard data can refer to the average data of the target's pose, making the target's reference model more standard and accurate. Rigid body transformation only performs rigid transformations on the reference model (such as fine-tuning translation and rotation) and does not deform the reference model (such as stretching or scaling), thus not affecting the authenticity of the reference model.

[0146] Optionally, an ICP (Iterative Closet Point) algorithm can be used to register the standard model corresponding to the standard data with the reference model to obtain a transformed reference model. Alternatively, another PBR (Point Based Registration) algorithm can be used to register the standard model corresponding to the standard data with the reference model to obtain a transformed reference model, although this embodiment of the present application is not limited thereto.

[0147] 3. Based on the transformed reference model, obtain the pose matrix of the target in the reference image.

[0148] Alternatively, the same key point detection and rotation matrix detection as above can be used to obtain the pose matrix of the target in the reference image. The pose matrix of the target in the reference image is also constructed based on the world coordinate system and can be denoted as P img ∈R 3×4 .

[0149] 4. Divide the pose matrix of the target in the image to be registered by the pose matrix of the target in the reference image to obtain the initial registration relationship between the image to be registered and the reference image.

[0150] Optionally, the initial registration relationship can be expressed as follows:

[0151]

[0152] Among them, P rest is the pose matrix of the target in the image to be registered. This initial registration relationship can be used to represent the transformation relationship between the target in the image to be registered and the target in the reference image.

[0153] The embodiments of the present application perform target registration based on a reference model corresponding to the target, making the registration more realistic and improving the accuracy of the registration. Furthermore, compared to performing image-level registration directly based on a reference image and the image to be registered, registering the target based on a reference model can effectively reduce the computational complexity of the registration process, thereby improving registration efficiency.

[0154] In another example, image-level registration can also be performed based on a direct reference image and the image to be registered, which is not limited in the present embodiment. For example, matching pixel points between an object in the reference image and an object in the image to be registered are first obtained, and then registration is performed between the object in the reference image and the object in the image to be registered based on the matching pixel points.

[0155] Step 204 : obtaining a preliminary registration result corresponding to the target in the image to be registered based on the initial registration relationship and the reference image.

[0156] The preliminary registration result refers to the result of preliminary registration of the object in the image to be registered. The pose of the object in the preliminary registration result is nearly identical to the pose of the object in the reference image. Optionally, the object in the image to be registered can be transformed based on the initial registration relationship to obtain the preliminary registration result.

[0157] When object registration is performed based on a reference model, the object in the image to be registered may be transformed according to the initial registration relationship to obtain a preliminary registration model corresponding to the object in the image to be registered.

[0158] When performing image-level target registration, the image to be registered can be transformed according to the initial registration relationship to obtain a preliminary registration image, and then a model is constructed on the preliminary registration image to obtain a preliminary registration model corresponding to the target in the image to be registered.

[0159] Step 205 : Based on the effective area of ​​the depth image corresponding to the image to be registered and the preliminary registration result, an optimized registration relationship between the image to be registered and the reference image is obtained; wherein the effective area of ​​the depth image includes the target.

[0160] Optionally, the valid area of ​​the depth image may refer to the area of ​​the target in the depth image. The optimized registration relationship may be used to characterize the conversion relationship between the target in the preliminary registration result and the target in the depth image.

[0161] In one example, the process of obtaining the optimized registration relationship may be as follows:

[0162] 1. Obtain the point cloud corresponding to the preliminary registration result.

[0163] Optionally, a point cloud corresponding to the preliminary registration result can be obtained by sampling from the preliminary registration result (such as the preliminary registration model) through methods such as uniform sampling, geometric sampling, random sampling, and grid sampling.

[0164] 2. Remove outliers in the point cloud corresponding to the preliminary registration result to obtain the optimized point cloud.

[0165] In the present embodiment, outliers refer to point clouds that are too far or too close to the target. For example, in a surgical navigation scenario, a point cloud extraction range can be set based on the point cloud depth and the normal surgical distance. Point clouds that are smaller than the lower limit of the point cloud extraction range and those that are larger than the upper limit of the point cloud extraction range are removed to obtain an optimized point cloud. This removes noise from the point cloud and improves target registration accuracy.

[0166] 3. According to the shooting direction corresponding to the image to be registered, the target point cloud is extracted from the optimized point cloud. The target point cloud refers to the point cloud observed in the shooting direction.

[0167] For example, the point cloud facing the image acquisition device can be directly intercepted from the optimized point cloud to obtain the target point cloud, thereby further eliminating noise and further improving the registration accuracy of the target.

[0168] 4. Based on the target point cloud and the point cloud corresponding to the valid area, iterative optimization is performed to obtain the optimized registration relationship between the image to be registered and the reference image.

[0169] Optionally, the point cloud corresponding to the valid area can be obtained by sampling from the valid area through methods such as uniform sampling, geometric sampling, random sampling, and grid sampling.

[0170] In one example, the effective region can be obtained using an effective region segmentation model. The effective region segmentation model can be constructed based on a segmentation network such as 2D Unet, DeepLab, or FCN (Full Convolutional Networks). The training loss of the effective region segmentation model can be calculated using a Cross Entroy loss function, a Dice Loss function, or the like.

[0171] In one example, an ICP algorithm can be used to iteratively optimize the target point cloud and the point cloud corresponding to the valid area to obtain an optimized registration relationship. Other PBR algorithms can also be used to obtain an optimized registration relationship based on the target point cloud and the point cloud corresponding to the valid area, and this embodiment of the application is not limited to this.

[0172] Step 206 , adjusting the preliminary registration result according to the optimized registration relationship, and obtaining an optimized registration result corresponding to the target in the registered image; wherein the optimized registration result is used to guide the subject to process the target.

[0173] Optionally, the preliminary registration result may be converted according to the optimized registration relationship to obtain an optimized registration result.

[0174] When the target registration is performed based on the reference model, the preliminary registration model can be adjusted according to the optimized registration relationship to obtain the optimized registration model corresponding to the target in the registered image.

[0175] When performing image-level target registration, the preliminary registered image can be transformed according to the initial registration relationship to obtain an optimized registered image, and then a model is constructed on the optimized registered image to obtain an optimized registration model corresponding to the target in the image to be registered.

[0176] The embodiment of the present application performs targeted optimization on the preliminary registration result of the target through the effective area of ​​the depth image, effectively reducing the registration error existing in the preliminary registration result and further improving the registration accuracy of the target.

[0177] In one example, after obtaining the optimized registration result (such as the optimized registration model), the optimized registration result can also be converted from the world coordinate system to the optimized registration result in the image acquisition device coordinate system, and VR (Virtual Reality), AR (Augmented Reality) and other view renderings can be performed from the perspective of the image acquisition device, thereby more intuitively guiding the object to process the target.

[0178] In summary, the technical solution provided by the embodiment of the present application performs preliminary registration of the target based on the pose matrix of the target in the image to be registered and the pose matrix of the target in the reference image, and then optimizes the preliminary registration result based on the effective area of ​​the depth image corresponding to the image to be registered and the preliminary registration result, thereby obtaining the optimized registration result corresponding to the target, realizing fully automated registration of the target, avoiding the introduction of human errors, and thus improving the registration accuracy of the target. At the same time, the preliminary registration result of the target is optimized in a targeted manner through the effective area of ​​the depth image, effectively reducing the registration error existing in the preliminary registration result, and further improving the registration accuracy of the target.

[0179] In addition, since automatic registration of the target can be achieved based on the image to be registered and the reference image, and then navigation of the target can be achieved, there is no need to use expensive large-scale equipment to navigate the target (such as surgical navigation of the target), which greatly improves the ease and convenience of target navigation and reduces the cost of target navigation.

[0180] In addition, the embodiment of the present application can reduce the error in posture acquisition by directly regressing the rotation matrix, and avoid the universal lock and LargePose problems that exist when regressing Euler angles, thereby improving the accuracy of posture acquisition and further improving the accuracy of target alignment.

[0181] Please refer to Figure 7 , which shows a flow chart of a target registration method provided by another embodiment of the present application. The execution subject of each step of the method can be Figure 1 In the terminal 10 or server 20 in the implementation environment of the solution shown, the method may include the following steps (701-705).

[0182] Step 701: Acquire an image to be registered and a reference image containing the same object.

[0183] Step 701 is the same as that described in the above embodiment. For the contents not described in the embodiment of this application, please refer to the above embodiment and will not be repeated here.

[0184] Step 702: Obtain a preliminary registration result after manual registration.

[0185] The preliminary registration result refers to the result after the object in the image to be registered is preliminarily registered. The pose of the object in the preliminary registration result is almost the same as the pose of the object in the reference image.

[0186] In situations where the target faces the image acquisition device from behind or above, resulting in no obvious key points, occlusion, or incompleteness, the rotation matrix detection model struggles to predict the target's pose. In these cases, a semi-automatic registration method is required. For example, a subject (e.g., a surgeon in a surgical navigation scenario) manually performs a preliminary registration of the target in the image to be registered with the target in the reference image, obtaining a preliminary registration result that is then made available to the client.

[0187] Exemplarily, the object marks the key points of the target in the image to be registered and the target in the reference image respectively, and then automatically obtains the initial registration relationship between the target in the image to be registered and the target in the reference image through the client, and then transforms the target in the image to be registered through the initial registration relationship to obtain a preliminary registration result.

[0188] Step 703 : Divide the effective area of ​​the depth image to obtain a rigid area and a deformable area corresponding to the effective area.

[0189] Optionally, the effective area of ​​the depth image can be obtained by using the effective area segmentation model, where the effective area includes the target.

[0190] Rigid regions are areas that are less prone to deformation. For example, on the head, the forehead and nose are areas that are less prone to deformation. Deformable regions are areas that are more prone to deformation. For example, the mouth and face are areas that are more prone to deformation.

[0191] For example, refer to Figure 8 Taking the head as an example, the effective area corresponding to the head can be automatically segmented to obtain areas such as skin 801, ears (not shown), face 802, eyes 803, eyebrows 804, mouth 805, and nose 806. Then, the rigid area and deformable area are divided based on the eyes 803. For example, the forehead (not shown) above the eyes 803 is determined as a rigid area (which may also include the nose 806), and the mouth 805 and face 802 below the eyes 803 are determined as deformable areas.

[0192] In step 704 , based on the point cloud corresponding to the preliminary registration result, the point cloud corresponding to the effective area, the point cloud weight parameters corresponding to the rigid area, and the point cloud weight parameters corresponding to the deformed area, an optimized registration relationship between the image to be registered and the reference image is obtained by iterative optimization.

[0193] Among them, the point cloud weight parameter corresponding to the rigid area is greater than the point cloud weight parameter corresponding to the deformed area. In this way, the interference of the deformed area can be reduced while utilizing the deformed area, which is conducive to improving the registration accuracy of the target.

[0194] For example, refer to Figure 9 Taking the head as an example, the semi-automatic registration process can be as follows: the client constructs a preoperative head model 901 based on the corresponding reference image of the head. The client then obtains a preliminary registration result 904, obtained by manually registering the preoperative head model 901 with the head in the current image to be registered 902. The client then segments the effective region of the depth image 903 at the current moment, obtaining the rigid region and deformable region corresponding to the effective region. The client then obtains the point cloud of the preliminary registration model 904 and the point cloud corresponding to the effective region, respectively. Using the ICP algorithm, the point cloud of the preliminary registration model 904 and the point cloud corresponding to the effective region are iteratively optimized to obtain an optimized registration relationship. This process primarily optimizes the registration of the rigid region, thereby assigning a greater weight to the point cloud of the rigid region. Specifically, the weight parameter of the point cloud corresponding to the rigid region is set to be greater than the weight parameter of the point cloud corresponding to the deformable region. The client then optimizes the preliminary registration model 904 based on the optimized registration relationship, obtaining an optimized registration model 905. Optionally, the client may render the optimized registration model 905 into a VR or AR view to show the optimized registration model 905 to the surgical performer.

[0195] Step 705: Adjust the preliminary registration result according to the optimized registration relationship to obtain the optimized registration result corresponding to the target in the registered image. The optimized registration result is used to guide the object to process the target.

[0196] Optionally, the preliminary registration result can be transformed according to the optimized registration relationship to obtain an optimized registration result. For example, when performing target registration based on a reference model of the target, the preliminary registration model corresponding to the target can be adjusted according to the optimized registration relationship to obtain an optimized registration model corresponding to the target in the registered image.

[0197] To sum up, the technical solution provided in the embodiments of the present application realizes semi-automatic registration of the target by performing targeted optimization on the preliminary registration results obtained by manual registration based on the effective area of ​​the automatically acquired depth image, avoids the object from manually outlining the effective area of ​​the depth image, and thus improves the registration efficiency and accuracy of the target.

[0198] In an exemplary embodiment, in a surgical navigation scenario, the target registration method provided by the embodiment of the present application can be used to navigate the head, brain, thigh, femur, abdomen, lungs, and other parts during surgery. The following will use the head as an example to illustrate the target registration method provided by the embodiment of the present application. The specific content can be as follows:

[0199] In order to improve the usability of target registration in surgical navigation, the embodiment of the present application transplants the target registration method to a mobile terminal, such as the client corresponding to the target application (such as a surgical navigation application) installed and running in the above-mentioned terminal 10.

[0200] For example, refer to Figure 10 , the surgical navigation system may include a tablet computer 1001, a depth camera 1002 and a navigation stick 1003. Among them, the tablet computer 1001 serves as a computing platform, on which a surgical navigation application is installed, which can be used to implement the target alignment method provided in the embodiment of the present application. The depth camera 1002 can capture images of the head during surgery (hereinafter referred to as intraoperative images) in real time, which include RGB images (such as color images) and depth images, i.e., RGB-D images. The navigation stick 1003 can be a visual navigation stick, which can be used to obtain the position of the needle tip in real time.

[0201] The tablet computer 1001 and the depth camera 1002 can communicate with each other via a network. The tablet computer 1001 can obtain intraoperative images from the depth camera 1002.

[0202] In one example, reference Figure 11 The usage process of the client corresponding to the above target application (such as a surgical navigation application) can be as follows:

[0203] 1. Acquire preoperative images of the patient's head. For example, in an emergency surgery for a hematoma, a rapid CT scan can be acquired. In a tumor surgery scenario, an MRI can be acquired.

[0204] 2. Specify the preoperative plan. This process mainly includes segmenting organs and specifying surgical approaches based on preoperative images. The segmented organs may include registration organs and navigation organs. The registration organs are used to align with intraoperative images (such as the RGB images and depth images mentioned above). For example, the registration organs may refer to rigid organs or structures, such as bony parts (skull, femur, spine, etc.). Navigation organs may include lesions, ventricles, etc., to assist the surgeon in positioning.

[0205] Optionally, the segmented organs can also include non-rigid organs and structures, such as the abdomen and lungs. Due to heartbeat and breathing, these organs may drift during surgery, greatly increasing the difficulty of surgical navigation. Therefore, rigid organs or structures can be selected and targeted for registration.

[0206] Since the present embodiment is a surgical navigation scenario, organ segmentation can be performed manually or automatically. The automatic segmentation algorithm is not limited to common 3D segmentation networks such as 3DUnet.

[0207] 3. Intraoperative registration: The client in the tablet computer 1001 obtains the intraoperative image in real time through the depth camera 1002, and adopts the target registration method provided in the embodiment of the present application to obtain the transformation relationship between the head corresponding to the intraoperative image and the head corresponding to the preoperative image, thereby obtaining the final registration result.

[0208] 4. Intraoperative navigation: The client in tablet computer 1001 spatially transforms the final registration result, renders it into a VR or AR view, and displays the VR or AR view on the screen. Optionally, the client in tablet computer 1001 can automatically detect the position of navigation stick 1003 and display the needle tip, as well as key structures such as registration organs and navigation organs, in the VR or AR view to assist the surgeon in head manipulation.

[0209] In the embodiment of the present application, intraoperative registration is divided into automatic registration and semi-automatic registration. When automatic registration is unavailable (e.g., key points cannot be detected) or there is a small probability of error (e.g., incomplete head), semi-automatic registration can be used.

[0210] In one example, reference Figure 12 ,The specific process of intraoperative registration and intraoperative navigation can be as follows:

[0211] Based on the preoperative image, a preoperative head model 1203 corresponding to the head is constructed. For example, a binary segmented image corresponding to the head can be obtained based on the skin corresponding to the head in the preoperative image. The binary segmented image is then transformed using the Marching Cube algorithm to obtain the preoperative head model 1203 corresponding to the head. The preoperative head model 1203 is composed of multiple triangular facets.

[0212] The preoperative head model 1203 is registered to the standard head model corresponding to the patient undergoing the surgery using the ICP algorithm to achieve standard posture alignment, thereby obtaining the aligned preoperative head model 1203 .

[0213] The target detection model is used to perform head detection on the RGB image 1201 corresponding to the intraoperative image to obtain a bounding box corresponding to the head. Then, based on the enlarged bounding box, the head image is cut out from the RGB image 1201.

[0214] The head image is subjected to posture detection through the rotation matrix detection model to obtain the corresponding rotation detection of the head. The head image is subjected to key point detection through the key point detection model to obtain multiple key points corresponding to the head (such as the left and right corners of the eyes, the tip of the nose, and the left and right corners of the mouth, etc.), and the target key point is determined from the multiple key points, and the 2D coordinates of the target key point are obtained. For example, when the head is facing the depth camera from the front, the corner of the eye can be determined as the target key point. When the head is facing the depth camera from the side, the tip of the nose can be determined as the target key point.

[0215] The depth image 1202 corresponding to the intraoperative image is aligned with the RGB image 1201 to obtain an aligned depth image 1202 , and depth information corresponding to the target key point is obtained from the aligned depth image 1202 .

[0216] Based on the depth information corresponding to the target key point and the 2D coordinates of the target key point, the 3D coordinates of the target key point are calculated, and then based on the 3D coordinates, the translation vector of the target key point in the world coordinate system is calculated.

[0217] Based on the above translation vector and rotation matrix, the pose matrix of the head in the RGB image is calculated, and the pose matrix corresponding to the preoperative head model 1203 after calibration is obtained. The pose matrix corresponding to the preoperative head model 1203 after calibration is divided by the pose matrix of the head in the RGB image to obtain the initial registration relationship corresponding to the head.

[0218] According to the initial registration relationship, the head in the RGB image is replaced to obtain a preliminary registration head model. Figure 13 , the preliminary registration head model 1301 and the preoperative head model 1302 are almost consistent.

[0219] Both key point detection and rotation matrix detection are based on RGB images. There is a certain error between the posture matrix obtained based on the RGB image and the posture matrix of the head during surgery. Therefore, it is necessary to further optimize the preliminary registration head model based on the depth image 1202 to achieve the registration accuracy required for surgical navigation.

[0220] Obtain the point cloud of the preliminary registered head model, remove outliers in the point cloud corresponding to the preliminary registered head model, and obtain an optimized point cloud. According to the shooting direction corresponding to the depth camera, extract the target point cloud from the optimized point cloud. The target point cloud refers to the point cloud observed in the shooting direction.

[0221] The effective area corresponding to the depth image 1202 and the three-dimensional head point cloud corresponding to the effective area are obtained, and outliers are removed from the three-dimensional head point cloud corresponding to the effective area to obtain an optimized three-dimensional head point cloud corresponding to the effective area.

[0222] The effective area is segmented to obtain a rigid area and a deformable area corresponding to the effective area, and a point cloud weight parameter corresponding to the rigid area is set to be greater than a point cloud weight parameter corresponding to the deformable area.

[0223] The ICP algorithm is used to iteratively obtain the transformation relationship between the preliminary registered head model and the head in the depth image, that is, the optimized registration relationship, based on the target point cloud corresponding to the preliminary registered head model, the optimized head three-dimensional point cloud corresponding to the effective area, the point cloud weight parameters corresponding to the rigid area, and the point cloud weight parameters corresponding to the deformable area.

[0224] According to the optimized registration relationship, the preliminary registered head model is optimized to obtain the optimized registered head model. Figure 14 , the optimized registration head model 1401 is consistent with the preoperative head model 1402 .

[0225] Optionally, based on the above embodiment, after obtaining the optimized registration result, the optimized registration result can be converted from the world coordinate system to the optimized registration result in the image acquisition device coordinate system; and the navigation stick and the needle tip can be synchronously displayed in the optimized registration result in the image acquisition device coordinate system according to the positions of the navigation stick and the needle tip in the registered image; wherein the optimized registration result includes a navigation organ, which is used to assist the object in processing the target.

[0226] For example, the optimized registered head model can be converted from the world coordinate system to the optimized registered head model in the depth camera coordinate system, and the optimized registered head model in the depth camera coordinate system can be rendered into a VR or AR view, and the navigation stick and the needle tip can be synchronously displayed in the optimized registered head model in the depth camera coordinate system according to the positions of the navigation stick and the needle tip in the intraoperative image. Figure 15 , which shows the optimized registration head model 1502 under different shooting angles and different display options, as well as the positions of the navigation stick 1501 and the tip of the navigation stick 1501 relative to the optimized registration head model 1502.

[0227] In cases where there are no obvious key points detected on the head of the person being operated on (such as the back of the head facing the depth camera) or the head is obscured, the surgeon can use a semi-automatic alignment method to manually align the preoperative head model and the real head during the operation. In order to ensure the accuracy and efficiency of surgical navigation, the embodiment of the present application automatically obtains the effective area corresponding to the depth image, and automatically optimizes the preliminary aligned head model through the effective area to obtain an optimized aligned head model, thereby improving the accuracy of surgical navigation.

[0228] For example, a preliminarily registered head model is obtained after manual registration. The valid area of ​​the depth image is divided to obtain a rigid area and a deformable area corresponding to the valid area. Based on the point cloud corresponding to the preliminarily registered head model, the point cloud corresponding to the valid area, the point cloud weight parameters corresponding to the rigid area, and the point cloud weight parameters corresponding to the deformable area, an optimized registration relationship is obtained through iterative optimization. The point cloud weight parameters corresponding to the rigid area are greater than the point cloud weight parameters corresponding to the deformable area. Finally, based on the optimized registration relationship, the preliminarily registered head model is optimized to obtain an optimized registered head model.

[0229] In summary, the technical solution provided by the embodiment of the present application performs preliminary registration of the target (such as the head) based on the pose matrix of the target in the intraoperative image and the pose matrix of the preoperative model of the target, and then optimizes the registration of the preliminary registration model based on the effective area of ​​the depth image corresponding to the preoperative image and the preliminary registration model, thereby obtaining an optimized registration model corresponding to the target, thereby achieving fully automated registration of the target during surgery, avoiding the introduction of human errors, and thus improving the registration accuracy of the target during surgery. At the same time, through the effective area of ​​the depth image, the preliminary registration model of the target is targetedly optimized, effectively reducing the registration error existing in the preliminary registration model, and further improving the registration accuracy of the target during surgery.

[0230] In addition, since automatic alignment of intraoperative targets can be achieved based on intraoperative images (RGB images and depth images) and preoperative images, navigation of intraoperative targets can be achieved without the need for expensive large-scale equipment to perform surgical navigation on the targets. This greatly improves the ease and convenience of intraoperative target navigation and reduces the cost of intraoperative target navigation.

[0231] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0232] Please refer to Figure 16 , which shows a block diagram of an object registration device provided by one embodiment of the present application. This device can be used to implement the above-mentioned object registration method. The device 1600 may include: a registration image acquisition module 1601, a pose matrix acquisition module 1602, an initial relationship acquisition module 1603, a preliminary result acquisition module 1604, an optimized relationship acquisition module 1605, and an optimized result acquisition module 1606.

[0233] The registration image acquisition module 1601 is used to acquire an image to be registered and a reference image containing the same object.

[0234] The pose matrix acquisition module 1602 is used to detect the image to be registered and obtain the pose matrix of the target in the image to be registered. The pose matrix is ​​used to represent the position and posture of the target in the image.

[0235] The initial relationship acquisition module 1603 is used to obtain the initial registration relationship between the image to be registered and the reference image based on the pose matrix of the target in the image to be registered and the pose matrix of the target in the reference image; wherein the initial registration relationship is used to preliminarily characterize the transformation relationship between the image to be registered and the reference image.

[0236] The preliminary result acquisition module 1604 is configured to acquire a preliminary registration result corresponding to the target in the image to be registered according to the initial registration relationship and the reference image.

[0237] The optimized relationship acquisition module 1605 is used to obtain the optimized registration relationship between the image to be registered and the reference image based on the effective area of ​​the depth image corresponding to the image to be registered and the preliminary registration result; wherein the effective area of ​​the depth image includes the target.

[0238] The optimization result acquisition module 1606 is used to adjust the preliminary registration result according to the optimized registration relationship, and obtain the optimized registration result corresponding to the target in the registered image; wherein the optimized registration result is used to guide the object to process the target.

[0239] In an exemplary embodiment, the optimization relationship acquisition module 1605 is configured to:

[0240] Obtaining a point cloud corresponding to the preliminary registration result;

[0241] Removing outliers in the point cloud corresponding to the preliminary registration result to obtain an optimized point cloud;

[0242] Extracting a target point cloud from the optimized point cloud according to the shooting direction corresponding to the image to be registered, wherein the target point cloud refers to a point cloud observed in the shooting direction;

[0243] Based on the target point cloud and the point cloud corresponding to the valid area, an optimized registration relationship between the image to be registered and the reference image is obtained through iterative optimization.

[0244] In an exemplary embodiment, Figure 17 As shown, the pose matrix acquisition module 1602 includes: a bounding box acquisition submodule 1602a, a screenshot image acquisition submodule 1602b, a target point acquisition submodule 1602c, a rotation matrix acquisition submodule 1602d and a pose matrix acquisition submodule 1602e.

[0245] The bounding box acquisition submodule 1602a is used to perform target detection on the image to be registered to obtain a bounding box corresponding to the target.

[0246] The screenshot image acquisition submodule 1602b is configured to capture a screenshot image including the target from the image to be registered based on the bounding box corresponding to the target.

[0247] The target point acquisition submodule 1602c is used to perform key point detection on the captured image to obtain target key points corresponding to the target.

[0248] The rotation matrix acquisition submodule 1602d is used to perform posture detection on the captured image to obtain a rotation matrix corresponding to the target, and the rotation matrix is ​​used to represent the posture of the target.

[0249] The pose matrix acquisition submodule 1602e is used to obtain the pose matrix of the target in the image to be registered based on the target key points and the rotation matrix.

[0250] In an exemplary embodiment, the screenshot image acquisition submodule 1602b is configured to:

[0251] Enlarging the bounding box to obtain an enlarged bounding box;

[0252] A captured image including the target is captured from the image to be registered according to the enlarged bounding box.

[0253] In an exemplary embodiment, the pose matrix acquisition submodule 1602e is configured to:

[0254] Acquiring depth information of the target key points based on the depth image corresponding to the image to be registered;

[0255] Based on the plane coordinates of the target key point in the image to be registered and the depth information of the target key point, obtaining a translation vector of the target key point in the world coordinate system; wherein the translation vector is used to represent the position of the target key point in the image to be registered;

[0256] Based on the translation vector and the rotation matrix, a pose matrix of the target in the image to be registered is constructed.

[0257] In an exemplary embodiment, the rotation matrix acquisition submodule 1602d is configured to:

[0258] Performing posture detection on the captured image to obtain an output vector corresponding to the target, wherein the output vector is used to represent the posture of the target in the captured image;

[0259] Splitting the output vector to obtain a first orthogonal rotation vector and a second orthogonal rotation vector;

[0260] Transforming the first orthogonal rotation vector to obtain a first sub-matrix;

[0261] Obtaining a second submatrix based on the first submatrix and the second orthogonal rotation vector;

[0262] transforming the second orthogonal rotation vector to obtain a third submatrix;

[0263] Obtaining a fourth submatrix based on the first submatrix and the third submatrix;

[0264] A rotation matrix corresponding to the target is obtained based on the first submatrix, the third submatrix, and the fourth submatrix.

[0265] In an exemplary embodiment, Figure 17 As shown, the initial relationship acquisition module 1603 includes: a reference model construction submodule 1603a, a reference model transformation submodule 1603b and an initial relationship acquisition submodule 1603c.

[0266] The reference model construction submodule 1603a is configured to construct a reference model corresponding to the target based on the reference image.

[0267] The reference model transformation submodule 1603b is configured to perform a rigid body transformation on the reference model based on the standard data corresponding to the target to obtain a transformed reference model.

[0268] The pose matrix acquisition module 1602 is further configured to acquire the pose matrix of the target in the reference image based on the transformed reference model.

[0269] The initial relationship acquisition submodule 1603c is used to divide the pose matrix of the target in the image to be registered by the pose matrix of the target in the reference image to obtain an initial registration relationship between the image to be registered and the reference image.

[0270] In an exemplary embodiment, the reference model construction submodule 1603a is used to:

[0271] Based on the surface data of the target in the reference image, obtaining a binary segmentation image corresponding to the target;

[0272] The binary segmented image is converted to obtain a reference model corresponding to the target; wherein the reference model is composed of a plurality of triangular facets.

[0273] In an exemplary embodiment, the preliminary result acquisition module 1604 is configured to adjust the target in the image to be registered according to the initial registration relationship, and obtain a preliminary registration model corresponding to the target in the registered image.

[0274] The optimization result acquisition module 1606 is used to adjust the preliminary registration model according to the optimized registration relationship to obtain the optimized registration model corresponding to the target in the registered image.

[0275] In an exemplary embodiment, Figure 17 As shown, the device 1600 further includes: a coordinate system conversion module 1607 and an optimization result display module 1608.

[0276] The coordinate system conversion module 1607 is used to convert the optimized registration result from the world coordinate system to the optimized registration result in the image acquisition device coordinate system.

[0277] The optimization result display module 1608 is used to synchronously display the navigation stick and the needle tip in the optimized registration result in the image acquisition device coordinate system according to the positions of the navigation stick and the needle tip in the registration image; wherein, the optimized registration result includes a navigation organ, and the navigation organ is used to assist the object in processing the target.

[0278] In an exemplary embodiment, Figure 17 As shown, the device 1600 further includes: an effective area division module 1609.

[0279] The preliminary result acquisition module 1604 is also used to obtain preliminary registration results after manual registration.

[0280] The effective area division module 1609 is configured to divide the effective area of ​​the depth image to obtain a rigid area and a deformable area corresponding to the effective area.

[0281] The optimized relationship acquisition module 1605 is used to iteratively optimize the optimized registration relationship between the image to be registered and the reference image based on the point cloud corresponding to the preliminary registration result, the point cloud corresponding to the valid area, the point cloud weight parameter corresponding to the rigid area, and the point cloud weight parameter corresponding to the deformed area; wherein the point cloud weight parameter corresponding to the rigid area is greater than the point cloud weight parameter corresponding to the deformed area.

[0282] In summary, the technical solution provided by the embodiment of the present application performs preliminary registration of the target based on the pose matrix of the target in the image to be registered and the pose matrix of the target in the reference image, and then optimizes the preliminary registration result based on the effective area of ​​the depth image corresponding to the image to be registered and the preliminary registration result, thereby obtaining the optimized registration result corresponding to the target, realizing fully automated registration of the target, avoiding the introduction of human errors, and thus improving the registration accuracy of the target. At the same time, the preliminary registration result of the target is optimized in a targeted manner through the effective area of ​​the depth image, effectively reducing the registration error existing in the preliminary registration result, and further improving the registration accuracy of the target.

[0283] In addition, since automatic registration of the target can be achieved based on the image to be registered and the reference image, and then navigation of the target can be achieved, there is no need to use expensive large-scale equipment to navigate the target (such as surgical navigation of the target), which greatly improves the ease and convenience of target navigation and reduces the cost of target navigation.

[0284] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0285] Please refer to Figure 18 , which shows a schematic diagram of the structure of a computer device provided in one embodiment of the present application. The computer device can be any electronic device with data calculation, processing, and storage functions, and can be used to implement the target registration method provided in the above embodiment. Specifically, it may include the following content.

[0286] The computer device 1800 includes a central processing unit (CPU, central processing unit), GPU (graphics processing unit), and FPGA (field programmable gate array) 1801, a system memory 1804 including RAM (random-access memory) 1802 and ROM (read-only memory) 1803, and a system bus 1805 connecting the system memory 1804 and the central processing unit 1801. The computer device 1800 also includes a basic input / output system (I / O system) 1806 for facilitating information transmission between various components within the server, and a mass storage device 1807 for storing an operating system 1813, application programs 1814, and other program modules 1815.

[0287] In some embodiments, the basic input / output system 1806 includes a display 1808 for displaying information and an input device 1809, such as a mouse or keyboard, for user input. Both the display 1808 and the input device 1809 are connected to the central processing unit 1801 via an input / output controller 1810 connected to the system bus 1805. The basic input / output system 1806 may also include an input / output controller 1810 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1810 also provides output to a display screen, printer, or other types of output devices.

[0288] The mass storage device 1807 is connected to the central processing unit 1801 via a mass storage controller (not shown) connected to the system bus 1805. The mass storage device 1807 and its associated computer-readable media provide non-volatile storage for the computer device 1800. In other words, the mass storage device 1807 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0289] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technology, CD-ROM, DVD (Digital Video Disc) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the above-mentioned ones. The above-mentioned system memory 1804 and mass storage device 1807 can be collectively referred to as memory.

[0290] According to an embodiment of the present application, the computer device 1800 can also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 1800 can be connected to the network 1812 via the network interface unit 1811 connected to the system bus 1805. Alternatively, the network interface unit 1811 can be used to connect to other types of networks or remote computer systems (not shown).

[0291] The memory further includes a computer program, which is stored in the memory and configured to be executed by one or more processors to implement the above-mentioned object registration method.

[0292] In an exemplary embodiment, a computer-readable storage medium is further provided, wherein a computer program is stored in the storage medium. When the computer program is executed by a processor, the computer program is used to implement the above-mentioned target registration method.

[0293] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or an optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0294] In an exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described object registration method.

[0295] It should be noted that the information (including but not limited to the subject's device information, subject's personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are all authorized by the subject or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the images involved in this application (such as images to be registered, reference images, sample images, etc.) were all obtained with full authorization.

[0296] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application do not limit this.

[0297] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A target registration method, characterized in that: The method comprises: Acquire an image to be registered and a reference image containing the same object; Detecting the image to be registered to obtain a pose matrix of the target in the image to be registered, wherein the pose matrix is ​​used to represent the position and posture of the target in the image; Based on the pose matrix of the target in the image to be registered and the pose matrix of the target in the reference image, obtaining an initial registration relationship between the image to be registered and the reference image; wherein the initial registration relationship is used to preliminarily characterize the transformation relationship between the image to be registered and the reference image; Obtaining a preliminary registration result corresponding to the target in the image to be registered according to the initial registration relationship and the reference image; Based on the effective area of ​​the depth image corresponding to the image to be registered and the preliminary registration result, obtaining an optimized registration relationship between the image to be registered and the reference image; wherein the effective area of ​​the depth image includes the target; According to the optimized registration relationship, the preliminary registration result is adjusted to obtain an optimized registration result corresponding to the target in the registered image; wherein the optimized registration result is used to guide the subject to process the target.

2. The method according to claim 1, characterized in that The obtaining of an optimized registration relationship between the image to be registered and the reference image based on the valid area of ​​the depth image corresponding to the image to be registered and the preliminary registration result includes: Obtaining a point cloud corresponding to the preliminary registration result; Removing outliers in the point cloud corresponding to the preliminary registration result to obtain an optimized point cloud; Extracting a target point cloud from the optimized point cloud according to the shooting direction corresponding to the image to be registered, wherein the target point cloud refers to a point cloud observed in the shooting direction; Based on the target point cloud and the point cloud corresponding to the valid area, an optimized registration relationship between the image to be registered and the reference image is obtained through iterative optimization.

3. The method according to claim 1, characterized in that The detecting the image to be registered to obtain a pose matrix of the target in the image to be registered includes: Performing target detection on the image to be registered to obtain a bounding box corresponding to the target; Based on the bounding box corresponding to the target, extracting a captured image including the target from the image to be registered; Performing key point detection on the captured image to obtain target key points corresponding to the target; Performing posture detection on the captured image to obtain a rotation matrix corresponding to the target, wherein the rotation matrix is ​​used to represent the posture of the target; Based on the target key points and the rotation matrix, a pose matrix of the target in the image to be registered is obtained.

4. The method according to claim 3, characterized in that The step of intercepting a captured image including the target from the image to be registered based on the bounding box corresponding to the target comprises: Enlarging the bounding box to obtain an enlarged bounding box; A captured image including the target is captured from the image to be registered according to the enlarged bounding box.

5. The method according to claim 3, characterized in that The step of obtaining a pose matrix of the target in the image to be registered based on the target key points and the rotation matrix includes: Acquiring depth information of the target key points based on the depth image corresponding to the image to be registered; Based on the plane coordinates of the target key point in the image to be registered and the depth information of the target key point, obtaining a translation vector of the target key point in the world coordinate system; wherein the translation vector is used to represent the position of the target key point in the image to be registered; Based on the translation vector and the rotation matrix, a pose matrix of the target in the image to be registered is constructed.

6. The method according to claim 3, characterized in that The performing posture detection on the captured image to obtain a rotation matrix corresponding to the target includes: Performing posture detection on the captured image to obtain an output vector corresponding to the target, wherein the output vector is used to represent the posture of the target in the captured image; Splitting the output vector to obtain a first orthogonal rotation vector and a second orthogonal rotation vector; Transforming the first orthogonal rotation vector to obtain a first sub-matrix; Obtaining a second submatrix based on the first submatrix and the second orthogonal rotation vector; transforming the second orthogonal rotation vector to obtain a third submatrix; Obtaining a fourth submatrix based on the first submatrix and the third submatrix; A rotation matrix corresponding to the target is obtained based on the first submatrix, the third submatrix, and the fourth submatrix.

7. The method according to claim 1, characterized in that The acquiring an initial registration relationship between the image to be registered and the reference image based on the pose matrix of the target in the image to be registered and the pose matrix of the target in the reference image includes: Based on the reference image, construct a reference model corresponding to the target; Based on the standard data corresponding to the target, the reference model is subjected to a rigid body transformation to obtain a transformed reference model; Based on the transformed reference model, obtaining a pose matrix of the target in the reference image; The pose matrix of the target in the image to be registered is divided by the pose matrix of the target in the reference image to obtain an initial registration relationship between the image to be registered and the reference image.

8. The method according to claim 7, characterized in that The constructing a reference model corresponding to the target based on the reference image includes: Based on the surface data of the target in the reference image, obtaining a binary segmentation image corresponding to the target; The binary segmented image is converted to obtain a reference model corresponding to the target; wherein the reference model is composed of a plurality of triangular facets.

9. The method according to claim 7, characterized in that The obtaining, based on the initial registration relationship and the reference image, a preliminary registration result corresponding to the target in the image to be registered, includes: Adjusting the target in the image to be registered according to the initial registration relationship to obtain a preliminary registration model corresponding to the target in the registered image; The adjusting the preliminary registration result according to the optimized registration relationship to obtain the optimized registration result corresponding to the target in the registered image includes: According to the optimized registration relationship, the preliminary registration model is adjusted to obtain an optimized registration model corresponding to the target in the registered image.

10. The method according to claim 1, characterized in that The method further comprises: Converting the optimized registration result from the world coordinate system to the optimized registration result in the image acquisition device coordinate system; Synchronously displaying the navigation stick and the needle tip in the optimized registration result in the image acquisition device coordinate system according to the positions of the navigation stick and the needle tip in the registration image; The optimized registration result includes a navigation organ, and the navigation organ is used to assist the object in processing the target.

11. The method according to claim 1, wherein The method further comprises: Obtain preliminary registration results after manual registration; Dividing the effective area of ​​the depth image to obtain a rigid area and a deformable area corresponding to the effective area; Iteratively optimizing the optimized registration relationship between the image to be registered and the reference image based on the point cloud corresponding to the preliminary registration result, the point cloud corresponding to the valid area, the point cloud weight parameter corresponding to the rigid area, and the point cloud weight parameter corresponding to the deformable area; The point cloud weight parameter corresponding to the rigid area is greater than the point cloud weight parameter corresponding to the deformable area.

12. A target registration device, characterized in that: The device comprises: A registration image acquisition module is used to acquire an image to be registered and a reference image containing the same object; A pose matrix acquisition module is used to detect the image to be registered and obtain a pose matrix of the target in the image to be registered, wherein the pose matrix is ​​used to represent the position and posture of the target in the image; an initial relationship acquisition module, configured to acquire an initial registration relationship between the image to be registered and the reference image based on the pose matrix of the target in the image to be registered and the pose matrix of the target in the reference image; wherein the initial registration relationship is used to preliminarily characterize the transformation relationship between the image to be registered and the reference image; A preliminary result acquisition module, configured to acquire a preliminary registration result corresponding to the target in the image to be registered based on the initial registration relationship and the reference image; an optimized relationship acquisition module, configured to acquire an optimized registration relationship between the image to be registered and the reference image based on a valid area of ​​the depth image corresponding to the image to be registered and the preliminary registration result; wherein the valid area of ​​the depth image includes the target; The optimization result acquisition module is used to adjust the preliminary registration result according to the optimized registration relationship and obtain the optimized registration result corresponding to the target in the registered image; wherein the optimized registration result is used to guide the object to process the target.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the object registration method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the object registration method according to any one of claims 1 to 11.

15. A computer program product or a computer program, characterized in that The computer program product or computer program includes computer instructions, which are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the object registration method according to any one of claims 1 to 11.

Images